PATCH: terminfo configuration redux

PATCH: terminfo configuration redux

[Peter Stephenson]

Here is another patch for terminfo configuration.  It's very much
like the original patch with a couple of things fixed.  I've
used tigetflag consistently as the test function; before it switched
between tigetflag and tigetstr.

To be honest, I simply don't know if this is good enough to fix the
warning about term.h from configure or not.

I have not attempted to search for termcap in non standard places.
There should be enough hooks (--enable-cppflags, --enable-libs) to
enable that if you need it.


One other thing I noticed when running configure was that this:

  AC_CHECK_PROG([PCRECONF], pcre-config, pcre-config)
  if test "x$ac_cv_prog_PCRECONF" = xpcre-config; then
    CFLAGS="$CFLAGS `pcre-config --cflags`"
  fi

causes problems: it actually adds a -I header.  If that goes into
CFLAGS, it isn't picked up by the preprocessor tests, which therefore
can't find pcre.h.  Hence the output really needs to be added to
CPPFLAGS (somewhat counterintuitively).  If pcre-config sometimes
outputs stuff for the compiler itself we are more stuck.


Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.8
diff -u -r1.8 configure.ac
--- configure.ac	20 Feb 2004 15:27:35 -0000	1.8
+++ configure.ac	25 Feb 2004 10:59:31 -0000
@@ -620,8 +620,7 @@
 AH_TEMPLATE([TERM_H_NEEDS_CURSES_H],
 [Define if term.h chokes without curses.h.])
 AC_SEARCH_LIBS(tgetent, [$termcap_curses_order])
-case "$LIBS" in
-*curses*)
+AC_SEARCH_LIBS(tigetflag, [$termcap_curses_order])
 AC_CHECK_HEADERS(curses.h, [],
 [AC_CACHE_CHECK(for Solaris 8 curses.h mistake, ac_cv_header_curses_solaris,
 AC_TRY_COMPILE([#include <curses.h>], [],
@@ -683,8 +682,7 @@
 #include <term.h>], [char **test = strnames; printf(*test);],
 AC_DEFINE(HAVE_STRNAMES) strnames=yes, strnames=no)
 AC_MSG_RESULT($strnames)
-]);;
-esac
+])
 
 dnl Some systems (Solaris 2.x, Linux Redhat 5.x) require
 dnl libnsl (Network Services Library) to find yp_all
Index: Src/Modules/terminfo.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/terminfo.c,v
retrieving revision 1.22
diff -u -r1.22 terminfo.c
--- Src/Modules/terminfo.c	14 Sep 2003 05:10:09 -0000	1.22
+++ Src/Modules/terminfo.c	25 Feb 2004 10:59:31 -0000
@@ -29,12 +29,18 @@
 
 #define USES_TERM_H 1
 #include "terminfo.mdh"
-#include "terminfo.pro"
 
+#if defined(HAVE_TIGETFLAG) && defined(HAVE_CURSES_H)
+# define USE_TERMINFO_MODULE 1
+#else
+# undef USE_TERMINFO_MODULE
+#endif
+
+#include "terminfo.pro"
 static char terminfo_nam[] = "terminfo";
 
 /**/
-#ifdef HAVE_TIGETSTR
+#ifdef USE_TERMINFO_MODULE
 
 /* The following two undefs are needed for Solaris 2.6 */
 # ifdef VINTR
@@ -44,9 +50,7 @@
 #  undef offsetof
 # endif
 
-# ifdef HAVE_CURSES_H
-#  include <curses.h>
-# endif
+# include <curses.h>
 # ifdef HAVE_TERM_H
 #  include <term.h>
 # endif
@@ -123,19 +127,19 @@
 }
 
 /**/
-#else /* !HAVE_TIGETSTR */
+#else /* !USE_TERMINFO_MODULE */
 
 #define bin_echoti bin_notavail
 
 /**/
-#endif /* !HAVE_TIGETSTR */
+#endif /* !USE_TERMINFO_MODULE */
 
 static struct builtin bintab[] = {
     BUILTIN("echoti", 0, bin_echoti, 1, -1, 0, NULL, NULL),
 };
 
 /**/
-#ifdef HAVE_TIGETSTR
+#ifdef USE_TERMINFO_MODULE
 
 /* Empty dummy function for special hash parameters. */
 
@@ -361,7 +365,7 @@
 }
 
 /**/
-#endif /* HAVE_TIGETSTR */
+#endif /* USE_TERMINOF_MODULE */
 
 /**/
 int
@@ -374,7 +378,7 @@
 int
 boot_(Module m)
 {
-#ifdef HAVE_TIGETSTR
+#ifdef USE_TERMINFO_MODULE
 # ifdef HAVE_SETUPTERM
     int errret;
 
@@ -394,7 +398,7 @@
 int
 cleanup_(Module m)
 {
-#ifdef HAVE_TIGETSTR
+#ifdef USE_TERMINFO_MODULE
     Param pm;
 
     if ((pm = (Param) paramtab->getnode(paramtab, terminfo_nam)) &&
Index: Src/Modules/terminfo.mdd
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/terminfo.mdd,v
retrieving revision 1.10
diff -u -r1.10 terminfo.mdd
--- Src/Modules/terminfo.mdd	23 Apr 2001 19:59:04 -0000	1.10
+++ Src/Modules/terminfo.mdd	25 Feb 2004 10:59:31 -0000
@@ -1,13 +1,13 @@
 name=zsh/terminfo
 
-link='if test "x$ac_cv_func_tigetstr" = xyes; then
+link='if test "x$ac_cv_func_tigetflag" = xyes -a "x$ac_cv_header_curses_h" = xyes; then
           if test "x$zsh_cv_shared_tigetstr" = xyes; then
 	      echo either
 	  else
 	      echo static
 	  fi
       else
-          echo either;
+          echo no;
       fi
 '
 load=yes

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: PATCH: terminfo configuration redux

[Bart Schaefer]

On Feb 25, 11:24am, Peter Stephenson wrote:
}
} Here is another patch for terminfo configuration.  It's very much
} like the original patch with a couple of things fixed.

Works fine for me; changed the configuration not a whit.  However,
I used "configure --with-curses-terminfo" ... should I also try
without that, or should it not matter?  (Sorry for not just checking
the source for the answer to that, but I have to run right now.)

Re: PATCH: terminfo configuration redux

[Peter Stephenson]

Bart Schaefer wrote:
> On Feb 25, 11:24am, Peter Stephenson wrote:
> }
> } Here is another patch for terminfo configuration.  It's very much
> } like the original patch with a couple of things fixed.
> 
> Works fine for me; changed the configuration not a whit.  However,
> I used "configure --with-curses-terminfo" ... should I also try
> without that, or should it not matter?  (Sorry for not just checking
> the source for the answer to that, but I have to run right now.)

It should matter less than it used to.  It now respects the search
order, but looks for curses anyway.  The potential problem is that
it will find symbols in both libraries and get into difficulties
linking with both.

I'm not sure how much we actually need to prefer the termcap library,
anyway.  On most systems now this is just a kludged-up subset of
curses where it exists at all.  However, I don't really know why
the choice was necessary in the first place --- maybe it was just
caution.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: PATCH: terminfo configuration redux

[peta]

Peter Stephenson <pws@csr.com> wrote:

> Here is another patch for terminfo configuration.  It's very much
> like the original patch with a couple of things fixed. 

It fixes a problem I'd previously reported on FreeBSD 4.6 with configure
not finding curses.  Clint Adams replied it was due to libtinfo being
linked to libcurses (zsh-workers 19260).

-- 
Peter Whaite (http://whaite.ca)

Re: PATCH: terminfo configuration redux

[Oliver Kiddle]

Bart wrote:
> On Feb 25, 11:24am, Peter Stephenson wrote:
> }
> } Here is another patch for terminfo configuration.  It's very much
> } like the original patch with a couple of things fixed.
> 
> Works fine for me; changed the configuration not a whit.  However,
> I used "configure --with-curses-terminfo" ... should I also try

Works fine for me on a few different systems. On the machine where I had
problems before, I tried it both with and without --with-curses-terminfo.

Oliver

Re: PATCH: terminfo configuration redux

[Peter Stephenson]

Peter Stephenson wrote:
> Here is another patch for terminfo configuration.

Now committed.

We should be in a position to start 4.2 test releases.  I will make a
4.2.0-pre-1 at some point.  (This means changing references to 4.1
in the documentation to 4.2.  I might easily miss some.)

Does anyone know of unresolved porting issues?  We get occasional reports
about weirdness with AIX and I don't know how we stand with the latest
code.  I don't think Etc/MACHINES is particularly up to date, either.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: PATCH: terminfo configuration redux

[Felix Rosencrantz]

I have some good news, and bad news.

I'm seeing the ERR problems on Fedora Core 1.
gcc -c -I. -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/pcre
-DHAVE_CONFIG_H -DMODULE -Wall -Wmissing-prototypes -ggdb -fPIC -o terminfo..o
terminfo.c
terminfo.c: In function `bin_echoti':
terminfo.c:74: warning: implicit declaration of function `tigetnum'
terminfo.c:79: warning: implicit declaration of function `tigetflag'
terminfo.c:91: warning: implicit declaration of function `tigetstr'
terminfo.c:117: warning: implicit declaration of function `putp'
terminfo.c:119: warning: implicit declaration of function `tparm'
terminfo.c: In function `boot_':
terminfo.c:381: warning: implicit declaration of function `setupterm'
terminfo.c:381: error: `ERR' undeclared (first use in this function)
terminfo.c:381: error: (Each undeclared identifier is reported only once
terminfo.c:381: error: for each function it appears in.)
make[3]: *** [terminfo..o] Error 1



Though on FreeBSD 4.7, it builds great. :)

-FR.

__________________________________
Do you Yahoo!?
Get better spam protection with Yahoo! Mail.
http://antispam.yahoo.com/tools

Re: PATCH: terminfo configuration redux

[Peter Stephenson]

Felix Rosencrantz wrote:
> I'm seeing the ERR problems on Fedora Core 1.

Can you work out how it can fail to include curses.h?  There are two
tests why that can't happen.

pws


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: PATCH: terminfo configuration redux

[Felix Rosencrantz]

System setup issues...  The ncurses-devel was not installed.  It built fine
after installing that package.

It would be helpful if during configure, there was some warning of this type of
failure... Though, I know this has been a pesky issue, and it seems to work
provided the includes/libraries are there.

-FR.

__________________________________
Do you Yahoo!?
Get better spam protection with Yahoo! Mail.
http://antispam.yahoo.com/tools

Re: PATCH: terminfo configuration redux

[Peter Stephenson]

Felix Rosencrantz wrote:
> System setup issues...  The ncurses-devel was not installed.  It built fine
> after installing that package.

Does that mean that curses.h is present even without ncurses-devel?  If
not, it shouldn't have tried to compile terminfo at all without that
installed.  If it is, I'm surprised since the usual practice is to put
headers in the -devel package.

pws

Re: PATCH: terminfo configuration redux

[Felix Rosencrantz]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 528 bytes --]

--- Peter Stephenson <pws@pwstephenson.fsnet.co.uk> wrote:
> Felix Rosencrantz wrote:
> > The ncurses-devel was not installed.
> Does that mean that curses.h is present even without ncurses-devel?
No.

The file curses.h is present only with ncurses-devel.  It gets installed by the
ncurses-devel package.  The build  dies on ERR problem in terminfo.c, if
ncurses-devel package is uninstalled. 

-FR.

__________________________________
Do you Yahoo!?
Yahoo! Search - Find what you’re looking for faster
http://search.yahoo.com

Re: PATCH: terminfo configuration redux

[Peter Stephenson]

Felix Rosencrantz wrote:
> --- Peter Stephenson <pws@pwstephenson.fsnet.co.uk> wrote:
> > Felix Rosencrantz wrote:
> > > The ncurses-devel was not installed.
> > Does that mean that curses.h is present even without ncurses-devel?
> No.
> 
> The file curses.h is present only with ncurses-devel.  It gets installed by=
>  the
> ncurses-devel package.  The build  dies on ERR problem in terminfo.c, if
> ncurses-devel package is uninstalled.=20

I don't see how that can happen.  terminfo.mdd won't let terminfo.c be
compiled if there's no curses.h (it should set link=no in
config.modules), and even if terminfo.c get's compiled
USE_TERMINFO_MODULE won't be defined unless HAVE_CURSES_H is present.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: PATCH: terminfo configuration redux

[Felix Rosencrantz]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 1012 bytes --]

Here is the output of configure pulling out lines containing term|curses:
checking termcap.h usability... yes
checking termcap.h presence... yes
checking for termcap.h... yes
checking termio.h usability... yes
checking termio.h presence... yes
checking for termio.h... yes
checking termios.h usability... yes
checking termios.h presence... yes
checking for termios.h... yes
checking POSIX termios... yes
checking TIOCGWINSZ in termios.h... no
checking for library containing tgetent... -ltermcap
checking for library containing tigetflag... -lcurses
checking curses.h usability... no
checking curses.h presence... no
checking for curses.h... no
checking for Solaris 8 curses.h mistake... no
checking term.h usability... no
checking term.h presence... no
checking for term.h... no
checking for setupterm... yes
library flags             : -ldl -lnsl -lcurses -ltermcap -lm  -lc

-FR.


__________________________________
Do you Yahoo!?
Yahoo! Search - Find what you’re looking for faster
http://search.yahoo.com

Re: PATCH: terminfo configuration redux

[Peter Stephenson]

Felix Rosencrantz wrote:
> Here is the output of configure pulling out lines containing term|curses:
> checking curses.h usability... no
> checking curses.h presence... no
> checking for curses.h... no

This should mean ac_cv_header_curses_h is not `yes', so terminfo won't
be compiled, and HAVE_CURSES_H is not defined, so that even if it is
compiled it is only a stub with no curses stuff in it.  Otherwise
something is very weirdly broken.  Maybe later parts of config.log give
a clue.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: PATCH: terminfo configuration redux

[Felix Rosencrantz]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 4313 bytes --]

--- Peter Stephenson <pws@csr.com> wrote:
> This should mean ac_cv_header_curses_h is not `yes', so terminfo won't
> be compiled, and HAVE_CURSES_H is not defined, so that even if it is
> compiled it is only a stub with no curses stuff in it.  Otherwise
> something is very weirdly broken.  Maybe later parts of config.log give
> a clue.

Yet the build still fails at terminfo.c because ERR is not defined.  

Some results from config.log are listed below The log is big, if there are more
details you need let me know.  I'm not sure which details are of most interest
to you.

-FR.

configure:5727: checking for library containing tgetent
configure:5758: gcc -o conftest  -Wall -Wmissing-prototypes -ggdb
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g conftest.c -lm  -lc >&5
/tmp/ccKOFvzo.o(.text+0x11): In function `main':
/configure:5815: undefined reference to `tgetent'
collect2: ld returned 1 exit status
configure:5761: $? = 1
configure:5803: gcc -o conftest  -Wall -Wmissing-prototypes -ggdb
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g conftest.c -ltinfo  -lm  -lc >&5
/usr/bin/ld: cannot find -ltinfo
collect2: ld returned 1 exit status
configure:5806: $? = 1
configure: failed program was:
configure:5803: gcc -o conftest  -Wall -Wmissing-prototypes -ggdb
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g conftest.c -ltermcap  -lm  -lc
>&5
configure:5806: $? = 0
configure:5809: test -s conftest
configure:5812: $? = 0
configure:5826: result: -ltermcap
configure:5833: checking for library containing tigetflag
configure:5864: gcc -o conftest  -Wall -Wmissing-prototypes -ggdb
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g conftest.c -ltermcap -lm  -lc >&5
/tmp/ccJBKnPf.o(.text+0x11): In function `main':
configure:5921: undefined reference to `tigetflag'
collect2: ld returned 1 exit status
configure:5867: $? = 1
configure: failed program was:
configure:5909: gcc -o conftest  -Wall -Wmissing-prototypes -ggdb
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g conftest.c -ltinfo  -ltermcap -lm
 -lc >&5
/usr/bin/ld: cannot find -ltinfo
collect2: ld returned 1 exit status
configure:5912: $? = 1
configure: failed program was:
configure:5909: gcc -o conftest  -Wall -Wmissing-prototypes -ggdb
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g conftest.c -ltermcap  -ltermcap
-lm  -lc >&5
/tmp/ccmqEjRC.o(.text+0x11): In function `main':
configure:5966: undefined reference to `tigetflag'
collect2: ld returned 1 exit status
configure:5912: $? = 1
configure: failed program was:
configure:5909: gcc -o conftest  -Wall -Wmissing-prototypes -ggdb
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g conftest.c -lcurses  -ltermcap
-lm  -lc >&5
configure:5912: $? = 0
configure:5915: test -s conftest
configure:5918: $? = 0
configure:5932: result: -lcurses
configure:5953: checking curses.h usability
configure:5966: gcc -c  -Wall -Wmissing-prototypes -ggdb -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 conftest.c >&5
configure:6059:20: curses.h: No such file or directory
configure:5969: $? = 1
configure: failed program was:
configure:5985: result: no
configure:5989: checking curses.h presence
configure:6000: gcc -E -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 conftest.c
configure:6061:20: curses.h: No such file or directory
configure:6006: $? = 1
configure: failed program was:
configure:6025: result: no
configure:6061: checking for curses.h
configure:6068: result: no
configure:6078: checking for Solaris 8 curses.h mistake
configure:6100: gcc -c  -Wall -Wmissing-prototypes -ggdb -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 conftest.c >&5
configure:6153:20: curses.h: No such file or directory
configure:6103: $? = 1
configure:6122: result: no
configure:6148: checking term.h usability
configure:6161: gcc -c  -Wall -Wmissing-prototypes -ggdb -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 conftest.c >&5
configure:6254:18: term.h: No such file or directory
configure:6164: $? = 1
configure:6180: result: no
configure:6184: checking term.h presence
configure:6195: gcc -E -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 conftest.c
configure:6256:18: term.h: No such file or directory
configure:6201: $? = 1
configure:6220: result: no
configure:6256: checking for term.h
configure:6263: result: no

__________________________________
Do you Yahoo!?
Yahoo! Search - Find what you’re looking for faster
http://search.yahoo.com

Re: PATCH: terminfo configuration redux

[Peter Stephenson]

Felix Rosencrantz wrote:
> Some results from config.log are listed below The log is big, if there are
> more
> details you need let me know.  I'm not sure which details are of most
> inter est to you.

We need to know why HAVE_CURSES_H and $ac_cv_header_curses_h are
apparently being defined without a curses.h being present.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: Is this a bug for zsh 4.2.3?

[Bart Schaefer]

[moved to -workers]

On Jan 24, 10:52am, Peter Stephenson wrote:
} Subject: Re: Is this a bug for zsh 4.2.3?
}
} Bart Schaefer wrote:
} > It's a pretty
} > serious bug, one that probably warrants a 4.2.4.
} 
} I don't have the energy, myself...

Hrm.  I don't have much either, I'm in the midst of an unpleasant chest
cold.  What's involved nowadays?  If, for example, I set up a patches
branch in CVS, can you make tarfiles to put wherever?  (Or are we now
trying to avoid having a patches branch?)

Though it would be nice to get the Cygwin patches for the configure
scripts (mentioned by someone last week) in there as well.

Meanwhile, what's the scheme nowadays for version numbering the dev
versions in CVS?  It's presently 4.2.3-dev-1, but it seems like it
should be either 4.3.0-dev-1 or 4.2.4-dev-1.

Re: Is this a bug for zsh 4.2.3?

[Peter Stephenson]

Bart Schaefer wrote:
> Hrm.  I don't have much either, I'm in the midst of an unpleasant chest
> cold.  What's involved nowadays?  If, for example, I set up a patches
> branch in CVS, can you make tarfiles to put wherever?  (Or are we now
> trying to avoid having a patches branch?)

I would think so...

> Though it would be nice to get the Cygwin patches for the configure
> scripts (mentioned by someone last week) in there as well.

I haven't seen what's involved (it seems to compile so I don't know
what the problem is).

> Meanwhile, what's the scheme nowadays for version numbering the dev
> versions in CVS?  It's presently 4.2.3-dev-1, but it seems like it
> should be either 4.3.0-dev-1 or 4.2.4-dev-1.

We keep the individual sets of digits in a logical order, so 4.2.3-dev-1
is after 4.2.3.  This makes automated version testing work (is_at_least,
for example).  Test versions break this but they're supposed to be short
lived (using 4.2.3-test-A would work and I think I've done something like
that in the past).

The other fix is to sacrifice a version number that will never be released.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: localtraps

[Peter Stephenson]

Bart Schaefer wrote:
> So localtraps eventually worked in the sense that the inner trap was
> removed, but the outer trap was never reset in the signal handler.

OK, this one's fairly straightforward: starttrapscope() and
endtrapscope() don't run inside traps.  This is presumably to stop the
exit trap from being triggered; I don't remember any conscious decision
to prevent traps being restored on exit from other traps.  What's more,
the trap appears to be being saved, so we may have had a memory leak.

Are we reasonably confident the other problems are specific to the
NetBSD style of signal handling?  I can't debug this, but we could
probably introduce an option so that the signals were always queued and
handled later (probably requiring more places where we need to check
that queues should be run).

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.89
diff -u -r1.89 exec.c
--- Src/exec.c	15 Apr 2005 10:40:14 -0000	1.89
+++ Src/exec.c	27 Apr 2005 09:47:39 -0000
@@ -3669,8 +3669,7 @@
 	memcpy(oldpipestats, pipestats, bytes);
     }
 
-    if (!intrap)
-	starttrapscope();
+    starttrapscope();
 
     tab = pparams;
     if (!(flags & PM_UNDEFINED))
@@ -3770,8 +3769,7 @@
 	opts[LOCALOPTIONS] = saveopts[LOCALOPTIONS];
     }
 
-    if (!intrap)
-	endtrapscope();
+    endtrapscope();
 
     if (trapreturn < -1)
 	trapreturn++;
Index: Src/signals.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/signals.c,v
retrieving revision 1.36
diff -u -r1.36 signals.c
--- Src/signals.c	6 Feb 2005 20:36:44 -0000	1.36
+++ Src/signals.c	27 Apr 2005 09:47:39 -0000
@@ -888,6 +888,10 @@
 void
 starttrapscope(void)
 {
+    /* No special SIGEXIT behaviour inside another trap. */
+    if (intrap)
+	return;
+
     /*
      * SIGEXIT needs to be restored at the current locallevel,
      * so give it the next higher one. dosavetrap() is called
@@ -917,8 +921,11 @@
     /*
      * Remember the exit trap, but don't run it until
      * after all the other traps have been put back.
+     * Don't do this inside another trap.
      */
-    if ((exittr = sigtrapped[SIGEXIT])) {
+    if (intrap)
+	exittr = 0;
+    else if ((exittr = sigtrapped[SIGEXIT])) {
 	if (exittr & ZSIG_FUNC) {
 	    exitfn = removehashnode(shfunctab, "TRAPEXIT");
 	} else {

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: localtraps

[Bart Schaefer]

On Apr 27, 10:54am, Peter Stephenson wrote:
}
} OK, this one's fairly straightforward: starttrapscope() and
} endtrapscope() don't run inside traps.

This works much better now, though there's still this extra newline
being printed just before INNER on each alternate interrupt:

-----
schaefer<501> TRAPINT() { setopt localoptions localtraps; echo OUTER; \
trap 'echo INNER' INT; sleep 2 }
schaefer<502> OUTER

INNER
OUTER

INNER
OUTER

INNER
OUTER

INNER
OUTER

INNER

schaefer<502> 
-----

Plus the prompt not reprinted until accept-line or send-break, but that
probably isn't a bug.

} Are we reasonably confident the other problems are specific to the
} NetBSD style of signal handling?

I'm not confident of that, no.  Let's wait and see what Vincent has to
say after this patch is applied.

Re: localtraps

[Vincent Stemen]

On Wed, Apr 27, 2005 at 02:09:31PM +0000, Bart Schaefer wrote:
> On Apr 27, 10:54am, Peter Stephenson wrote:
> }
> } OK, this one's fairly straightforward: starttrapscope() and
> } endtrapscope() don't run inside traps.
> 
> This works much better now, though there's still this extra newline
> being printed just before INNER on each alternate interrupt:
> ...
> 
> Plus the prompt not reprinted until accept-line or send-break, but that
> probably isn't a bug.
> 
> } Are we reasonably confident the other problems are specific to the
> } NetBSD style of signal handling?
> 
> I'm not confident of that, no.  Let's wait and see what Vincent has to
> say after this patch is applied.

Sorry about being a bit slow about getting back to you on this.
Thanks Bart for letting me know this thread was moved to zsh-workers.

I checked out zsh-4.2.5 from CVS and then updated to the
zsh-4_2-patches branch.

It still does not work correctly but the behavior has changed.
In the results below, I will refer to zsh-4.2.1 as "oldz" and the
patched 4.2.5 as "newz".

#======== <test script> ==========
sigterm1()
{
#   setopt LOCAL_TRAPS

    trap sigterm2 TERM
    echo "sigterm1(): sending SIGTERM"
    kill -s TERM $$
#   trap sigterm1 TERM
    sleep 1
}

sigterm2()
{
    echo "sigterm2()"
}

trap sigterm1 TERM

echo
echo "main: sending SIGTERM"
kill -s TERM $$
echo "main: sending SIGTERM"
kill -s TERM $$
#======== </test script> ==========

** Uncomment "setopt LOCAL_TRAPS"

<oldz>
main: sending SIGTERM
sigterm1(): sending SIGTERM
sigterm2()
main: sending SIGTERM
sigterm2()
</oldz>

As we already know, localtraps is ignored.

<newz>
main: sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
^C
</newz>

Continued indefinitely.  The "trap sigterm2 TERM" statement is ignored
when localtraps is on.


** Comment out "setopt LOCAL_TRAPS" and uncomment "trap sigterm1 TERM"
   in sigterm1() to try to accomplish the same thing without localopts.

<oldz>
main: sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
^C
</oldz>

<newz>
main: sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
^C
</newz>

<bsd sh>
main: sending SIGTERM
sigterm1(): sending SIGTERM
sigterm2()
main: sending SIGTERM
sigterm1(): sending SIGTERM
sigterm2()
</bsd sh>

newz is the same as oldz.

It works under BSD-sh because it processes the signal immediately so
that it does not execute the "trap sigterm1 TERM" before calling the
sig handler again.  This is the desired behavior.


On a side note, even though the binaries all seemed to compile OK, I
got a compilation error on NetBSD with 4.2.5 with the zsh-4_2-patches
that looks to me like a failure in generating the manuals.  Here is
the error.

-------------------------------------------------------------
rm -f zleparameter.so
gcc  -s -Wl,-x -shared --whole-archive -o zleparameter.so   zleparameter..o    -ltermcap
 -lm  -lc
(  echo 'STARTDEF()';  echo 'def(version)(0)('4.2.5')';  echo 'def(date)(0)(''April 6, 2
005'')';  echo 'ENDDEF()#' | tr '#' '\\';  ) > ./version.yo
case zsh.1 in  */*) target=zsh.1 ;;  *) target=./zsh.1 ;;  esac;  case ': yodl' in :*) ;
; *)  : yodl -I. -w zman.yo version.yo zsh.yo | sed -e '1s/\\-/-/g' -e '/^\.'\''/d' > $t
arget  ;; esac;  test -f $target
*** Error code 1

Stop.
make: stopped in /home/src/zsh-4.2.5-20050430/Doc
*** Error code 1

Stop.
make: stopped in /home/src/zsh-4.2.5-20050430
-------------------------------------------------------------


To compare the result, I compiled the 4.2.5 release and it ended without
error after the line,

-------------------------------------------------------------
gcc  -s -Wl,-x -shared --whole-archive -o zleparameter.so   zleparameter..o    -ltermcap
 -lm  -lc
-------------------------------------------------------------

So it is something in the latest patches.


-- 
Vincent Stemen
Avoid the VeriSign/Network Solutions domain registration trap!
Read how Network Solutions (NSI) was involved in stealing our domain name.
http://www.InetAddresses.net

Re: localtraps

[Peter Stephenson]

Vincent Stemen wrote:
> On a side note, even though the binaries all seemed to compile OK, I
> got a compilation error on NetBSD with 4.2.5 with the zsh-4_2-patches
> that looks to me like a failure in generating the manuals.  Here is
> the error.
> 
> -------------------------------------------------------------
> rm -f zleparameter.so
> gcc  -s -Wl,-x -shared --whole-archive -o zleparameter.so   zleparameter..o  
>   -ltermcap
>  -lm  -lc
> (  echo 'STARTDEF()';  echo 'def(version)(0)('4.2.5')';  echo 'def(date)(0)('
> 'April 6, 2
> 005'')';  echo 'ENDDEF()#' | tr '#' '\\';  ) > ./version.yo
> case zsh.1 in  */*) target=zsh.1 ;;  *) target=./zsh.1 ;;  esac;  case ': yod
> l' in :*) ;
> ; *)  : yodl -I. -w zman.yo version.yo zsh.yo | sed -e '1s/\\-/-/g' -e '/^\.'
> \''/d' > $t
> arget  ;; esac;  test -f $target
> *** Error code 1

You don't have yodl, right?  Configure hasn't detected it so it's put a
": " in front.  This should mean it runs through OK, but without
generating the documentation, although it will produce an empty file.
It's not obvious why it gave an error --- unless the wrap between "$t"
and "arget" was there in the original, which is unlikely.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: yodl and Z shell documentation (was localtraps)

[Vincent Stemen]

On Tue, May 03, 2005 at 11:04:07AM +0100, Peter Stephenson wrote:
> Vincent Stemen wrote:
> > On a side note, even though the binaries all seemed to compile OK, I
> > got a compilation error on NetBSD with 4.2.5 with the zsh-4_2-patches
> > that looks to me like a failure in generating the manuals.  Here is
> > the error.
> > 
> > -------------------------------------------------------------
> > rm -f zleparameter.so
> > gcc  -s -Wl,-x -shared --whole-archive -o zleparameter.so   zleparameter..o  
> >   -ltermcap
> >  -lm  -lc
> > (  echo 'STARTDEF()';  echo 'def(version)(0)('4.2.5')';  echo 'def(date)(0)('
> > 'April 6, 2
> > 005'')';  echo 'ENDDEF()#' | tr '#' '\\';  ) > ./version.yo
> > case zsh.1 in  */*) target=zsh.1 ;;  *) target=./zsh.1 ;;  esac;  case ': yod
> > l' in :*) ;
> > ; *)  : yodl -I. -w zman.yo version.yo zsh.yo | sed -e '1s/\\-/-/g' -e '/^\.'
> > \''/d' > $t
> > arget  ;; esac;  test -f $target
> > *** Error code 1
> 
> You don't have yodl, right?  Configure hasn't detected it so it's put a
> ": " in front.  This should mean it runs through OK, but without
> generating the documentation, although it will produce an empty file.
> It's not obvious why it gave an error --- unless the wrap between "$t"
> and "arget" was there in the original, which is unlikely.

You are right.  I didn't.  Thanks.  I installed yodl-1.31.18nb3 NetBSD
package and it compiled ok.  I was not familiar with yodl.  I have
been using perldoc for documentation for most of the projects I am
working on.

I am unclear about the status and future of yodl.  I am curious, is
this the documentation format planned for the foreseeable future of Z
shell?

The listed Home page of
http://www.xs4all.nl/~jantien/yodl/
says

    Yodl has been discontinued

    If you were looking for yodl, you're out of luck. Yodl has been
    discontinued, you are advised to use texinfo.

    We have dropped yodl more than four years ago*), and it has been
    unmaintained since.

    *) Sep 3 1999 lilypond-1.2.6.tar.gz


The NetBSD pkgsrc Makefile contains

    MASTER_SITES=   ftp://ftp.lilypond.org/pub/yodl/development/

but when I try to connect, I get
Unknown host ftp.lilypond.org

I did find what appears to be the new home page on SourceForge at
http://yodl.sourceforge.net/
which only has a new version, yodl_2.01.01.tar.gz.

However, NetBSD does not have that version in it's pkgsrc ports.

Out of curiosity, I unpacked it and checked the docs.  There is no
Makefile and no configure script.  Turns out it has a build script
that uses icmake.  icmake is also not in the NetBSD pkgsrc tree.
Having never heard of icmake, I researched further. I discovered
icmake was originally developed for MS-DOS platforms and it's makefile
uses nearly full C language syntax.  Seems like it defeats it's own
purpose to have to write a full C program to handle your build
process.

Also, the docs in the new one don't seem to make any reference to the
current status or why the original home page says it has been
discontinued.

Re: localtraps

[Bart Schaefer]

On May 3, 11:04am, Peter Stephenson wrote:
} Subject: Re: localtraps
}
} > arget  ;; esac;  test -f $target
} > *** Error code 1
} 
} It's not obvious why it gave an error --- unless the wrap between "$t"
} and "arget" was there in the original, which is unlikely.

It gave an error because "test -f $target" failed, because the target
does not exist.  This is a result of a change I recently committed.
Tell me again why we should create and test for an empty file in the
event tht yodl doesn't exist?

Seems to me the test is wrong even in the case where yodl *does* exist,
since the file is created by a redirection and therefore it will be
present (and empty) even if yodl fails outright.

Re: yodl and Z shell documentation (was localtraps)

[Peter Stephenson]

Vincent Stemen wrote:
> I am unclear about the status and future of yodl.  I am curious, is
> this the documentation format planned for the foreseeable future of Z
> shell?

We (for some values of we, at least) would quite like to migrate to
something XML-based such as Docbook-XML, but that will be a lot of work,
so we're stuck with Yodl till then.  It is a bit past its sell-by date.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: localtraps

[Peter Stephenson]

Bart Schaefer wrote:
> It gave an error because "test -f $target" failed, because the target
> does not exist.  This is a result of a change I recently committed.
> Tell me again why we should create and test for an empty file in the
> event tht yodl doesn't exist?

No idea.  Probably the code originally tested for something completely
different, e.g. there wasn't a redirection or there was a cp, or whatever.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Compiling without yodl (Re: localtraps)

[Bart Schaefer]

On May 4, 10:28am, Peter Stephenson wrote:
} Subject: Re: localtraps
}
} Bart Schaefer wrote:
} > It gave an error because "test -f $target" failed, because the target
} > does not exist.  This is a result of a change I recently committed.

I take that back -- the last part about the change I committed, that is.
Vincent reported the error for the 4.2.x patches branch, but my change
was committed only on the head revision.

} > Tell me again why we should create and test for an empty file in the
} > event tht yodl doesn't exist?
} 
} No idea.  Probably the code originally tested for something completely
} different, e.g. there wasn't a redirection or there was a cp, or whatever.

May I suggest:

Index: Doc/Makefile.in
===================================================================
diff -c -r1.14 Makefile.in
--- Doc/Makefile.in	18 Feb 2005 17:05:15 -0000	1.14
+++ Doc/Makefile.in	4 May 2005 14:21:14 -0000
@@ -114,11 +114,10 @@
 	  */*) target=$@ ;; \
 	  *) target=$(sdir)/$@ ;; \
 	esac; \
-	case '$(YODL)' in :*) ;; *) \
+	case '$(YODL)' in :*) touch $$target ;; *) \
 	    echo $(YODL) -o $$target -I$(sdir) -w zman.yo version.yo $< ; \
 	    $(YODL) -I$(sdir) -w zman.yo version.yo $< | sed -e '1s/\\-/-/g' -e '/^\.'\''/d' > $$target \
 	;; esac; \
-	test -f $$target
 
 ps: us_ps a4_ps
 .PHONY: ps
@@ -146,17 +145,15 @@
 	  */*) target=$@ ;; \
 	  *) target=$(sdir)/$@ ;; \
 	esac; \
-	case '$(YODL)' in :*) ;; *) \
+	case '$(YODL)' in :*) touch $$target ;; *) \
 	    echo $(YODL) -o $$target -I$(sdir) -DZSHALL -w zman.yo version.yo zsh.yo; \
 	    $(YODL) -I$(sdir) -DZSHALL -w zman.yo version.yo zsh.yo | sed -e '1s/\\-/-/g' -e '/^\.'\''/d' > $$target \
 	;; esac; \
-	test -f $$target
 
 ../META-FAQ: META-FAQ.yo Zsh/metafaq.yo
-	case '$(YODL)' in :*) ;; *) \
+	case '$(YODL)' in :*) touch $(sdir_top)/META-FAQ ;; *) \
 	    $(YODL) -I$(sdir) META-FAQ.yo | sed -e '/NEXTLINE/N' -e '/DELLINE/d' -e '/^SECTHEAD$$/{N;s/^SECTHEAD.//;h;s/./-/g;H;g;}' -e 's/  *$$//' > $(sdir_top)/META-FAQ \
 	;; esac
-	test -f $(sdir_top)/META-FAQ
 
 $(YODLDOC): version.yo
 

Re: yodl and Z shell documentation (was localtraps)

[Oliver Kiddle]

On 4 May, Peter wrote:
> We (for some values of we, at least) would quite like to migrate to
> something XML-based such as Docbook-XML, but that will be a lot of work,
> so we're stuck with Yodl till then.  It is a bit past its sell-by date.

makeinfo can produce XML output from our .texi file which is a starting
point. The result needs a lot of reworking and adjusting, some of which
can probably be automated using XSLT. At the moment, makeinfo produces
invalid XML because of a subsection in the middle of an item in
mod_socket.yo. I'll apply the following patch so I only need to fix this
once.

I'm not quite sure how easy it'll be to produce the same set of man
pages as we currently have from docbook. The change may also be a step
backwards for anyone using info documentation.

Oliver

Index: mod_socket.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/mod_socket.yo,v
retrieving revision 1.4
diff -u -r1.4 mod_socket.yo
--- mod_socket.yo       22 May 2003 09:48:28 -0000      1.4
+++ mod_socket.yo       10 May 2005 09:34:54 -0000
@@ -10,6 +10,8 @@
 item(tt(zsocket) [ tt(-altv) ] [ tt(-d) var(fd) ] [ var(args) ])(
 tt(zsocket) is implemented as a builtin to allow full use of shell
 command line editing, file I/O, and job control mechanisms.
+)
+enditem()
 
 subsect(Outbound Connections)
 cindex(sockets, outbound Unix domain)
@@ -62,5 +64,3 @@
 In order to elicit more verbose output, use tt(-v).
 )
 enditem()
-)
-enditem()

Re: yodl and Z shell documentation (was localtraps)

[Bart Schaefer]

On May 10, 11:45am, Oliver Kiddle wrote:
} Subject: Re: yodl and Z shell documentation (was localtraps)
}
} > something XML-based such as Docbook-XML, but that will be a lot of work
} 
} I'm not quite sure how easy it'll be to produce the same set of man
} pages as we currently have from docbook. The change may also be a step
} backwards for anyone using info documentation.

I definitely do not want to lose the info documentation.  I use it a lot,
particularly the index lookup feature.

Re: yodl and Z shell documentation (was localtraps)

[Oliver Kiddle]

Bart wrote:

> I definitely do not want to lose the info documentation.  I use it a lot,
> particularly the index lookup feature.

We wouldn't lose it but the conversion may not be as smooth. I think the
easiest way to turn docbook into Texinfo is using docbook2x
(http://docbook2x.sourceforge.net/). I don't know much about it,
however. Do docbook2x's own info files look good to you?

Things like indexes may have to be done differently. I don't really like
the current set of six indices anyway.

docbook2x also converts to man but I suspect it may be better to use
db2man.xsl even if that requires a bit of extra XSL to stuff relevant
things inside <manualpage>.

Oliver

Re: yodl and Z shell documentation (was localtraps)

[Bart Schaefer]

On May 10,  4:42pm, Oliver Kiddle wrote:
} Subject: Re: yodl and Z shell documentation (was localtraps)
}
} Bart wrote:
} 
} > I definitely do not want to lose the info documentation.  I use it a lot,
} > particularly the index lookup feature.
} 
} We wouldn't lose it but the conversion may not be as smooth. I think the
} easiest way to turn docbook into Texinfo is using docbook2x

This is not increasing my enthusiasm.  The whole reason we chose yodl
way back when was because that meant *one* piece of format-conversion
software was required to produce all the output formats we wanted.
Requiring a greater number of conversion tools from a wider variety of 
sources and that may be available on a smaller number of target OSs, is
not a step forward in my opinion.

} (http://docbook2x.sourceforge.net/). I don't know much about it,
} however. Do docbook2x's own info files look good to you?

It looks OK based on the Cygwin screen shot, but I'm not going to
install it just to find out.

} Things like indexes may have to be done differently. I don't really like
} the current set of six indices anyway.

Well, no, neither do I, but that's a limitation of the info format.

Re: localtraps and signal handling on NetBSD

[Vincent Stemen]

On Mon, May 09, 2005 at 10:32:28AM +0100, Peter Stephenson wrote:
> Vincent Stemen wrote:
> > I was just wondering if you are planning to do any further work on the
> > signal/localtrap code.
> 
> As far as I've followed the problems, they're mostly system specific.
> I'm sure there's more that can be done but I'm not really clear what.
> I'm not a great expert on signals anyway (traps are a  little
> different).
> 
> pws

Ok.  For now, I found a not so clean work around for the code I am
working on, that would not necessarily work in other situations.  

If you come up with any other ideas to try, I am willing to test for
you and assist in any way I can.

I did notice in the zsh source that it sets an intrap flag once a
signal has been received, which apparently causes it to treat the next
signal special by queuing it and waiting until it exits the signal
handler before processing it.

I have an idea, that might not be to difficult to implement, that I
think would solve the problem, unless you know of other problems it
would create.

If, inside the signal handler, I execute the trap statement on the
same signal, turn off the intrap status because the signal is now
re-trapped.

I don't know the zsh source well enough to be sure where to make that
change, but from initial browsing, it looks like it might be easy to
do.

Does that sound like something you would be willing to try, and send
me a patch to test on NetBSD?

-- 
Vincent Stemen
Avoid the VeriSign/Network Solutions domain registration trap!
Read how Network Solutions (NSI) was involved in stealing our domain name.
http://inetaddresses.net/about_NSI.html

Re: yodl and Z shell documentation (was localtraps)

[Oliver Kiddle]

Bart wrote:

> This is not increasing my enthusiasm.  The whole reason we chose yodl
> way back when was because that meant *one* piece of format-conversion
> software was required to produce all the output formats we wanted.

If you consider that we currently need texi2html to produce HTML and
pdfetex to produce PDF (no to mention a full TeX installation), the
current situation doesn't quite live up to that. 

> Requiring a greater number of conversion tools from a wider variety of 
> sources and that may be available on a smaller number of target OSs, is
> not a step forward in my opinion.

Well docbook would be a single source and XSL-FO our only intermediate
source. Docbook as a format is largely independent of the actual tools.
There's a defacto standard set of XSL stylesheets for producing HTML,
man, plain text (I think) and XSL-FO. But you've got a choice of XSL
translator (xsltproc, saxon, xalan etc). For PDFs there's again a choice
for converting XSL-FO (fop (java), PassiveTex and xmlroff). So as far as
target OSs go, we'd be in a far better position than with yodl.

If we do go through with the transition, you'll probably find the main
disadvantage is that docbook source and XML is somewhat more verbose
than yodl.

Oliver

Re: yodl and Z shell documentation (was localtraps)

[Bart Schaefer]

On May 11, 11:59am, Oliver Kiddle wrote:
}
} If we do go through with the transition, you'll probably find the main
} disadvantage is that docbook source and XML is somewhat more verbose
} than yodl.

As, say, China is somewhat more populous than Australia.

Re: yodl and Z shell documentation (was localtraps)

[Clint Adams]

[-- Attachment #1: Type: text/plain, Size: 183 bytes --]

> As, say, China is somewhat more populous than Australia.

I'm going to attach my incomplete DocBook version of the FAQ
before I lose it in some catastrophic disk and power failure.

[-- Attachment #2: FAQ.xml --]
[-- Type: application/xml, Size: 98485 bytes --]

Re: localtraps and signal handling on NetBSD

[Peter Stephenson]

Vincent Stemen wrote:
> I have an idea, that might not be to difficult to implement, that I
> think would solve the problem, unless you know of other problems it
> would create.
> 
> If, inside the signal handler, I execute the trap statement on the
> same signal, turn off the intrap status because the signal is now
> re-trapped.

I don't think that's the problem.  intrap has a quite limited effect,
determining how the shell behaves when already handling signals.

The vague suspicion is that the problem is based on whether or not the
trap handler is run inside the signal handler, and whether the system
resets the signal.  It's possible to tweak the queueing code, which
would force more traps to be run after the handler exits rather than
within it, but I'm really not sure enough about what's going on to
suggest anything definite.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: localtraps and signal handling on NetBSD

[Vincent Stemen]

On Fri, May 13, 2005 at 12:26:59PM +0100, Peter Stephenson wrote:
> Vincent Stemen wrote:
> > I have an idea, that might not be to difficult to implement, that I
> > think would solve the problem, unless you know of other problems it
> > would create.
> > 
> > If, inside the signal handler, I execute the trap statement on the
> > same signal, turn off the intrap status because the signal is now
> > re-trapped.
> 
> I don't think that's the problem.  intrap has a quite limited effect,
> determining how the shell behaves when already handling signals.
> 
> The vague suspicion is that the problem is based on whether or not the
> trap handler is run inside the signal handler, and whether the system
> resets the signal.  It's possible to tweak the queueing code, which
> would force more traps to be run after the handler exits rather than
> within it, but I'm really not sure enough about what's going on to
> suggest anything definite.

I am not sure I am following you correctly.  If, by handler, you are
referring to the shell script function signal handler, I don't think
queuing more signals to run after the handler exits will help.  The
fact that it waits until the handler returns before responding to the
next signal is the problem. Or are you referring to a handler in the
zsh source?

I did some more detailed multi-platform testing, that did not involve
Z shell's localtraps, to compare the behavior of different shells on
different platforms.  In this particular test, I focused on just the
handling of the same signal once you are inside a signal handler.  I
thought I would post the results in the hope that it might help.  I
used my machines and various machines available through the
SourceForge.net compile farm.  I got the same behavior on every
platform, including Linux.

Here is my test script:

# --------- <test script> -----------
sigterm1()
{
    trap 'echo "-- sigterm2 --"' TERM
    echo "sigterm1(): sending SIGTERM"
    kill -TERM $$
    trap sigterm1 TERM
    sleep 1
}

trap sigterm1 TERM

echo
echo "main: sending SIGTERM"
kill -TERM $$
echo "main: sending SIGTERM"
kill -TERM $$
# --------- </test script> -----------


Since I got the same result on every platform, we are not dealing with
any OS specific issues in this particular case.

Here are the shells on various platforms that I tested with that
correctly handled the signals and produced the output below.

PD KSH v5.2.14 99/07/13.2 on NetBSD 2.0
PD KSH v5.2.14 99/07/13.2 on Linux 2.4.21-27.0.1.ELsmp
ksh on SunOS x86-solaris1 5.9
ash on Linux 2.4.21-27.0.1.ELsmp
sh on NetBSD 2.0
sh on NetBSD 1.6.1
sh on FreeBSD 4.10-BETA
sh on OpenBSD 3.4
sh on SunOS x86-solaris1 5.9

<correct behavior>
main: sending SIGTERM
sigterm1(): sending SIGTERM
-- sigterm2 --
main: sending SIGTERM
sigterm1(): sending SIGTERM
-- sigterm2 --
</correct behavior>


The summary of the results is that the only two shells that do not
handle the signals correctly once in the handler are bash and zsh on
all platforms.  sh on Solaris and all the BSD's, ksh, and ash
(commonly available on Linux systems) behaved properly.

Here is the output of zsh and bash.

<zsh>
main: sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
sigterm1(): sending SIGTERM
... continues forever
</zsh>

I believe we thought that the signals were being handled differently
in zsh on Linux than BSD earlier, but in this test that does not
appear to be the case.  In all cases zsh does not process the next
signal until it exits the signal handler, so the trap following the
kill command resets the signal before the next signal is processed,
causing it to loop endlessly. I also got the same result on the
patched version of zsh 4.2.5 that we were testing with earlier.

<bash>
main: sending SIGTERM
sigterm1(): sending SIGTERM
main: sending SIGTERM
sigterm1(): sending SIGTERM
</bash>

The problem with bash is that it disables further signals all together
once it is in the signal handler and the trap statement does not
re-enable them.  Either problem can be a show stopper.


-- 
Vincent Stemen
Avoid the VeriSign/Network Solutions domain registration trap!
Read how Network Solutions (NSI) was involved in stealing our domain name.
http://inetaddresses.net/about_NSI.html

Re: localtraps and signal handling on NetBSD

[Peter Stephenson]

Vincent Stemen wrote:
> # --------- <test script> -----------
> sigterm1()
> {
>     trap 'echo "-- sigterm2 --"' TERM
>     echo "sigterm1(): sending SIGTERM"
>     kill -TERM $$
>     trap sigterm1 TERM
>     sleep 1
> }
> 
> trap sigterm1 TERM
> 
> echo
> echo "main: sending SIGTERM"
> kill -TERM $$
> echo "main: sending SIGTERM"
> kill -TERM $$
> # --------- </test script> -----------
>
> Here is the output of zsh and bash.
> 
> <zsh>
> main: sending SIGTERM
> sigterm1(): sending SIGTERM
> sigterm1(): sending SIGTERM
> sigterm1(): sending SIGTERM
> sigterm1(): sending SIGTERM
> ... continues forever
> </zsh>

Yes, I can see this, but this isn't the same example I was looking at
when I saw different behaviour from you.

What I think is happening here is that signals *are* being queued.  Then
the SIGTERM is delivered after sigterm1 has returned, at which point
sigterm1 has been reinstalled as the signal handler, so gets called
again ad infinitum.

In this case bash and zsh are actually trying a bit harder to do the
right thing, at least as far as safety of the execution environment is
concerned.  Not queueing signals can cause significant reentrancy
problems.  However, it may be possible to unqueue signals temporarily at
some point.  That's still not trivial; the code's currently not written
to allow that, since as I said before the queue/unqueues are all nested
and you need to have some clue about the environment to decide whether
temporarily unqueuing is safe.  Possibly some queue/unqueue pair can be
moved deeper into the code, leaving some points at which signals will be
handled, but that's not trivial either.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: Z shell signal handling

[Vincent Stemen]

On Mon, May 16, 2005 at 11:46:49AM +0100, Peter Stephenson wrote:
> Vincent Stemen wrote:
> > # --------- <test script> -----------
> > sigterm1()
> > {
> >     trap 'echo "-- sigterm2 --"' TERM
> >     echo "sigterm1(): sending SIGTERM"
> >     kill -TERM $$
> >     trap sigterm1 TERM
> >     sleep 1
> > }
> > 
> > trap sigterm1 TERM
> > 
> > echo
> > echo "main: sending SIGTERM"
> > kill -TERM $$
> > echo "main: sending SIGTERM"
> > kill -TERM $$
> > # --------- </test script> -----------
> >
> > Here is the output of zsh and bash.
> > 
> > <zsh>
> > main: sending SIGTERM
> > sigterm1(): sending SIGTERM
> > sigterm1(): sending SIGTERM
> > sigterm1(): sending SIGTERM
> > sigterm1(): sending SIGTERM
> > ... continues forever
> > </zsh>
> 
> Yes, I can see this, but this isn't the same example I was looking at
> when I saw different behaviour from you.

Yes, I think that is because we were discussing localtraps earlier as
well, but I thought it was better to simplify the test case as much as
possible and set any localtrap issues aside for now and focus on the
fundamental signal behavior that can be directly compared to other
shells.

> What I think is happening here is that signals *are* being queued.  Then
> the SIGTERM is delivered after sigterm1 has returned, at which point
> sigterm1 has been reinstalled as the signal handler, so gets called
> again ad infinitum.

Yes.  That is the problem basically as I see it.


> In this case bash and zsh are actually trying a bit harder to do the
> right thing, at least as far as safety of the execution environment is
> concerned.  Not queueing signals can cause significant reentrancy
> problems.  However, it may be possible to unqueue signals temporarily at
> some point.  That's still not trivial; the code's currently not written
> to allow that, since as I said before the queue/unqueues are all nested
> and you need to have some clue about the environment to decide whether
> temporarily unqueuing is safe.  Possibly some queue/unqueue pair can be
> moved deeper into the code, leaving some points at which signals will be
> handled, but that's not trivial either.

I have a proposal as to the way I think the signals should be handled,
although, as you say, it may not be trivial.  First, let me provide
one more example to elaborate on the current bahavior comparing zsh to
sh.  There is a problem with the way sh does it also, which I will
describe below.

# --- <test script> ---
signal()
{
  echo "-- signal() --"
  echo "sleeping"
  sleep 1
  echo "sleeping again"
  sleep 1
}

trap signal INT
echo "main: sending SIGINT"
kill -INT $$
# --- </test script> ---

To test, I hit ^C several times while it is in signal().

<zsh>
main: sending SIGINT
-- signal() --
sleeping
^C^C^C^Csleeping again
^C-- signal() --
sleeping
sleeping again
</zsh>


<sh>
main: sending SIGINT
-- signal() --
sleeping
^C-- signal() --
sleeping
^C-- signal() --
sleeping
^C-- signal() --
sleeping
sleeping again
sleeping again
sleeping again
sleeping again
</sh>

zsh seems to queue one signal and re-calls the handler once.  All
additional signals are ignored.  I am not sure I see the benefit of
this.

sh would appear to queue many signals, but actually it is not having
to queue them because it calls the handler immediately on each one.
So, I don't know if it even has any kind of queuing mechanism.

There are a couple potential problems I can see with the sh approach,
that I am guessing zsh and bash were trying to fix.
  1. It enters the handler with the signal still enabled, so the programmer
     needs to know to immediately use trap to disable the signal, once
     in the handler, if he does not want get hit with another signal
     before exiting the handler.

  2. Even if you disable the signal in the handler, there might be the
     remote possibility of a race case where it could catch another
     signal before getting it disabled.  If you are talking about
     dealing with interrupts at the kernel level, I would certainly
     consider this an issue.  But with signals in shell script, I
     would generally consider the chance of this very low or near
     zero.  I don't know of many things that would generate multiple
     signals that close together.  The user certainly is not going to
     be able to hit ^C or type the kill command that fast.  Maybe if
     you had multiple other processes sending signals to the same
     process and two of them happen to hit at near the same time.

     This leaves problem 1 as the primary issue.

Here is my proposal of the way I think it should be handled, which is
very simple conceptually.  Although I understand, Peter, that
implementing it might not be so simple.  I will, of course, leave that
up to you or whoever might have the time, ability, and hopefully
willingness to work on it to determine.

Proposal:

When a signal is caught, enter the handler with the signal
automatically disabled the way bash does.  Then allow the programmer
to re-enable the signal with a trap statement inside the handler if
needed.  If another signal is caught after that, process it
immediately like sh does, rather than waiting for the handler to
return.

This way, there would be no potential race cases, programmers won't
get themselves into trouble by not thinking of disabling the signal in
the handler, and our hands won't be tied when we need to do more
sophisticated signal handling.

It seems to me that would solve all the problems and possibly
eliminate the need to have complex signal queuing code.  I not sure
you would need to do any queuing at all since all signals would be
processed immediately.


-- 
Vincent Stemen
Avoid the VeriSign/Network Solutions domain registration trap!
Read how Network Solutions (NSI) was involved in stealing our domain name.
http://inetaddresses.net/about_NSI.html

Re: Z shell signal handling

[Bart Schaefer]

On May 17,  3:15am, Vincent Stemen wrote:
} Subject: Re: Z shell signal handling
}
} There are a couple potential problems I can see with the sh approach,
} that I am guessing zsh and bash were trying to fix.

The biggest problem is that C library routines like malloc() are not
re-entrant, and are heavily used by the shell.  The issue is not with
race conditions, etc., at the level of shell function execution, but
deep within the shell implementation, possibly in places that are not
within the shell implementer's control.

When PWS speaks about whether it's "safe" to enable signals, he means
that if it's NOT safe, the shell is going to corrupt memory, possibly
crash, or even open holes in system security.  Behaving the way that
the script programmer expects is secondary to these considerations.

Re: 4.3.1 released

[Peter Stephenson]

Vincent Lefevre wrote:
> I get a strange configure error:
> 
> [...]
> checking if the system adequately supports multibyte chars... no
> ./configure: line 15889: test: =: unary operator expected
> checking if your system uses ELF binaries... no
> [...]

Annoying, but should be harmless:  the behaviour if test failed is the
same as the default one, apart from the error message.

> The configure script contains:
> 
> # Check whether --enable-dynamic-nss or --disable-dynamic-nss was given.
> if test "${enable_dynamic_nss+set}" = set; then
>   enableval="$enable_dynamic_nss"
>   zsh_cv_c_dynamic_nss=$enableval
> fi;
> 
> Line 15889 is zsh_cv_c_dynamic_nss=$enableval.

I suspect it's actually the following (though the line number doesn't
agree):

Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.49
diff -u -r1.49 configure.ac
--- configure.ac	19 Feb 2006 19:36:32 -0000	1.49
+++ configure.ac	28 Feb 2006 14:16:53 -0000
@@ -2112,7 +2112,7 @@
 AH_TEMPLATE([DISABLE_DYNAMIC_NSS],
 [Define to 1 if you want to avoid calling functions that will require
  dynamic NSS modules.])
-if test $zsh_cv_c_dynamic_nss = no; then
+if test x$zsh_cv_c_dynamic_nss = xno; then
   AC_DEFINE(DISABLE_DYNAMIC_NSS)
 fi
 

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: 4.3.1 released

[Vincent Lefevre]

On 2006-02-28 14:19:56 +0000, Peter Stephenson wrote:
> Vincent Lefevre wrote:
> > I get a strange configure error:
> > 
> > [...]
> > checking if the system adequately supports multibyte chars... no
> > ./configure: line 15889: test: =: unary operator expected
> > checking if your system uses ELF binaries... no
> > [...]
> 
> Annoying, but should be harmless:  the behaviour if test failed is the
> same as the default one, apart from the error message.

Is it still harmless for the tests using "!="?

  perl -pe 's/(test|-o|-a) +(\$\S+ +!?= +)/\1 x\2x/g' configure.ac

may give a global fix for the configure.ac file.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: 4.3.1 released

[Peter Stephenson]

Vincent Lefevre wrote:
> On 2006-02-28 14:19:56 +0000, Peter Stephenson wrote:
> > Vincent Lefevre wrote:
> > > I get a strange configure error:
> > > 
> > > [...]
> > > checking if the system adequately supports multibyte chars... no
> > > ./configure: line 15889: test: =: unary operator expected
> > > checking if your system uses ELF binaries... no
> > > [...]
> > 
> > Annoying, but should be harmless:  the behaviour if test failed is the
> > same as the default one, apart from the error message.
> 
> Is it still harmless for the tests using "!="?
> 
>   perl -pe 's/(test|-o|-a) +(\$\S+ +!?= +)/\1 x\2x/g' configure.ac
> 
> may give a global fix for the configure.ac file.

Although it would certainly be sensible to be safe, the existing tests
have tended to evolve so that any time an unquoted variable appears the
variable has been set (this one is a recent addition which I failed to
spot).  It would be a good idea to fix all these (particularly so that
it encourages people to write safe tests in future, avoiding the current
problem) but I don't think it's necessary for an immediately release.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: 4.3.1 released

[Bart Schaefer]

On Feb 28,  1:06pm, Peter Stephenson wrote:
}
} Please try this out on as many systems as possible as soon as you get a
} chance; if there are any significant problems I'll produce a 4.3.2.

New PC at home, so new OS (CentOS 4.2 =~ RedHat EL4u2) and new compiler
and consequently new warning messages (new to me, at least).  Appended
below.  However, the "./configure && make && make test" all otherwise
went without a hitch, and it seems to run just fine.


exec.c: In function `addvars':
exec.c:1695: warning: dereferencing type-punned pointer will break strict-aliasing rules
exec.c: In function `execcmd':
exec.c:1941: warning: dereferencing type-punned pointer will break strict-aliasing rules
glob.c: In function `xpandredir':
glob.c:1797: warning: dereferencing type-punned pointer will break strict-aliasing rules
subst.c: In function `singsub':
subst.c:288: warning: dereferencing type-punned pointer will break strict-aliasing rules
subst.c: In function `multsub':
subst.c:334: warning: dereferencing type-punned pointer will break strict-aliasing rules
subst.c: In function `paramsubst':
subst.c:2716: warning: dereferencing type-punned pointer will break strict-aliasing rules
utils.c: In function `finddir':
utils.c:653: warning: dereferencing type-punned pointer will break strict-aliasing rules
mapfile.c: In function `scanpmmapfile':
mapfile.c:313: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmparameters':
parameter.c:175: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmcommands':
parameter.c:305: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanfunctions':
parameter.c:514: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanbuiltins':
parameter.c:619: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmoptions':
parameter.c:781: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmmodules':
parameter.c:898: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c:908: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c:915: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c:923: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmhistory':
parameter.c:1016: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmjobtexts':
parameter.c:1127: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmjobstates':
parameter.c:1224: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmjobdirs':
parameter.c:1286: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmnameddirs':
parameter.c:1411: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanpmuserdirs':
parameter.c:1464: warning: dereferencing type-punned pointer will break strict-aliasing rules
parameter.c: In function `scanaliases':
parameter.c:1755: warning: dereferencing type-punned pointer will break strict-aliasing rules
zleparameter.c: In function `scanpmwidgets':
zleparameter.c:135: warning: dereferencing type-punned pointer will break strict-aliasing rules

Re: 4.3.1 released

[Peter Stephenson]

Bart Schaefer wrote:
> New PC at home, so new OS (CentOS 4.2 =~ RedHat EL4u2) and new compiler
> and consequently new warning messages (new to me, at least).  Appended
> below.  However, the "./configure && make && make test" all otherwise
> went without a hitch, and it seems to run just fine.
> 
> exec.c: In function `addvars':
> exec.c:1695: warning: dereferencing type-punned pointer will break strict-ali
> asing rules
[etc.]

We've been getting these from gcc 3+ for a while.  I think Wayne
suggested some casts to void * would be enough to remove them.  However, I
for one didn't really understand the implications enough.  It would be
good if anyone could put there hand on their heart and say that was
the right way to go (or if not, what was).

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page still at http://www.pwstephenson.fsnet.co.uk/

Re: 4.3.1 released

[Peter Stephenson]

Vincent Lefevre <vincent@vinc17.org> wrote:
>   perl -pe 's/(test|-o|-a) +(\$\S+ +!?= +)/\1 x\2x/g' configure.ac
> 
> may give a global fix for the configure.ac file.

I've committed a fix along these lines to the archive.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: "type punned" warnings

[Wayne Davison]

On Tue, Feb 28, 2006 at 11:51:52PM +0000, Peter Stephenson wrote:
> We've been getting these from gcc 3+ for a while.  I think Wayne
> suggested some casts to void * would be enough to remove them.

Yeah, that was a pretty ugly kluge of dubious usefulness.

I've created a patch that fixes the aliasing at a fundamental level.
It's really big, so I'll provide a link instead of attaching it:

    http://opencoder.net/punning.diff

There were two classes of warnings:

 1. Multiple structure pointers were being cast to a HashNode (struct
    hashnode *).

 2. A LinkList (struct linklist *) was being cast to a LinkNode (struct
    linknode *).

I fixed the first class of warnings by putting a "struct hashnode node;"
at the start of every structure that was getting cast to HashNode.  This
required sprinkling some "node." prefixes around the code for the three
affected structure variables (next, nam, and flags).

I fixed the second class of warnings by making a LinkList a union of a
"struct linklist list;" and a "struct linknode node;".  This tells the
compiler that the type punning between the two pointer classes needs to
be taken into account when optimizing.  This required sprinkling some
"list." prefixes around the code and referring to the "node" when
appropriate.  I also added an "int flag" to the linklist structure,
since it now fits in the union (which an uncommitted patch of mine will
like having around).

I also think that it would be nice to rename the "last" variable in the
LinkNode structure to be "prev", which is a clearer name to me.  I
didn't do that (at least, not yet) because it would just clutter up an
already huge patch.

..wayne..

Re: "type punned" warnings

[Peter Stephenson]

Wayne Davison wrote:
> I fixed the first class of warnings by putting a "struct hashnode node;"
> at the start of every structure that was getting cast to HashNode.
> 
> I fixed the second class of warnings by making a LinkList a union of a
> "struct linklist list;" and a "struct linknode node;".

That's much better.  (Actually, I have something similar in some data
handling code I'm responsible for, and I keep gnashing my teeth I can't
make the base class a real base class, but this is fine.)

> I also think that it would be nice to rename the "last" variable in the
> LinkNode structure to be "prev", which is a clearer name to me.

Yes, that confused me for a long time.

You could also check in the patch to add flags to nodes on the way
through paramsubst(), which finally starts making multsub() look less of
a disaster.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page still at http://www.pwstephenson.fsnet.co.uk/

Re: "type punned" warnings

[Peter Stephenson]

Wayne Davison wrote:
> I've created a patch that fixes the aliasing at a fundamental level.

Looks like there's something missing in the committed version.

gcc -c -I. -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -DHAVE_CONFIG_H -O2 -Wall -Wno-implicit -Wmissing-prototypes -ggdb  -o builtin.o builtin.c
builtin.c: In function  `typeset_single':
builtin.c:2145: error:  `struct param' has no member named `flags'
builtin.c:2145: error:  `struct param' has no member named `flags'

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: "type punned" warnings

[Peter Stephenson]

Peter Stephenson wrote:
> gcc -c -I. -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -DHAVE_CONFIG_H -O2 -W
> all -Wno-implicit -Wmissing-prototypes -ggdb  -o builtin.o builtin.c
> builtin.c: In function  `typeset_single':
> builtin.c:2145: error:  `struct param' has no member named `flags'
> builtin.c:2145: error:  `struct param' has no member named `flags'

It was just in a single DPUTS().

Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.155
diff -u -r1.155 builtin.c
--- Src/builtin.c	7 Mar 2006 21:30:36 -0000	1.155
+++ Src/builtin.c	8 Mar 2006 15:46:52 -0000
@@ -2142,7 +2142,7 @@
 	if (!(pm = setsparam(pname, ztrdup(value))))
 	    return NULL;
 	if (pm != ipm) {
-	    DPUTS(ipm->flags != pm->flags,
+	    DPUTS(ipm->node.flags != pm->node.flags,
 		  "BUG: parameter recreated with wrong flags");
 	    unsetparam_pm(ipm, 0, 1);
 	}

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Vanishing files ?

[Peter Stephenson]

Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> Anssi Saari wrote:
> > bash: ./foo: /bin/foo: bad interpreter: No such file or directory
> 
> This turns out to be easy; the function already does path look up and
> even has code to search for the interpreter (to emulate #! on older
> systems where it wasn't implemented natively).  It relies on the error
> being ENOENT (no such file or directory), not ENOEXEC (exec format
> error); this is the case on all the systems I know about (haven't check
> any relevant standards).

Unfortunately this doesn't work very well in a case like the one
we have where the interpreter name is strange, since the \r is
interpreted literally and you end up with something like

: no such file or directorybad interpreter: /bin/sh

To fix this properly we need to be able to display two properly formatted
strings in an error message: it's many, many years past the point where
error messages should have used variable-length argument lists.  This would
be a nice, self-contained project for someone; we can probably assume
stdarg is available now (since we seem to assume a lot of ISO C elsewhere).

Until someone fixes that, the less bad alternative is probably to assume
that the interpreter name is more likely to need special handling than the
script name.

I note that nicechar() and wcs_nicechar() print \n as \n, but \r as ^M.
This isn't too bad, but it's a little inconsistent.  We should probably
handle at least the standard C codes, unless anyone can see why we
shouldn't.  Anyway, until then, you get things like

zsh: /home/pws/tmp/noexec: bad interpreter: /bin/sh^M: no such file or
directory

(wrapped the line myself to avoid giving the pleasure to our mail
software).

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.99
diff -u -r1.99 exec.c
--- Src/exec.c	29 May 2006 15:36:26 -0000	1.99
+++ Src/exec.c	30 May 2006 17:37:45 -0000
@@ -409,9 +409,12 @@
 			    char *buf;
 			    if (*ptr)
 				*ptr = '\0';
-			    buf = tricat("%s: bad interpreter: ", ptr2,
-					 ": %e");
-			    zerr(buf, pth, eno);
+			    /*
+			     * TODO: needs variable argument handling
+			     * in zerrmsg() etc. to do this properly.
+			     */
+			    buf = dyncat(pth, ": bad interpreter: %s: %e");
+			    zerr(buf, ptr2, eno);
 			} else if (*ptr) {
 			    *ptr = '\0';
 			    argv[-2] = ptr2;


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Vanishing files ?

[Peter Stephenson]

Peter Stephenson wrote:
> To fix this properly we need to be able to display two properly formatted
> strings in an error message: it's many, many years past the point where
> error messages should have used variable-length argument lists.  This would
> be a nice, self-contained project for someone; we can probably assume
> stdarg is available now (since we seem to assume a lot of ISO C elsewhere).

OK, guess what...  This was not the most interesting hour and a half of
my life.  I won't bore you with the patch, instead check it straight in;
see changes in utils.c for the important bit.  It's possible I've
mistyped something, so please create as many errors as possible to
check...

If anyone wants to supplement this with varargs support, be my guest.
gcc 4 doesn't even support it any more, but it would extend the backward
compatibility back a bit further.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Vanishing files ?

[Wayne Davison]

On Tue, May 30, 2006 at 11:29:25PM +0100, Peter Stephenson wrote:
> I won't bore you with the patch, instead check it straight in;
> see changes in utils.c for the important bit.

Cool!  Things are looking good so far.

> If anyone wants to supplement this with varargs support, be my guest.

Hopefully that won't be necessary.  I know that rsync has just used
stdarg for its variable-arg functions, and I haven't seen anyone
complain about it yet.

..wayne..

Re: Vanishing files ?

[Bart Schaefer]

On May 30, 11:29pm, Peter Stephenson wrote:
}
} see changes in utils.c for the important bit.  It's possible I've
} mistyped something, so please create as many errors as possible to
} check...

Hmm, do we need a ztst file that does nothing but cause errors?
 
} If anyone wants to supplement this with varargs support, be my guest.

Ordinarily this would be my sort of thing, but I don't think I have a
sufficiently old compiler around any more.  I'm a bit concerned that,
we having recommended as of March (see workers/22638) that IRIX 6.5
should compile with the old "cc" rather than "c99", this will now have
broken the build under "cc" as well.

Recursion error and line numbers

[Bart Schaefer]

This is tangential to the segfault etc. reported for

local foo=(); local bar

In workers/22495 I wrote:

> zsh% local
> local:10: maximum nested function level reached
> zsh% 


There are not 10 lines in the function "local", nor are there 50 lines
when I try the "compinit" test and mysteriously get the error twice:

zsh% compinit
local:50: maximum nested function level reached
local:1: maximum nested function level reached
zsh% 

Obviously the recursion depth isn't 1 or 10 or 50 either.  Where is that
number coming from?  I would have expected "1" in all cases.

Re: Recursion error and line numbers

[Peter Stephenson]

Bart Schaefer wrote:
> local foo=(); local bar
> 
> In workers/22495 I wrote:
> 
> > zsh% local
> > local:10: maximum nested function level reached
> > zsh% 

On the two PCs I tried this on (neither particularly antiquated) it
crashed, so the maximum function depth of 4096 isn't limited enough.
How about 1000?  That seems to stop it.

> There are not 10 lines in the function "local", nor are there 50 lines
> when I try the "compinit" test and mysteriously get the error twice:
> 
> zsh% compinit
> local:50: maximum nested function level reached
> local:1: maximum nested function level reached
> zsh% 
> 
> Obviously the recursion depth isn't 1 or 10 or 50 either.  Where is that
> number coming from?  I would have expected "1" in all cases.

It's a subtle effect of line number handling and I haven't quite worked
out why yet.  It seems that in the case in question we haven't quite got
around to setting the line number in the function yet (on any of the
4096 calls).   Compare with:

% local() { :; local bar; }
% local
local: maximum nested function level reached

No line number (0 is suppressed).

% local() { :;             
local bar; }
% local       
local:1: maximum nested function level reached

The 1 you're expecting.

This is obviously a bug but I haven't it traced it.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Recursion error and line numbers

[Bart Schaefer]

On Jun 18,  1:38pm, Peter Stephenson wrote:
} Subject: Re: Recursion error and line numbers
}
} On the two PCs I tried this on (neither particularly antiquated) it
} crashed, so the maximum function depth of 4096 isn't limited enough.

I have 3GB RAM + 6GB swap on my home machine now (total overkill for
most of what it does) so I'm probably not a good test case for this
kind of thing any more ...

} How about 1000?  That seems to stop it.

It looks as if it's been 4096 ever since we added it back in about
April 2000.  A default of 1000 is most likely OK, given that it's
configure-time settable.

What we need is something like perl's "goto &routine;" to implement a
tail-call optimization.  Of course, that just turns it into an infinite
loop.

} > Obviously the recursion depth isn't 1 or 10 or 50 either.  Where is that
} > number coming from?  I would have expected "1" in all cases.
} 
} It's a subtle effect of line number handling and I haven't quite worked
} out why yet.

Perhaps it has something to do with having left off the braces around
the function body?

Compare:

torch% deep() deep 
torch% deep
deep:2: maximum nested function level reached
torch% deep() { deep }
torch% deep
deep: maximum nested function level reached
torch% deep()         
function> deep
torch% deep
deep:7: maximum nested function level reached
torch% deep()
function> { deep }
torch% deep
deep:1: maximum nested function level reached

Note that without the braces, the line numbers seem to be counting the
number lines of input that have been read; the number must be coming in
from the surrounding scope.

Re: Recursion error and line numbers

[Peter Stephenson]

Bart Schaefer wrote:
> } How about 1000?  That seems to stop it.
> 
> It looks as if it's been 4096 ever since we added it back in about
> April 2000.  A default of 1000 is most likely OK, given that it's
> configure-time settable.

Yes, I think making the average PC work is a good thing to aim for in
the default settings.  Here's the patch.

> } > Obviously the recursion depth isn't 1 or 10 or 50 either.  Where is that
> } > number coming from?  I would have expected "1" in all cases.
> } 
> } It's a subtle effect of line number handling and I haven't quite worked
> } out why yet.
> 
> Perhaps it has something to do with having left off the braces around
> the function body?

That seems to be it.  Hmm... some difference in behaviour is not *so*
stupid.  A function written in that form is a bit like an inlined macro,
and you might usually be more interested in the line number in the
context rather than the function.  Still, the combination of function
name and line number which isn't in the function is clearly wrong.

Index: README
===================================================================
RCS file: /cvsroot/zsh/zsh/README,v
retrieving revision 1.32
diff -u -r1.32 README
--- README	20 Mar 2006 11:06:23 -0000	1.32
+++ README	19 Jun 2006 10:17:31 -0000
@@ -63,6 +63,12 @@
 hit for anyone not using PINE.  The previous default can be restored with:
   zstyle ':completion:*' pine-directory ~/mail
 
+The default maximum function depth (configurable with
+--enable-max-function-depth) has been decreased to 1000 from 4096.  The
+previous value was observed to be large enough that crashes still occurred
+on some fairly common PC configurations.  This change is only likely to
+affect some highly specialised uses of the shell.
+
 Documentation
 -------------
 
Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.54
diff -u -r1.54 configure.ac
--- configure.ac	5 Jun 2006 16:55:38 -0000	1.54
+++ configure.ac	19 Jun 2006 10:17:33 -0000
@@ -325,13 +325,13 @@
 AH_TEMPLATE([MAX_FUNCTION_DEPTH],
 [Define for function depth limits])
 AC_ARG_ENABLE(max-function-depth,
-AC_HELP_STRING([--enable-max-function-depth=MAX], [limit function depth to MAX, default 4096]),
+AC_HELP_STRING([--enable-max-function-depth=MAX], [limit function depth to MAX, default 1000]),
 [if test x$enableval = xyes; then
-  AC_DEFINE(MAX_FUNCTION_DEPTH, 4096)
+  AC_DEFINE(MAX_FUNCTION_DEPTH, 1000)
 elif test x$enableval != xno; then
   AC_DEFINE_UNQUOTED(MAX_FUNCTION_DEPTH, $enableval)
 fi],
-[AC_DEFINE(MAX_FUNCTION_DEPTH, 4096)]
+[AC_DEFINE(MAX_FUNCTION_DEPTH, 1000)]
 )
 
 dnl Do you want to look for pcre support?

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

BUG: cmdstack empty

[Bart Schaefer]

With the latest 4.3.2-dev-1 as of 2006-07-08:

schaefer<509> foo() {     
function> [[ 1 eq 2 ]]
zsh: condition expected: eq
BUG: cmdstack empty

Re: BUG: cmdstack empty

[Peter Stephenson]

Bart Schaefer wrote:
> With the latest 4.3.2-dev-1 as of 2006-07-08:
> 
> schaefer<509> foo() {     
> function> [[ 1 eq 2 ]]
> zsh: condition expected: eq
> BUG: cmdstack empty

Index: Src/parse.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
retrieving revision 1.55
diff -u -r1.55 parse.c
--- Src/parse.c	26 Jun 2006 10:04:09 -0000	1.55
+++ Src/parse.c	9 Jul 2006 14:44:24 -0000
@@ -1672,7 +1672,6 @@
 		    lineno += oldlineno;
 		    ecnpats = onp;
 		    ecssub = oecssub;
-		    cmdpop();
 		    YYERROR(oecused);
 		}
 		yylex();
Index: Src/prompt.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/prompt.c,v
retrieving revision 1.37
diff -u -r1.37 prompt.c
--- Src/prompt.c	7 Mar 2006 21:31:30 -0000	1.37
+++ Src/prompt.c	9 Jul 2006 14:44:25 -0000
@@ -31,7 +31,7 @@
 #include "prompt.pro"
 
 /* text attribute mask */
- 
+
 /**/
 unsigned txtattrmask;
 
@@ -41,7 +41,7 @@
 mod_export unsigned txtchange;
 
 /* the command stack for use with %_ in prompts */
- 
+
 /**/
 unsigned char *cmdstack;
 /**/
@@ -59,7 +59,7 @@
     "cmdsubst", "mathsubst", "elif-then", "heredoc",
     "heredocd", "brace",     "braceparam", "always",
 };
- 
+
 /* The buffer into which an expanded and metafied prompt is being written, *
  * and its size.                                                           */
 
@@ -1304,3 +1304,22 @@
     }
     return 1;
 }
+
+/**/
+void
+cmdpush(int cmdtok)
+{
+    if (cmdsp >= 0 && cmdsp < CMDSTACKSZ)
+	cmdstack[cmdsp++] = (unsigned char)cmdtok;
+}
+
+/**/
+void
+cmdpop(void)
+{
+    if (cmdsp <= 0) {
+	DPUTS(1, "BUG: cmdstack empty");
+	fflush(stderr);
+    } else
+	cmdsp--;
+}
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.91
diff -u -r1.91 zsh.h
--- Src/zsh.h	28 Jun 2006 13:12:55 -0000	1.91
+++ Src/zsh.h	9 Jul 2006 14:44:27 -0000
@@ -1768,20 +1768,6 @@
 /****************************************/
 
 #define CMDSTACKSZ 256
-#define cmdpush(X) do { \
-                       if (cmdsp >= 0 && cmdsp < CMDSTACKSZ) \
-                           cmdstack[cmdsp++]=(X); \
-                   } while (0)
-#ifdef DEBUG
-# define cmdpop()  do { \
-                       if (cmdsp <= 0) { \
-			   fputs("BUG: cmdstack empty\n", stderr); \
-			   fflush(stderr); \
-		       } else cmdsp--; \
-                   } while (0)
-#else
-# define cmdpop()   do { if (cmdsp > 0) cmdsp--; } while (0)
-#endif
 
 #define CS_FOR          0
 #define CS_WHILE        1

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Menu-driven version of history-beginning-search-backward

[Peter Stephenson]

Bart Schaefer wrote:
> On 7/26/06, Peter Stephenson <pws@csr.com> wrote:
> > I was too lazy to search the mailing list archive for anything like
> > this, but in any case it would be good to have something like it in
> > the distribution.
> 
> April 2004, zsh-users/7370.

(Moved to zsh-workers.)

Oh yes, that's a completion widget instead.  It looks like it doesn't do
the "-beginning" bit, but it does do menu selection which is hard from
outside completion.  There's probably room for both.  I think I'll stick
mine in Functions/Zle.

Index: Doc/Zsh/contrib.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/contrib.yo,v
retrieving revision 1.58
diff -u -r1.58 contrib.yo
--- Doc/Zsh/contrib.yo	19 Apr 2006 18:03:02 -0000	1.58
+++ Doc/Zsh/contrib.yo	28 Jul 2006 09:07:20 -0000
@@ -605,6 +605,29 @@
 bindkey '\e^P' history-beginning-search-backward-end
 bindkey '\e^N' history-beginning-search-forward-end)
 )
+tindex(history-beginning-search-menu)
+item(tt(history-beginning-search-menu))(
+This function implements yet another form of history searching.  The
+text before the cursor is used to select lines from the history,
+as for tt(history-beginning-search-backward) except that all matches are
+shown in a numbered menu.  Typing the appropriate digits inserts the
+full history line.  Note that leading zeroes must be typed (they are only
+shown when necessary for removing ambiguity).  The entire history is
+searched; there is no distinction between forwards and backwards.
+
+With a prefix argument, the search is not anchored to the start of
+the line; the string typed by the use may appear anywhere in the line
+in the history.
+
+If the widget name contains `tt(-end)' the cursor is moved to the end of
+the line inserted.  If the widget name contains `tt(-space)' any space
+in the text typed is treated as a wildcard and can match anything (hence
+a leading space is equivalent to giving a prefix argument).  Both
+forms can be combined, for example:
+
+example(zle -N history-beginning-search-menu-space-end \ 
+       history-beginning-search-menu)
+)
 tindex(history-pattern-search)
 tindex(history-pattern-search-backward)
 tindex(history-pattern-search-forward)
Index: Functions/Zle/history-beginning-search-menu
===================================================================
RCS file: Functions/Zle/history-beginning-search-menu
diff -N Functions/Zle/history-beginning-search-menu
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Functions/Zle/history-beginning-search-menu	28 Jul 2006 09:07:20 -0000
@@ -0,0 +1,89 @@
+# Menu-driven alternative to history-beginning-search-backward.
+# As it uses a menu there is no sense of "forward" or "backward", however;
+# the entire history is searched.
+#
+# Configuration:
+#   autoload -U history-beginning-search-menu
+#   zle -N history-beginning-search-menu
+#   bindkey '\eP' history-beginning-search-menu
+#
+# Example:
+#   % /bin/su<ESC-P>
+#   Enter digit:
+#   1 /bin/su -c 'make install'            4 /bin/su - perforce
+#   2 /bin/su                              5 /bin/su -c
+#   3 /bin/su -c 'chown pws:pws **/*(u0)'
+#
+# Typing "1" expands the line to
+#   % /bin/su -c 'make install'
+#
+# With a prefix argument, the search is not anchored to the beginning,
+# so for example "/su" could expand to "p4 files //depot/support/..."
+#
+# If this is bound to a widget containing "-end", e.g.
+#   zle -N history-beginning-search-menu-end history-beginning-search-menu
+# then the cursor is put at the end of the line, else it is left
+# after the matched characters.
+#
+# If this is bound to a widget containing "-space", then any space in
+# the line so far is matched as a wildcard.  (This means putting a space
+# at the start of the line is equivalent to specifying a prefix
+# argument.)
+
+emulate -L zsh
+setopt extendedglob
+
+zmodload -i zsh/parameter
+
+local -aU matches
+local -a display
+
+local search=$LBUFFER
+
+if [[ $WIDGET = *-space* ]]; then
+  search=${search//(#m)[*?#<>]/\\$MATCH/}
+  search=${search// /*}
+fi
+
+if (( ${+NUMERIC} )); then
+  matches=(${(o)history[(R)*${search}*]})
+else
+  matches=(${(o)history[(R)${search}*]})
+fi
+
+# Filter out any match that's the same as the original.
+# Note this isn't a pattern this time.
+matches=(${matches:#${LBUFFER}})
+
+integer n=${#matches}
+integer width=${#n}
+
+(( n == 0 )) && return 1
+
+# Hey, this works...
+integer i
+display=(${matches/(#m)*/${(l.$width..0.):-$((++i))} $MATCH})
+zle -R "Enter digit${${width##1}:+s}:" $display
+
+local chars
+read -k$width chars
+
+if [[ $chars != [[:digit:]]## || $chars -eq 0 || $chars -gt $n ]]; then
+  return 1
+fi
+
+if [[ $WIDGET = *-end* ]]; then
+  LBUFFER=${matches[$chars]} RBUFFER=
+else
+  integer newcursor
+  if (( ${+NUMERIC} )); then
+    # Advance cursor so that it's still after the string typed
+    local -a match mbegin mend
+    if [[ $matches[$chars] = (#b)(*${LBUFFER})* ]]; then
+      newcursor=${#match[1]}
+    fi
+  fi
+
+  BUFFER=${matches[$chars]}
+  (( newcursor )) && CURSOR=$newcursor
+fi

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Menu-driven version of history-beginning-search-backward

[Peter Stephenson]

Minor tweak with just interesting enough consequences to bother posting
about.  The menu displayed with "zle -R" doesn't get cleared; I think
that's because the code doesn't remember the list of things it's
displaying as it does in the completion code.  I've hacked it here to
clear the list by brute force.

This is related to another annoyance that the completion listing code is
tied to completion and then hooked into zle_refresh.c.  If it were a
properly separate module (it's already a different library, but the API
is a bit non-existent) we could use it for tasks like menu selection in
a case like this, outside the completion code.

This is a pipe dream.

Index: Functions/Zle/history-beginning-search-menu
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/history-beginning-search-menu,v
retrieving revision 1.1
diff -u -r1.1 history-beginning-search-menu
--- Functions/Zle/history-beginning-search-menu	28 Jul 2006 09:52:35 -0000	1.1
+++ Functions/Zle/history-beginning-search-menu	28 Jul 2006 10:01:53 -0000
@@ -41,6 +41,10 @@
 local search=$LBUFFER
 
 if [[ $WIDGET = *-space* ]]; then
+  # We need to quote metacharacters in the search string
+  # since they are otherwise active in the reverse subscript.
+  # We need to avoid quoting other characters since they aren't
+  # and just stay quoted, rather annoyingly.
   search=${search//(#m)[*?#<>]/\\$MATCH/}
   search=${search// /*}
 fi
@@ -68,7 +72,14 @@
 local chars
 read -k$width chars
 
+# Hmmm... this isn't great.  The only way of clearing the display
+# appears to be to overwrite it completely.  I think that's because
+# displaying strings in this way doesn't set the completion list
+# properly.
+display=(${display//?/ })
+
 if [[ $chars != [[:digit:]]## || $chars -eq 0 || $chars -gt $n ]]; then
+  zle -R '' $display
   return 1
 fi
 
@@ -87,3 +98,5 @@
   BUFFER=${matches[$chars]}
   (( newcursor )) && CURSOR=$newcursor
 fi
+
+zle -R '' $display

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

env -u completion doesn't work when there is a command

[Vincent Lefevre]

Hi,

With zsh 4.3.2, if I type:

  env -u LIBR[TAB]

I get:

  env -u LIBRARY_PATH

(because LIBRARY_PATH is defined here), but if I type:

  env -u LIBR[TAB] gcc

where the cursor is after LIBR, the completion doesn't work.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: env -u completion doesn't work when there is a command

[Bart Schaefer]

On Jul 30,  8:00pm, Vincent Lefevre wrote:
}
}   env -u LIBR[TAB] gcc
} 
} where the cursor is after LIBR, the completion doesn't work.

This is a generalized problem with _arguments, in that there's no way
to specify that an option (-u in this case) has a mandatory argument
that must be in a separate word.

Consequently, when a "rest" specification ('*::arguments: _normal')
appears and there is any ambiguity, it takes precedence.  Probably
something could be done to _arguments to change this, but I don't
have my hip waders and piton hammer handy today.

There are work-arounds possible in the caller of _arguments, mostly by
examining $words[CURRENT-1] and modifying the _arguments specification.
Maybe someone can suggest the best attack?

Re: env -u completion doesn't work when there is a command

[Peter Stephenson]

Bart Schaefer wrote:
> On Jul 30,  8:00pm, Vincent Lefevre wrote:
> }
> }   env -u LIBR[TAB] gcc
> } 
> } where the cursor is after LIBR, the completion doesn't work.
> 
> This is a generalized problem with _arguments, in that there's no way
> to specify that an option (-u in this case) has a mandatory argument
> that must be in a separate word.

I'm not sure it's that, since using the specification
  '-u+:env var to remove:compadd ${(k)parameters[(R)*export*]}'
doesn't seem to fix it.  This should says that if there's no option
immediately after the -u there must be one in the next word, which would
cover the current case.

It seems comparguments -O is returning false, causing _arguments to
think there are no more arguments to complete.

comparguments is defined in zsh/computil and is documented thus:

       comparguments
              This  is  used by the _arguments function to do the argument and
              command line parsing.  Like compdescribe it has an option -i  to
              do  the  parsing  and initialize some internal state and various
              options to access the state information to decide what should be
              completed.

Right.  Slightly more helpfully, the code for comparguments -O says:

        /* This returns the descriptions for the options in the arrays whose
         * names are given as arguments.  The descriptions are strings in a
         * form usable by _describe.  The return value says if there are any
         * options to be completed. */

so I would guess this is looking at the wrong word to decide if there
are options to be completed, since we're getting to

      _tags "$subcs[@]"

at _arguments:219 instead of 

      _tags "$subcs[@]" options

at _arguments:217.  However, as that's the only comment it's hard to
tell.  Actually, "$subcs[@]" expands to argument-rest, so maybe it's
comparguments -D, which sets subcs, that's too blame.  The comment for
that says

        /* This returns the descriptions, actions and sub-contexts for the
         * things _arguments has to execute at this place on the line (the
         * sub-contexts are used as tags).
         * The return value is particularly important here, it says if 
         * there are arguments to completely at all. */

The argument-rest comes from ca_set_data(), an uncommented 91-line
function that does the guts of comparguments -D.  This uses the state
originally set up by comparguments -i, however.  comparguments -i calls
ca_parse_line() which has a few comments suggesting it might be
searching for options and arguments.  I think that might be the place to
start.  But who knows?

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Menu-driven version of history-beginning-search-backward

[Peter Stephenson]

Still tweaking.

Read the keys for the menu separately, so that it will abort on the
first non-digit instead of trying to read all digits first and finding
the first one isn't a digit.

Use HISTNO to get to the history line, consistent with other history
functions; this allows things like accept-line-and-down-history to work
properly.  The obvious way to do this was to reverse match on the
history again to get the line number (if it had been possible to extract
a key and a value into separate arrays I'd have done that; everything I
thought of involved looping over the elements).  This time I had to get
the quoting of the key exactly right; having done it I put a note in the
manual.  I wish we didn't need all these explanatory notes for
subscripts.

Index: Doc/Zsh/params.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/params.yo,v
retrieving revision 1.29
diff -u -r1.29 params.yo
--- Doc/Zsh/params.yo	12 May 2006 14:58:44 -0000	1.29
+++ Doc/Zsh/params.yo	1 Aug 2006 17:23:25 -0000
@@ -211,6 +211,18 @@
 example([[ ${array[(i)pattern]} -le ${#array} ]])
 
 If tt(KSH_ARRAYS) is in effect, the tt(-le) should be replaced by tt(-lt).
+
+Note that in subscripts with both `tt(r)' and `tt(R)' pattern characters
+are active even if they were substituted for a parameter (regardless
+of the setting of tt(GLOB_SUBST) which controls this feature in normal
+pattern matching).  It is therefore necessary to quote pattern characters
+for an exact string match.  Given a string in tt($key), and assuming
+the tt(EXTENDED_GLOB) option is set, the following is sufficient to
+match an element of an array tt($array) containing exactly the value of
+tt($key):
+
+example(key2=${key//(#m)[\][+LPAR()+RPAR()\\*?#<>]/\\$MATCH}
+print ${array[(R)$key2]})
 )
 item(tt(R))(
 Like `tt(r)', but gives the last match.  For associative arrays, gives
Index: Functions/Zle/history-beginning-search-menu
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/history-beginning-search-menu,v
retrieving revision 1.2
diff -u -r1.2 history-beginning-search-menu
--- Functions/Zle/history-beginning-search-menu	28 Jul 2006 10:21:07 -0000	1.2
+++ Functions/Zle/history-beginning-search-menu	1 Aug 2006 17:23:25 -0000
@@ -45,7 +45,7 @@
   # since they are otherwise active in the reverse subscript.
   # We need to avoid quoting other characters since they aren't
   # and just stay quoted, rather annoyingly.
-  search=${search//(#m)[*?#<>]/\\$MATCH/}
+  search=${search//(#m)[\][()\\*?#<>]/\\$MATCH/}
   search=${search// /*}
 fi
 
@@ -69,8 +69,19 @@
 display=(${matches/(#m)*/${(l.$width..0.):-$((++i))} $MATCH})
 zle -R "Enter digit${${width##1}:+s}:" $display
 
-local chars
-read -k$width chars
+integer i
+local char chars
+
+# Abort on first non-digit entry instead of requiring all
+# characters to be typed (as "read -k$width chars" would do).
+for (( i = 0; i < $width; i++ )); do
+  read -k char
+  if [[ $char != [[:digit:]] ]]; then
+    zle -R '' $display
+    return 1
+  fi
+  chars+=$char
+done
 
 # Hmmm... this isn't great.  The only way of clearing the display
 # appears to be to overwrite it completely.  I think that's because
@@ -78,25 +89,37 @@
 # properly.
 display=(${display//?/ })
 
-if [[ $chars != [[:digit:]]## || $chars -eq 0 || $chars -gt $n ]]; then
+if [[ $chars -eq 0 || $chars -gt $n ]]; then
   zle -R '' $display
   return 1
 fi
 
-if [[ $WIDGET = *-end* ]]; then
-  LBUFFER=${matches[$chars]} RBUFFER=
-else
-  integer newcursor
+integer newcursor
+if [[ $WIDGET != *-end* ]]; then
   if (( ${+NUMERIC} )); then
     # Advance cursor so that it's still after the string typed
     local -a match mbegin mend
     if [[ $matches[$chars] = (#b)(*${LBUFFER})* ]]; then
-      newcursor=${#match[1]}
+       newcursor=${#match[1]}
     fi
+  else
+    # Maintain cursor
+    newcursor=$CURSOR
   fi
+fi
 
-  BUFFER=${matches[$chars]}
-  (( newcursor )) && CURSOR=$newcursor
+# Find the history lines that contain the matched string and
+# go to the last one.  This allows accept-line-and-down-history etc.
+# to work.
+local -a lines
+local matchq=${matches[$chars]//(#m)[\][()\\*?#<>]/\\$MATCH}
+lines=(${(kon)history[(R)$matchq]})
+HISTNO=$lines[-1]
+
+if (( newcursor )); then
+  CURSOR=$newcursor
+elif [[ $WIDGET = *-end* ]]; then
+  CURSOR=${#BUFFER}
 fi
 
 zle -R '' $display

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Menu-driven version of history-beginning-search-backward

[Peter Stephenson]

I suppose this is going to come to an end soon.  It must be nice being
smart.

Index: Doc/Zsh/params.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/params.yo,v
retrieving revision 1.30
diff -u -r1.30 params.yo
--- Doc/Zsh/params.yo	1 Aug 2006 17:35:17 -0000	1.30
+++ Doc/Zsh/params.yo	1 Aug 2006 20:17:51 -0000
@@ -221,7 +221,7 @@
 match an element of an array tt($array) containing exactly the value of
 tt($key):
 
-example(key2=${key//(#m)[\][+LPAR()+RPAR()\\*?#<>]/\\$MATCH}
+example(key2=${key//(#m)[\][+LPAR()+RPAR()\\*?#<>~]/\\$MATCH}
 print ${array[(R)$key2]})
 )
 item(tt(R))(
Index: Functions/Zle/history-beginning-search-menu
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/history-beginning-search-menu,v
retrieving revision 1.3
diff -u -r1.3 history-beginning-search-menu
--- Functions/Zle/history-beginning-search-menu	1 Aug 2006 17:35:18 -0000	1.3
+++ Functions/Zle/history-beginning-search-menu	1 Aug 2006 20:17:51 -0000
@@ -40,12 +40,12 @@
 
 local search=$LBUFFER
 
+search=${search//(#m)[\][()\\*?#<>~]/\\$MATCH/}
 if [[ $WIDGET = *-space* ]]; then
   # We need to quote metacharacters in the search string
   # since they are otherwise active in the reverse subscript.
   # We need to avoid quoting other characters since they aren't
   # and just stay quoted, rather annoyingly.
-  search=${search//(#m)[\][()\\*?#<>]/\\$MATCH/}
   search=${search// /*}
 fi
 
@@ -112,7 +112,7 @@
 # go to the last one.  This allows accept-line-and-down-history etc.
 # to work.
 local -a lines
-local matchq=${matches[$chars]//(#m)[\][()\\*?#<>]/\\$MATCH}
+local matchq=${matches[$chars]//(#m)[\][()\\*?#<>~]/\\$MATCH}
 lines=(${(kon)history[(R)$matchq]})
 HISTNO=$lines[-1]
 
-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Menu-driven version of history-beginning-search-backward

[Bart Schaefer]

On Aug 1,  9:18pm, Peter Stephenson wrote:
}
} I suppose this is going to come to an end soon.  It must be nice being
} smart.

I have no idea how to interpret that.

Bug#335481: zsh: zsh/sched waits for next return to prompt even with NOTIFY set

[Jared K. Sorensen]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

So there is no plan to "fix" the sched module to work without requiring a
return at the prompt? If you're going to require a return from the prompt to
set off the command, you should document that at least, so you users don't
expect it to run like a real scheduler. Call me a TCSH JUNKIE, but I thought
sched would run without an acknowledge from the user after the assigned
time. Please provide someway to make this functional to occur at the
assigned time without user input because (I echo Ramkumar's sentiments) this
makes it a completely useless module since I primarily use it to run
programs in the middle of the night when the licenses are most likely to be
available to send off builds to the program I need to run and I'd like to
not have to wake-up and hit the return key every few hours to make it work.

Jared K. Sorensen
-- 
/=\ | / /=\
_ | |=   \
\=/ | \ \=/

[-- Attachment #2: Type: text/html, Size: 938 bytes --]

zstyle is badly broken as of 20060817

[Bart Schaefer]

schaefer<532> zstyle ':completion:*' list-prompt ''
schaefer<533> zstyle -L | grep list-prompt 
zstyle ':completion:*' list-prompt ''
schaefer<534> zstyle -d ':completion:*' list-prompt
schaefer<535> zstyle | grep list-prompt
list-prompt
schaefer<536> zstyle -L | grep list-prompt
schaefer<537> 

So the list-prompt style is still there even after supposedly being deleted,
but zstyle -L does not list it (and completion behaves as if it is still
set, which is how I even noticed).

schaefer<545> zstyle -s ":completion:${curcontext}:default" list-prompt tmp
schaefer<546> echo $?
0

It's as if it is set with a pattern of "*".

Re: zstyle is badly broken as of 20060817

[Peter Stephenson]

On Sat, 19 Aug 2006 11:11:55 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> schaefer<532> zstyle ':completion:*' list-prompt ''
> schaefer<533> zstyle -L | grep list-prompt 
> zstyle ':completion:*' list-prompt ''
> schaefer<534> zstyle -d ':completion:*' list-prompt
> schaefer<535> zstyle | grep list-prompt
> list-prompt
> schaefer<536> zstyle -L | grep list-prompt
> schaefer<537> 
> 
> So the list-prompt style is still there even after supposedly being deleted,
> but zstyle -L does not list it (and completion behaves as if it is still
> set, which is how I even noticed).

Your subject line implies this is something new, but it isn't.  When the
last pattern for a style is deleted the style hangs around, with the
effects you see.  You'd find it was cosmetic, although confusing and
potentially memory hogging, in that there were no contexts given for the
style.  This fixes it.

> schaefer<545> zstyle -s ":completion:${curcontext}:default" list-prompt tmp
> schaefer<546> echo $?
> 0

I didn't seem to get this effect.  It would have to be a different bug,
since searching the list of context patterns appears to be correctly
terminated even if it's empty.

Index: Src/Modules/zutil.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/zutil.c,v
retrieving revision 1.18
diff -u -r1.18 zutil.c
--- Src/Modules/zutil.c	16 Aug 2006 09:06:40 -0000	1.18
+++ Src/Modules/zutil.c	20 Aug 2006 16:44:11 -0000
@@ -58,9 +58,27 @@
 
 /* Memory stuff. */
 
+/*
+ * Free the information for one of the patterns associated with
+ * a style.
+ *
+ * If the style s is passed, prev is the previous pattern in the list,
+ * found when scanning.  We use this to update the list of patterns.
+ * If this results in their being no remaining patterns, the style
+ * itself is removed from the list of styles.  This isn't optimised,
+ * since it's not a very frequent operation; we simply scan down the list
+ * to find the previous entry.
+ */
 static void
-freestypat(Stypat p)
+freestypat(Stypat p, Style s, Stypat prev)
 {
+    if (s) {
+	if (prev)
+	    prev->next = p->next;
+	else
+	    s->pats = p->next;
+    }
+
     zsfree(p->pat);
     freepatprog(p->prog);
     if (p->vals)
@@ -68,6 +86,20 @@
     if (p->eval)
 	freeeprog(p->eval);
     zfree(p, sizeof(*p));
+
+    if (s && !s->pats) {
+	/* No patterns left, free style */
+	if (s == zstyles) {
+	    zstyles = s->next;
+	} else {
+	    Style s2;
+	    for (s2 = zstyles; s2->next != s; s2 = s2->next)
+		;
+	    s2->next = s->next;
+	}
+	zsfree(s->name);
+	zfree(s, sizeof(*s));
+    }
 }
 
 static void
@@ -80,7 +112,7 @@
 	sn = s->next;
 	for (p = s->pats; p; p = pn) {
 	    pn = p->next;
-	    freestypat(p);
+	    freestypat(p, NULL, NULL);
 	}
 	zsfree(s->name);
 	zfree(s, sizeof(*s));
@@ -96,8 +128,9 @@
     Style s;
 
     for (s = zstyles; s; s = s->next)
-	if (!strcmp(name, s->name))
+	if (!strcmp(name, s->name)) {
 	    return s;
+	}
 
     return NULL;
 }
@@ -397,27 +430,22 @@
 			    for (q = NULL, p = s->pats; p;
 				 q = p, p = p->next) {
 				if (!strcmp(p->pat, pat)) {
-				    if (q)
-					q->next = p->next;
-				    else
-					s->pats = p->next;
-				    freestypat(p);
+				    freestypat(p, s, q);
 				    break;
 				}
 			    }
 			}
 		    }
 		} else {
+		    Style next;
 		    Stypat p, q;
 
-		    for (s = zstyles; s; s = s->next) {
+		    /* careful! style itself may be deleted */
+		    for (s = zstyles; s; s = next) {
+			next = s->next;
 			for (q = NULL, p = s->pats; p; q = p, p = p->next) {
 			    if (!strcmp(p->pat, args[1])) {
-				if (q)
-				    q->next = p->next;
-				else
-				    s->pats = p->next;
-				freestypat(p);
+				freestypat(p, s, q);
 				break;
 			    }
 			}

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zstyle is badly broken as of 20060817

[Bart Schaefer]

On Aug 20,  5:54pm, Peter Stephenson wrote:
} Subject: Re: zstyle is badly broken as of 20060817
}
} > So the list-prompt style is still there even after supposedly being
} > deleted, but zstyle -L does not list it (and completion behaves as
} > if it is still set, which is how I even noticed).
} 
} Your subject line implies this is something new, but it isn't.

I didn't know whether it was new or not, but I've concluded lately
that I ought to mention when I encountered a bug so that people
using later builds aren't confused.  This is one of the drawbacks
of using the mailing list to report bugs rather than having a 
tracking system where they can be marked resolved.  (No, I'm not
suggesting we change.)

} > schaefer<545> zstyle -s ":completion:${curcontext}:default" list-prompt tmp
} > schaefer<546> echo $?
} > 0
} 
} I didn't seem to get this effect.

The way I noticed the trouble in the first place was that once I set a
list-prompt style I always got the scrolling display, even after I'd
supposedly deleted the style.

} It would have to be a different bug,

You're right, it is.  Now the style goes away, but complist continues
to have the scrolling behavior.

Also now there's no way to get the style back again:

schaefer<501> zstyle ":completion:*:default" list-prompt ''
schaefer<502> zstyle | grep list-prompt
list-prompt
schaefer<503> zstyle -d  ":completion:*:default" list-prompt
schaefer<504> zstyle | grep list-prompt
schaefer<505> zstyle ":completion:*:default" list-prompt "Go On"
schaefer<506> zstyle | grep list-prompt
schaefer<507> echo $?
1
schaefer<508> 

And yet I if I invoke completion after 502, then I still get the list
displayed scrolling after 508.

Re: zstyle is badly broken as of 20060817

[Bart Schaefer]

On Aug 20,  1:44pm, Bart Schaefer wrote:
}
} schaefer<501> zstyle ":completion:*:default" list-prompt ''
} schaefer<502> zstyle | grep list-prompt
} list-prompt
} schaefer<503> zstyle -d  ":completion:*:default" list-prompt
} schaefer<504> zstyle | grep list-prompt
} schaefer<505> zstyle ":completion:*:default" list-prompt "Go On"
} schaefer<506> zstyle | grep list-prompt
} schaefer<507> echo $?
} 1
} schaefer<508> 
} 
} And yet I if I invoke completion after 502, then I still get the list
} displayed scrolling after 508.

And after 503, I should add.

Re: zstyle is badly broken as of 20060817

[Peter Stephenson]

Bart Schaefer wrote:
> Also now there's no way to get the style back again:
> 
> schaefer<501> zstyle ":completion:*:default" list-prompt ''
> schaefer<502> zstyle | grep list-prompt
> list-prompt
> schaefer<503> zstyle -d  ":completion:*:default" list-prompt
> schaefer<504> zstyle | grep list-prompt
> schaefer<505> zstyle ":completion:*:default" list-prompt "Go On"
> schaefer<506> zstyle | grep list-prompt
> schaefer<507> echo $?
> 1
> schaefer<508> 

Deleting styles had a bad effect because there's a zlstyles variable
which always points to the last variable added, so that new styles are
added there.  This doesn't work so well if you can remove styles.  I've
just removed zlstyles.  If an optimisation is really necessary a better
one would be to hash the styles.  This fixed the symptom in my case.

I haven't looked at the effect of list-prompt after being removed.  I
probably won't get a chance for some weeks.  Anyone else is welcome to
look at anything.

Index: Src/Modules/zutil.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/zutil.c,v
retrieving revision 1.19
diff -u -r1.19 zutil.c
--- Src/Modules/zutil.c	20 Aug 2006 17:04:58 -0000	1.19
+++ Src/Modules/zutil.c	20 Aug 2006 21:36:56 -0000
@@ -54,7 +54,7 @@
     
 /* List of styles. */
 
-static Style zstyles, zlstyles;
+static Style zstyles;
 
 /* Memory stuff. */
 
@@ -117,7 +117,7 @@
 	zsfree(s->name);
 	zfree(s, sizeof(*s));
     }
-    zstyles = zlstyles = NULL;
+    zstyles = NULL;
 }
 
 /* Get the style struct for a name. */
@@ -233,11 +233,8 @@
     s->pats = NULL;
     s->name = ztrdup(name);
 
-    if (zlstyles)
-	zlstyles->next = s;
-    else
-	zstyles = s;
-    zlstyles = s;
+    s->next = zstyles;
+    zstyles = s;
 
     return s;
 }
@@ -1748,7 +1745,7 @@
 int
 setup_(UNUSED(Module m))
 {
-    zstyles = zlstyles = NULL;
+    zstyles = NULL;
 
     return 0;
 }

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zstyle is badly broken as of 20060817

[Bart Schaefer]

On Aug 20, 10:38pm, Peter Stephenson wrote:
}
} I haven't looked at the effect of list-prompt after being removed.  I
} probably won't get a chance for some weeks.  Anyone else is welcome to
} look at anything.

Turns out this is happening because $LISTPROMPT remains set even after
the style is deleted.  This is presently unavoidable because complist
doesn't get a chance to examine LISTPROMPT until after _main_complete
has exited ... the same reason it's not possible to have both the "do
you wish ..." question and scrolling when you answer "yes".

So to disable list-prompt you have to both remove the style and also
unset LISTPROMPT.

Re: zstyle is badly broken as of 20060817

[Vincent Lefevre]

Hi,

On 2006-08-20 22:44:43 -0700, Bart Schaefer wrote:
> So to disable list-prompt you have to both remove the style and also
> unset LISTPROMPT.

This should be documented in the man pages and by compinstall
(after choosing 3 3 3).

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: Bug#335481: zsh: zsh/sched waits for next return to prompt even with NOTIFY set

[Peter Stephenson]

On Fri, 11 Aug 2006 14:03:27 -0600
"Jared K. Sorensen" <jksorensen@gmail.com> wrote:
> So there is no plan to "fix" the sched module to work without requiring a
> return at the prompt?

This is a long-standing problem, and as you say it makes sched a good
deal less useful than it might otherwise be.

I got a chance to look at it.  As we already do quite sophisticated
things while the line editor is waiting for shell input (key timeouts
and polling of file descriptors) it's not so difficult to expand this so
that a sched timer can fire at this point.  I haven't looked at
extending this to be fully asynchronous: it's not clear you want that,
since your terminal's in use by another process at that point, so it
would need to be optional, and I don't feel like messing around with
signal masks and SIGALRM.

I've tried to make it more generally useful (in particular for the
calendar functions I'm playing around with separately), so the patch
has the following visible effects.  Opinions welcome:


sched events will now fire while waiting for user input in zle.

As a side effect of the new mechanism I've upped the arbitrary limit on
key timeouts, which used to be 5 seconds, to (typically) 24 days, which
ought to be long enough ("I'm just on holiday for a few weeks, Dan, I'll
finish that key sequence when I get back").  I don't like arbitrary
limits but you can always argue they are saving the user from their own
stupidity.  There's no definite right answer (apart from using the
correct value in KEYTIMEOUT).

sched events are a little bit more neatly handled than before: the queue
is in time order, and I fixed a bug that deleting a sched job from
itself would cause a crash.  (I found it by inspecting the code but I've
just confirmed it.)

sched now accepts a number of seconds (when there's no ":" in the time),
either as a relative time or absolute time (counting from the epoch in
that case).  This fits in well with what zsh/datetime can provide.

"print" is now aware of being used to output to the terminal when the
line editor is active and will take appropriate action (equivalent to
"zle -I").  This is to make output from sched jobs more user friendly.
The only problem with this I can see is if the "print" is only being
used, say, to update a terminal emulator title bar, and the likely of
doing that at a point where you want to make it invisible to the user
that a function is being run isn't that great anyway so I can't see a
reason why this shouldn't be the default.  The option "-t" to print
turns it off, which is how to do the terminal emulator thing.


Internally, I've done the following:


The interface to prepromptfns is neater, and I've added timedfns for the
new facility, which still keeps sched at a slight difference
interfacewise; however timedfns needs to be visible in zle.  sched still
uses prepromptfns, too, since if a scheduled event has just gone off
before a prompt it saves printing the command line and immediately
trashing it.

get_rawbyte() has been rewritten to do timeouts more generally.  One
effect of this is that a key timeout will be restarted if a watched fd
(zle -F) becomes ready or if a timed function triggers while waiting for
input.  The effect of this ought to be unnoticeably small.

I've tried to make use of time_t a little more consistent.  It's used to
store 100ths of a second, too, so that I only need one test for the
range of the variable used.  I've tried to minimise range problems by
using a zlong to take the value from zstrtol, but actually I haven't
done any more range checking so an overflow in sched is possible.

I renamed the variable keytmout to do_keytmout because it's not actually
a timeout, it's a flag, and it's been confusing me for ages.

I've tried to make the trashzle() mechanism as self-standing as
possible.  It seems to work in the obvious cases (e.g. repeated "print"s
from a single sched event).

While monkeying with pointers, I fixed a warning from gcc in the
filesub() calls.  This isn't related to the rest.

Index: README
===================================================================
RCS file: /cvsroot/zsh/zsh/README,v
retrieving revision 1.36
diff -u -r1.36 README
--- README	4 Aug 2006 15:31:03 -0000	1.36
+++ README	7 Sep 2006 18:48:18 -0000
@@ -69,6 +69,12 @@
     are allowed in identifiers even though the shell will recognise
     alphanumeric multibyte characters.
 
+The sched builtin now keeps entries in time order.  This means that
+after adding an entry the index of an existing entry used for deletion
+may change, if that entry had a later time than the new entry.  However,
+deleting a entry with a later time will now never change the index of an
+entry with an earlier time, which could happen with the previous method.
+
 The completion style pine-directory must now be set to use completion
 for PINE mailbox folders; previously it had the default ~/mail.  This
 change was necessary because otherwise recursive directories under
Index: Doc/Zsh/builtins.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/builtins.yo,v
retrieving revision 1.86
diff -u -r1.86 builtins.yo
--- Doc/Zsh/builtins.yo	24 Jul 2006 22:00:20 -0000	1.86
+++ Doc/Zsh/builtins.yo	7 Sep 2006 18:48:19 -0000
@@ -797,7 +797,7 @@
 `tt(-)' in this context are swapped.
 )
 findex(print)
-xitem(tt(print) [ tt(-abcDilmnNoOpPrsz) ] [ tt(-u) var(n) ] [ tt(-f) var(format) ] [ tt(-C) var(cols) ])
+xitem(tt(print) [ tt(-abcDilmnNoOpPrstz) ] [ tt(-u) var(n) ] [ tt(-f) var(format) ] [ tt(-C) var(cols) ])
 item(  [ tt(-R) [ tt(-en) ]] [ var(arg) ... ])(
 With the `tt(-f)' option the arguments are printed as described by tt(printf).
 With no flags or with the flag `tt(-)', the arguments are printed on
@@ -878,6 +878,19 @@
 item(tt(-s))(
 Place the results in the history list instead of on the standard output.
 )
+item(tt(-t))(
+Sometimes the tt(print) or tt(echo) builtin may used to send output to
+the terminal while the line editor is active.  This can happen, for
+example, from within editor or completion widgets, or from within
+a command queued to run at a particular time by the tt(sched) builtin.
+In such cases the shell normally tries to arrange that the output
+is displayed cleanly by clearing the line by edited and redisplaying
+it later.  This behaviour can be disabled by using the tt(-t) option
+to tt(print).  This is useful, for example, if the arguments to tt(print)
+are only used to update the header line of the terminal emulator.
+
+This option is not necessary if output is not directed to the terminal.
+)
 item(tt(-u) var(n))(
 Print the arguments to file descriptor var(n).
 )
Index: Doc/Zsh/mod_sched.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/mod_sched.yo,v
retrieving revision 1.1.1.7
diff -u -r1.1.1.7 mod_sched.yo
--- Doc/Zsh/mod_sched.yo	20 Dec 1999 11:24:39 -0000	1.1.1.7
+++ Doc/Zsh/mod_sched.yo	7 Sep 2006 18:48:19 -0000
@@ -8,11 +8,33 @@
 cindex(timed execution)
 cindex(execution, timed)
 xitem(tt(sched) [tt(PLUS())]var(hh)tt(:)var(mm) var(command) ...)
+xitem(tt(sched) [tt(PLUS())]var(seconds) var(command) ...)
 item(tt(sched) [ tt(-)var(item) ])(
 Make an entry in the scheduled list of commands to execute.
-The time may be specified in either absolute or relative time.
+The time may be specified in either absolute or relative time,
+and either as hours and minutes separated by a colon, or seconds.
+An absolute number of seconds indicates the time since the epoch
+(1970/01/01 00:00); this is useful in combination with the features in
+the tt(zsh/datetime) module, see
+ifzman(the zsh/datetime module entry in zmanref(zshmodules))\
+ifnzman(noderef(The zsh/datetime Module))\
+.
+
 With no arguments, prints the list of scheduled commands.
 With the argument `tt(-)var(item)', removes the given item
-from the list.
+from the list.  The numbering of the list is continguous and entries are
+in time order, so the numbering can change when entries are added or
+deleted.
+
+Commands are executed either immediately before a prompt, or while
+the shell's line editor is waiting for input.  In the latter case
+it is useful to be able to produce output that does not interfere
+with the line being edited: the tt(print) builtin will automatically
+arrange for this.  In commands that produce output in other ways,
+placing the code:
+
+example(print -n '')
+
+at the start will have the correct effect.
 )
 enditem()
Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.160
diff -u -r1.160 builtin.c
--- Src/builtin.c	24 Jul 2006 22:00:20 -0000	1.160
+++ Src/builtin.c	7 Sep 2006 18:48:21 -0000
@@ -99,7 +99,7 @@
 #endif
 
     BUILTIN("popd", 0, bin_cd, 0, 1, BIN_POPD, NULL, NULL),
-    BUILTIN("print", BINF_PRINTOPTS, bin_print, 0, -1, BIN_PRINT, "abcC:Df:ilmnNoOpPrRsu:z-", NULL),
+    BUILTIN("print", BINF_PRINTOPTS, bin_print, 0, -1, BIN_PRINT, "abcC:Df:ilmnNoOpPrRstu:z-", NULL),
     BUILTIN("printf", 0, bin_print, 1, -1, BIN_PRINTF, NULL, NULL),
     BUILTIN("pushd", BINF_SKIPINVALID | BINF_SKIPDASH | BINF_DASHDASHVALID, bin_cd, 0, 2, BIN_PUSHD, "sPL", NULL),
     BUILTIN("pushln", 0, bin_print, 0, -1, BIN_PRINT, NULL, "-nz"),
@@ -3460,7 +3460,7 @@
 {
     int flen, width, prec, type, argc, n, narg, curlen = 0;
     int nnl = 0, fmttrunc = 0, ret = 0, maxarg = 0;
-    int flags[5], *len;
+    int flags[5], *len, output_to_fout;
     char *start, *endptr, *c, *d, *flag, *buf, spec[13], *fmt = NULL;
     char **first, **argp, *curarg, *flagch = "0+- #", save = '\0', nullstr = '\0';
     size_t rcount, count = 0;
@@ -3595,6 +3595,27 @@
 	}
     }
 
+    /*
+     * Is output going to FILE *fout?
+     * It isn't if we're printing to the zle line or to the
+     * shell history.  Otherwise it is (I think).
+     */
+    output_to_fout = !OPT_ISSET(ops, 'z') && !OPT_ISSET(ops, 's');
+
+    if (zleactive && output_to_fout && !OPT_ISSET(ops, 't') &&
+	/*
+	 * The following checks if fout is associated with the
+	 * shell's terminal.  It's a little clumsy but saves argument.
+	 */
+	isatty(fileno(fout)) && !strcmp(ttystrname, ttyname(fileno(fout)))) {
+	/*
+	 * If zle is active, we want to take action to avoid
+	 * our output messing up the line being edited, so arrange to
+	 * have it cleared and redrawn.
+	 */
+	trashzleptr();
+    }
+
     /* -o and -O -- sort the arguments */
     if (OPT_ISSET(ops,'o')) {
 	if (fmt && !*args) return 0;
@@ -3773,7 +3794,7 @@
 	return ret;
     }
     
-    if (OPT_ISSET(ops,'z') || OPT_ISSET(ops,'s')) {
+    if (!output_to_fout) {
 #ifdef HAVE_OPEN_MEMSTREAM
     	if ((fout = open_memstream(&buf, &mcount)) == NULL)
 	    zwarnnam(name, "open_memstream failed");
@@ -4055,7 +4076,7 @@
 	/* if there are remaining args, reuse format string */
     } while (*argp && argp != first && !fmttrunc && !OPT_ISSET(ops,'r'));
 
-    if (OPT_ISSET(ops,'z') || OPT_ISSET(ops,'s')) {
+    if (!output_to_fout) {
 #ifdef HAVE_OPEN_MEMSTREAM
 	putc(0, fout);
 	fflush(fout);
Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.66
diff -u -r1.66 init.c
--- Src/init.c	30 May 2006 22:35:03 -0000	1.66
+++ Src/init.c	7 Sep 2006 18:48:22 -0000
@@ -869,7 +869,6 @@
     nohistsave = 1;
     dirstack = znewlinklist();
     bufstack = znewlinklist();
-    prepromptfns = znewlinklist();
     hsubl = hsubr = NULL;
     lastpid = 0;
     bshin = SHIN ? fdopen(SHIN, "r") : stdin;
Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.55
diff -u -r1.55 subst.c
--- Src/subst.c	24 Jul 2006 22:00:21 -0000	1.55
+++ Src/subst.c	7 Sep 2006 18:48:23 -0000
@@ -70,9 +70,24 @@
 		return;
 	    }
 	} else {
-	    if (isset(SHFILEEXPANSION))
-		filesub((char **)getaddrdata(node),
-			flags & (PF_TYPESET|PF_ASSIGN));
+	    if (isset(SHFILEEXPANSION)) {
+		/*
+		 * Here and below we avoid taking the address
+		 * of a void * and then pretending it's a char **
+		 * instead of a void ** by a little inefficiency.
+		 * This could be avoided with some extra linked list
+		 * machinery, but that would need quite a lot of work
+		 * to ensure consistency.  What we really need is
+		 * templates...
+		 */
+		char *cptr = (char *)getdata(node);
+		filesub(&cptr, flags & (PF_TYPESET|PF_ASSIGN));
+		/*
+		 * The assignment is so simple it's not worth
+		 * testing if cptr changed...
+		 */
+		setdata(node, cptr);
+	    }
 	    if (!(node = stringsubst(list, node, flags & PF_SINGLE, asssub))) {
 		unqueue_signals();
 		return;
@@ -92,9 +107,11 @@
 		    xpandbraces(list, &node);
 		}
 	    }
-	    if (unset(SHFILEEXPANSION))
-		filesub((char **)getaddrdata(node),
-			flags & (PF_TYPESET|PF_ASSIGN));
+	    if (unset(SHFILEEXPANSION)) {
+		char *cptr = (char *)getdata(node);
+		filesub(&cptr, flags & (PF_TYPESET|PF_ASSIGN));
+		setdata(node, cptr);
+	    }
 	} else if (!(flags & PF_SINGLE) && !keep)
 	    uremnode(list, node);
 	if (errflag) {
Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.134
diff -u -r1.134 utils.c
--- Src/utils.c	11 Aug 2006 21:30:38 -0000	1.134
+++ Src/utils.c	7 Sep 2006 18:48:25 -0000
@@ -911,10 +911,136 @@
     return 1;
 }
 
-/* extra functions to call before displaying the prompt */
+/*
+ * Extra functions to call before displaying the prompt.
+ * The data is a Prepromptfn.
+ */
+
+static LinkList prepromptfns;
+
+/* Add a function to the list of pre-prompt functions. */
+
+/**/
+mod_export void
+addprepromptfn(voidvoidfnptr_t func)
+{
+    Prepromptfn ppdat = (Prepromptfn)zalloc(sizeof(struct prepromptfn));
+    ppdat->func = func;
+    if (!prepromptfns)
+	prepromptfns = znewlinklist();
+    zaddlinknode(prepromptfns, ppdat);
+}
+
+/* Remove a function from the list of pre-prompt functions. */
+
+/**/
+mod_export void
+delprepromptfn(voidvoidfnptr_t func)
+{
+    LinkNode ln;
+
+    for (ln = firstnode(prepromptfns); ln; ln = nextnode(ln)) {
+	Prepromptfn ppdat = (Prepromptfn)getdata(ln);
+	if (ppdat->func == func) {
+	    (void)remnode(prepromptfns, ln);
+	    zfree(ppdat, sizeof(struct prepromptfn));
+	    return;
+	}
+    }
+#ifdef DEBUG
+    dputs("BUG: failed to delete node from prepromptfns");
+#endif
+}
+
+/*
+ * Functions to call at a particular time even if not at
+ * the prompt.  This is handled by zle.  The data is a
+ * Timedfn.  The functions must be in time order, but this
+ * is enforced by addtimedfn().
+ */
+
+/**/
+mod_export LinkList timedfns;
+
+/* Add a function to the list of timed functions. */
+
+/**/
+mod_export void
+addtimedfn(voidvoidfnptr_t func, time_t when)
+{
+    Timedfn tfdat = (Timedfn)zalloc(sizeof(struct timedfn));
+    tfdat->func = func;
+    tfdat->when = when;
+
+    if (!timedfns) {
+	timedfns = znewlinklist();
+	zaddlinknode(timedfns, tfdat);
+    } else {
+	LinkNode ln = firstnode(timedfns);
+
+	/*
+	 * Insert the new element in the linked list.  We do
+	 * rather too much work here since the standard
+	 * functions insert after a given node, whereas we
+	 * want to insert the new data before the first element
+	 * with a greater time.
+	 *
+	 * In practice, the only use of timed functions is
+	 * sched, which only adds the one function; so this
+	 * whole branch isn't used beyond the following block.
+	 */
+	if (!ln) {
+	    zaddlinknode(timedfns, tfdat);
+	    return;
+	}
+	for (;;) {
+	    Timedfn tfdat2;
+	    LinkNode next = nextnode(ln);
+	    if (!next) {
+		zaddlinknode(timedfns, tfdat);
+		return;
+	    }
+	    tfdat2 = (Timedfn)getdata(next);
+	    if (when < tfdat2->when) {
+		zinsertlinknode(timedfns, ln, tfdat);
+		return;
+	    }
+	    ln = next;
+	}
+    }
+}
+
+/*
+ * Delete a function from the list of timed functions.
+ * Note that if the function apperas multiple times only
+ * the first occurrence will be removed.
+ *
+ * Note also that when zle calls the function it does *not*
+ * automatically delete the entry from the list.  That must
+ * be done by the function called.  This is recommended as otherwise
+ * the function will keep being called immediately.  (It just so
+ * happens this "feature" fits in well with the only current use
+ * of timed functions.)
+ */
 
 /**/
-mod_export LinkList prepromptfns;
+mod_export void
+deltimedfn(voidvoidfnptr_t func)
+{
+    LinkNode ln;
+
+    for (ln = firstnode(timedfns); ln; ln = nextnode(ln)) {
+	Timedfn ppdat = (Timedfn)getdata(ln);
+	if (ppdat->func == func) {
+	    (void)remnode(timedfns, ln);
+	    zfree(ppdat, sizeof(struct timedfn));
+	    return;
+	}
+    }
+#ifdef DEBUG
+    dputs("BUG: failed to delete node from timedfns");
+#endif
+}
 
 /* the last time we checked mail */
 
@@ -1027,10 +1153,12 @@
 	lastmailcheck = time(NULL);
     }
 
-    /* Some people have claimed that C performs type    *
-     * checking, but they were later found to be lying. */
-    for(ln = firstnode(prepromptfns); ln; ln = nextnode(ln))
-	(**(void (**) _((void)))getdata(ln))();
+    if (prepromptfns) {
+	for(ln = firstnode(prepromptfns); ln; ln = nextnode(ln)) {
+	    Prepromptfn ppnode = (Prepromptfn)getdata(ln);
+	    ppnode->func();
+	}
+    }
 }
 
 /**/
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.95
diff -u -r1.95 zsh.h
--- Src/zsh.h	11 Aug 2006 21:30:38 -0000	1.95
+++ Src/zsh.h	7 Sep 2006 18:48:25 -0000
@@ -333,40 +333,38 @@
 /* Abstract types for zsh */
 /**************************/
 
-typedef struct linknode  *LinkNode;
-typedef union  linkroot  *LinkList;
-typedef struct hashnode  *HashNode;
-typedef struct hashtable *HashTable;
-
-typedef struct optname   *Optname;
-typedef struct reswd     *Reswd;
 typedef struct alias     *Alias;
-typedef struct param     *Param;
-typedef struct paramdef  *Paramdef;
+typedef struct asgment   *Asgment;
+typedef struct builtin   *Builtin;
 typedef struct cmdnam    *Cmdnam;
-typedef struct shfunc    *Shfunc;
+typedef struct complist  *Complist;
+typedef struct conddef   *Conddef;
 typedef struct funcstack *Funcstack;
 typedef struct funcwrap  *FuncWrap;
-typedef struct options	 *Options;
-typedef struct builtin   *Builtin;
-typedef struct nameddir  *Nameddir;
-typedef struct module    *Module;
-typedef struct linkedmod *Linkedmod;
-
-typedef struct patprog   *Patprog;
-typedef struct process   *Process;
-typedef struct job       *Job;
-typedef struct value     *Value;
-typedef struct conddef   *Conddef;
-typedef struct redir     *Redir;
-typedef struct complist  *Complist;
+typedef struct hashnode  *HashNode;
+typedef struct hashtable *HashTable;
 typedef struct heap      *Heap;
 typedef struct heapstack *Heapstack;
 typedef struct histent   *Histent;
 typedef struct hookdef   *Hookdef;
-
-typedef struct asgment   *Asgment;
-
+typedef struct job       *Job;
+typedef struct linkedmod *Linkedmod;
+typedef struct linknode  *LinkNode;
+typedef union  linkroot  *LinkList;
+typedef struct module    *Module;
+typedef struct nameddir  *Nameddir;
+typedef struct options	 *Options;
+typedef struct optname   *Optname;
+typedef struct param     *Param;
+typedef struct paramdef  *Paramdef;
+typedef struct patprog   *Patprog;
+typedef struct prepromptfn *Prepromptfn;
+typedef struct process   *Process;
+typedef struct redir     *Redir;
+typedef struct reswd     *Reswd;
+typedef struct shfunc    *Shfunc;
+typedef struct timedfn   *Timedfn;
+typedef struct value     *Value;
 
 /********************************/
 /* Definitions for linked lists */
@@ -432,6 +430,28 @@
         __n0.dat = (void *) (V0); \
     } while (0)
 
+/*************************************/
+/* Specific elements of linked lists */
+/*************************************/
+
+typedef void (*voidvoidfnptr_t) _((void));
+
+/*
+ * Element of the prepromptfns list.
+ */
+struct prepromptfn {
+    voidvoidfnptr_t func;
+};
+
+
+/*
+ * Element of the timedfns list.
+ */
+struct timedfn {
+    voidvoidfnptr_t func;
+    time_t when;
+};
+
 /********************************/
 /* Definitions for syntax trees */
 /********************************/
Index: Src/Builtins/sched.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Builtins/sched.c,v
retrieving revision 1.5
diff -u -r1.5 sched.c
--- Src/Builtins/sched.c	30 May 2006 22:35:03 -0000	1.5
+++ Src/Builtins/sched.c	7 Sep 2006 18:48:25 -0000
@@ -44,6 +44,49 @@
  
 static struct schedcmd *schedcmds;
 
+/* Check scheduled commands; call this function from time to time. */
+
+/**/
+static void
+checksched(void)
+{
+    time_t t;
+    struct schedcmd *sch;
+
+    if(!schedcmds)
+	return;
+    t = time(NULL);
+    /*
+     * List is ordered, so we only need to consider the
+     * head element.
+     */
+    while (schedcmds && schedcmds->time <= t) {
+	/*
+	 * Remove the entry to be executed from the list
+	 * before execution:  this makes quite sure that
+	 * the entry hasn't been monkeyed with when we
+	 * free it.
+	 */
+	sch = schedcmds;
+	schedcmds = sch->next;
+
+	execstring(sch->cmd, 0, 0);
+	zsfree(sch->cmd);
+	zfree(sch, sizeof(struct schedcmd));
+
+	/*
+	 * Fix time for future events.
+	 * I had this outside the loop, for a little extra efficiency.
+	 * However, it then occurred to me that having the list of
+	 * forthcoming entries up to date could be regarded as
+	 * a feature, and the inefficiency is negligible.
+	 */
+	deltimedfn(checksched);
+	if (schedcmds)
+	    addtimedfn(checksched, schedcmds->time);
+    }
+}
+
 /**/
 static int
 bin_sched(UNUSED(char *nam), char **argv, UNUSED(Options ops), UNUSED(int func))
@@ -64,13 +107,20 @@
 	    zwarnnam("sched", "usage for delete: sched -<item#>.");
 	    return 1;
 	}
-	for (schl = (struct schedcmd *)&schedcmds, sch = schedcmds, sn--;
+	for (schl = NULL, sch = schedcmds, sn--;
 	     sch && sn; sch = (schl = sch)->next, sn--);
 	if (!sch) {
 	    zwarnnam("sched", "not that many entries");
 	    return 1;
 	}
-	schl->next = sch->next;
+	if (schl)
+	    schl->next = sch->next;
+	else {
+	    deltimedfn(checksched);
+	    schedcmds = sch->next;
+	    if (schedcmds)
+		addtimedfn(checksched, schedcmds->time);
+	}
 	zsfree(sch->cmd);
 	zfree(sch, sizeof(struct schedcmd));
 
@@ -98,85 +148,95 @@
     /* The first argument specifies the time to schedule the command for.  The
     remaining arguments form the command. */
     if (*s == '+') {
-	/* + introduces a relative time.  The rest of the argument is an
-	hour:minute offset from the current time.  Once the hour and minute
-	numbers have been extracted, and the format verified, the resulting
-	offset is simply added to the current time. */
-	h = zstrtol(s + 1, &s, 10);
-	if (*s != ':') {
-	    zwarnnam("sched", "bad time specifier");
-	    return 1;
-	}
-	m = zstrtol(s + 1, &s, 10);
-	if (*s) {
+	/*
+	 * + introduces a relative time.  The rest of the argument may be an
+	 * hour:minute offset from the current time.  Once the hour and minute
+	 * numbers have been extracted, and the format verified, the resulting
+	 * offset is simply added to the current time.
+	 */
+	zlong zl = zstrtol(s + 1, &s, 10);
+	if (*s == ':') {
+	    m = zstrtol(s + 1, &s, 10);
+	    if (*s) {
+		zwarnnam("sched", "bad time specifier");
+		return 1;
+	    }
+	    t = time(NULL) + (long)zl * 3600 + m * 60;
+	} else if (!*s) {
+	    /*
+	     * Alternatively, it may simply be a number of seconds.
+	     * This is here for consistency with absolute times.
+	     */
+	    t = time(NULL) + (time_t)zl;
+	} else {
 	    zwarnnam("sched", "bad time specifier");
 	    return 1;
 	}
-	t = time(NULL) + h * 3600 + m * 60;
     } else {
-	/* If there is no +, an absolute time of day must have been given.
-	This is in hour:minute format, optionally followed by a string starting
-	with `a' or `p' (for a.m. or p.m.).  Characters after the `a' or `p'
-	are ignored. */
-	h = zstrtol(s, &s, 10);
-	if (*s != ':') {
-	    zwarnnam("sched", "bad time specifier");
-	    return 1;
-	}
-	m = zstrtol(s + 1, &s, 10);
-	if (*s && *s != 'a' && *s != 'A' && *s != 'p' && *s != 'P') {
+	/*
+	 * If there is no +, an absolute time must have been given.
+	 * This may be in hour:minute format, optionally followed by a string
+	 * starting with `a' or `p' (for a.m. or p.m.).  Characters after the
+	 * `a' or `p' are ignored.
+	 */
+	zlong zl = zstrtol(s, &s, 10);
+	if (*s == ':') {
+	    h = (long)zl;
+	    m = zstrtol(s + 1, &s, 10);
+	    if (*s && *s != 'a' && *s != 'A' && *s != 'p' && *s != 'P') {
+		zwarnnam("sched", "bad time specifier");
+		return 1;
+	    }
+	    t = time(NULL);
+	    tm = localtime(&t);
+	    t -= tm->tm_sec + tm->tm_min * 60 + tm->tm_hour * 3600;
+	    if (*s == 'p' || *s == 'P')
+		h += 12;
+	    t += h * 3600 + m * 60;
+	    /*
+	     * If the specified time is before the current time, it must refer
+	     * to tomorrow.
+	     */
+	    if (t < time(NULL))
+		t += 3600 * 24;
+	} else if (!*s) {
+	    /*
+	     * Otherwise, it must be a raw time specifier.
+	     */
+	    t = (long)zl;
+	} else {
 	    zwarnnam("sched", "bad time specifier");
 	    return 1;
 	}
-	t = time(NULL);
-	tm = localtime(&t);
-	t -= tm->tm_sec + tm->tm_min * 60 + tm->tm_hour * 3600;
-	if (*s == 'p' || *s == 'P')
-	    h += 12;
-	t += h * 3600 + m * 60;
-	/* If the specified time is before the current time, it must refer to
-	tomorrow. */
-	if (t < time(NULL))
-	    t += 3600 * 24;
     }
     /* The time has been calculated; now add the new entry to the linked list
     of scheduled commands. */
-    sch = (struct schedcmd *) zshcalloc(sizeof *sch);
+    sch = (struct schedcmd *) zalloc(sizeof *sch);
     sch->time = t;
     sch->cmd = zjoin(argv, ' ', 0);
-    sch->next = NULL;
-    for (sch2 = (struct schedcmd *)&schedcmds; sch2->next; sch2 = sch2->next);
-    sch2->next = sch;
-    return 0;
-}
-
-/* Check scheduled commands; call this function from time to time. */
-
-/**/
-static void
-checksched(void)
-{
-    time_t t;
-    struct schedcmd *sch, *schl;
-
-    if(!schedcmds)
-	return;
-    t = time(NULL);
-    for (schl = (struct schedcmd *)&schedcmds, sch = schedcmds; sch;
-	 sch = (schl = sch)->next) {
-	if (sch->time <= t) {
-	    execstring(sch->cmd, 0, 0);
-	    schl->next = sch->next;
-	    zsfree(sch->cmd);
-	    zfree(sch, sizeof(struct schedcmd));
-	    sch = schl;
+    /* Insert into list in time order */
+    if (schedcmds) {
+	if (sch->time < schedcmds->time) {
+	    deltimedfn(checksched);
+	    sch->next = schedcmds;
+	    schedcmds = sch;
+	    addtimedfn(checksched, t);
+	} else {
+	    for (sch2 = schedcmds;
+		 sch2->next && sch2->next->time < sch->time;
+		 sch2 = sch2->next)
+		;
+	    sch->next = sch2->next;
+	    sch2->next = sch;
 	}
+    } else {
+	sch->next = NULL;
+	schedcmds = sch;
+	addtimedfn(checksched, t);
     }
+    return 0;
 }
 
-static void (*p_checksched) _((void)) = checksched;
-static struct linknode n_checksched = { NULL, NULL, &p_checksched };
-
 static struct builtin bintab[] = {
     BUILTIN("sched", 0, bin_sched, 0, -1, 0, NULL, NULL),
 };
@@ -194,7 +254,7 @@
 {
     if(!addbuiltins(m->nam, bintab, sizeof(bintab)/sizeof(*bintab)))
 	return 1;
-    uaddlinknode(prepromptfns, &n_checksched);
+    addprepromptfn(&checksched);
     return 0;
 }
 
@@ -209,7 +269,7 @@
 	zsfree(sch->cmd);
 	zfree(sch, sizeof(*sch));
     }
-    uremnode(prepromptfns, &n_checksched);
+    delprepromptfn(&checksched);
     deletebuiltins(m->nam, bintab, sizeof(bintab)/sizeof(*bintab));
     return 0;
 }
Index: Src/Zle/zle_main.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_main.c,v
retrieving revision 1.89
diff -u -r1.89 zle_main.c
--- Src/Zle/zle_main.c	24 Jul 2006 22:00:21 -0000	1.89
+++ Src/Zle/zle_main.c	7 Sep 2006 18:48:26 -0000
@@ -122,7 +122,11 @@
 mod_export int eofchar;
 
 static int eofsent;
-static long keytimeout;
+/*
+ * Key timeout in hundredths of a second:  we use time_t so
+ * that we only have the limits on one integer type to worry about.
+ */
+static time_t keytimeout;
 
 #if defined(HAVE_SELECT) || defined(HAVE_POLL)
 /* Terminal baud rate */
@@ -387,11 +391,110 @@
 # define read    breakread
 #endif
 
+/*
+ * Possible forms of timeout.
+ */
+enum ztmouttp {
+    /* No timeout in use. */
+    ZTM_NONE,
+    /*
+     * Key timeout in use (do_keytmout flag set).  If this goes off
+     * we return without anything being read.
+     */
+    ZTM_KEY,
+    /*
+     * Function timeout in use (from timedfns list).
+     * If this goes off we call any functions which have reached
+     * the time and then continue processing.
+     */
+    ZTM_FUNC,
+    /*
+     * Timeout hit the maximum allowed; if it fires we
+     * need to recalculate.  As we may use poll() for the timeout,
+     * which takes an int value in milliseconds, we might need this
+     * for times long in the future.  (We make no attempt to extend
+     * the range of time beyond that of time_t, however; that seems
+     * like a losing battle.)
+     *
+     * For key timeouts we just limit the value to
+     * ZMAXTIMEOUT; that's already absurdly large.
+     *
+     * The following is the maximum signed range over 1024 (2^10), which
+     * is a little more convenient than 1000, but done differently
+     * to avoid problems with unsigned integers.  We assume 8-bit bytes;
+     * there's no general way to fix up if that's wrong.
+     */
+    ZTM_MAX
+#define	ZMAXTIMEOUT	((time_t)(1 << (sizeof(time_t)*8-11)))
+};
+
+struct ztmout {
+    /* Type of timeout setting, see enum above */
+    enum ztmouttp tp;
+    /*
+     * Value for timeout in 100ths of a second if type is not ZTM_NONE.
+     */
+    time_t exp100ths;
+};
+
+/*
+ * See if we need a timeout either for a key press or for a
+ * timed function.
+ */
+
+static void
+calc_timeout(struct ztmout *tmoutp, int do_keytmout)
+{
+    if (do_keytmout && keytimeout > 0) {
+	if (keytimeout > ZMAXTIMEOUT * 100 /* 24 days for a keypress???? */)
+	    tmoutp->exp100ths = ZMAXTIMEOUT * 100;
+	else
+	    tmoutp->exp100ths = keytimeout;
+	tmoutp->tp = ZTM_KEY;
+    } else
+	tmoutp->tp = ZTM_NONE;
+
+    if (timedfns) {
+	for (;;) {
+	    LinkNode tfnode = firstnode(timedfns);
+	    Timedfn tfdat;
+	    time_t diff, exp100ths;
+
+	    if (!tfnode)
+		break;
+
+	    tfdat = (Timedfn)getdata(tfnode);
+	    diff = tfdat->when - time(NULL);
+	    if (diff < 0) {
+		/* Already due; call it and rescan. */
+		tfdat->func();
+		continue;
+	    }
+
+	    if (diff > ZMAXTIMEOUT) {
+		tmoutp->exp100ths = ZMAXTIMEOUT * 100;
+		tmoutp->tp = ZTM_MAX;
+	    } else if (diff > 0) {
+		exp100ths = diff * 100;
+		if (tmoutp->tp != ZTM_KEY ||
+		    exp100ths < tmoutp->exp100ths) {
+		    tmoutp->exp100ths = exp100ths;
+		    tmoutp->tp = ZTM_FUNC;
+		}
+	    }
+	    break;
+	}
+	/* In case we called a function which messed up the display... */
+	if (resetneeded)
+	    zrefresh();
+    }
+}
+
 static int
-raw_getbyte(int keytmout, char *cptr)
+raw_getbyte(int do_keytmout, char *cptr)
 {
-    long exp100ths;
     int ret;
+    struct ztmout tmout;
 #if defined(HAS_TIO) && \
   (defined(sun) || (!defined(HAVE_POLL) && !defined(HAVE_SELECT)))
     struct ttyinfo ti;
@@ -402,204 +505,254 @@
 # endif
 #endif
 
+    calc_timeout(&tmout, do_keytmout);
+
     /*
-     * Handle timeouts and watched fd's.  We only do one at once;
-     * key timeouts take precedence.  This saves tricky timing
-     * problems with the key timeout.
+     * Handle timeouts and watched fd's.  If a watched fd or a function
+     * timeout triggers we restart any key timeout.  This is likely to
+     * be harmless: the combination is extremely rare and a function
+     * is likely to occupy the user for a little while anyway.  We used
+     * to make timeouts take precedence, but we can't now that the
+     * timeouts may be external, so we may have both a permanent watched
+     * fd and a long-term timeout.
      */
-    if ((nwatch || keytmout)
+    if ((nwatch || tmout.tp != ZTM_NONE)
 #ifdef FIONREAD
 	&& ! delayzsetterm
 #endif
 	) {
-	if (!keytmout || keytimeout <= 0)
-	    exp100ths = 0;
-	else if (keytimeout > 500)
-	    exp100ths = 500;
-	else
-	    exp100ths = keytimeout;
 #if defined(HAVE_SELECT) || defined(HAVE_POLL)
-	if (!keytmout || exp100ths) {
-	    int i, errtry = 0, selret;
+	int i, errtry = 0, selret;
 # ifdef HAVE_POLL
-	    int poll_timeout;
-	    int nfds;
-	    struct pollfd *fds;
-# else
-	    int fdmax;
-	    struct timeval *tvptr;
-	    struct timeval expire_tv;
+	int nfds;
+	struct pollfd *fds;
 # endif
 # if defined(HAS_TIO) && defined(sun)
-	    /*
-	     * Yes, I know this is complicated.  Yes, I know we
-	     * already have three bits of code to poll the terminal
-	     * down below.  No, I don't want to do this either.
-	     * However, it turns out on certain OSes, specifically
-	     * Solaris, that you can't poll typeahead for love nor
-	     * money without actually trying to read it.  But
-	     * if we are trying to select (and we need to if we
-	     * are watching other fd's) we won't pick that up.
-	     * So we just try and read it without blocking in
-	     * the time-honoured (i.e. absurdly baroque) termios
-	     * fashion.
-	     */
-	    gettyinfo(&ti);
-	    ti.tio.c_cc[VMIN] = 0;
-	    settyinfo(&ti);
-	    ret = read(SHTTY, cptr, 1);
-	    ti.tio.c_cc[VMIN] = 1;
-	    settyinfo(&ti);
-	    if (ret > 0)
-		return 1;
+	/*
+	 * Yes, I know this is complicated.  Yes, I know we
+	 * already have three bits of code to poll the terminal
+	 * down below.  No, I don't want to do this either.
+	 * However, it turns out on certain OSes, specifically
+	 * Solaris, that you can't poll typeahead for love nor
+	 * money without actually trying to read it.  But
+	 * if we are trying to select (and we need to if we
+	 * are watching other fd's) we won't pick that up.
+	 * So we just try and read it without blocking in
+	 * the time-honoured (i.e. absurdly baroque) termios
+	 * fashion.
+	 */
+	gettyinfo(&ti);
+	ti.tio.c_cc[VMIN] = 0;
+	settyinfo(&ti);
+	ret = read(SHTTY, cptr, 1);
+	ti.tio.c_cc[VMIN] = 1;
+	settyinfo(&ti);
+	if (ret > 0)
+	    return 1;
 # endif
 # ifdef HAVE_POLL
-	    nfds = keytmout ? 1 : 1 + nwatch;
-	    /* First pollfd is SHTTY, following are the nwatch fds */
-	    fds = zalloc(sizeof(struct pollfd) * nfds);
-	    if (exp100ths)
-		poll_timeout = exp100ths * 10;
+	nfds = 1 + nwatch;
+	/* First pollfd is SHTTY, following are the nwatch fds */
+	fds = zalloc(sizeof(struct pollfd) * nfds);
+	fds[0].fd = SHTTY;
+	/*
+	 * POLLIN, POLLIN, POLLIN,
+	 * Keep those fd's POLLIN...
+	 */
+	fds[0].events = POLLIN;
+	for (i = 0; i < nwatch; i++) {
+	    fds[i+1].fd = watch_fds[i];
+	    fds[i+1].events = POLLIN;
+	}
+# endif
+	do {
+# ifdef HAVE_POLL
+	    int poll_timeout;
+
+	    if (tmout.tp != ZTM_NONE)
+		poll_timeout = tmout.exp100ths * 10;
 	    else
 		poll_timeout = -1;
 
-	    fds[0].fd = SHTTY;
-	    /*
-	     * POLLIN, POLLIN, POLLIN,
-	     * Keep those fd's POLLIN...
-	     */
-	    fds[0].events = POLLIN;
-	    if (!keytmout) {
+	    selret = poll(fds, errtry ? 1 : nfds, poll_timeout);
+# else
+	    int fdmax = SHTTY;
+	    struct timeval *tvptr;
+	    struct timeval expire_tv;
+
+	    FD_ZERO(&foofd);
+	    FD_SET(SHTTY, &foofd);
+	    if (!errtry) {
 		for (i = 0; i < nwatch; i++) {
-		    fds[i+1].fd = watch_fds[i];
-		    fds[i+1].events = POLLIN;
+		    int fd = watch_fds[i];
+		    FD_SET(fd, &foofd);
+		    if (fd > fdmax)
+			fdmax = fd;
 		}
 	    }
-# else
-	    fdmax = SHTTY;
-	    tvptr = NULL;
-	    if (exp100ths) {
-		expire_tv.tv_sec = exp100ths / 100;
-		expire_tv.tv_usec = (exp100ths % 100) * 10000L;
+
+	    if (tmout.tp != ZTM_NONE) {
+		expire_tv.tv_sec = tmout.exp100ths / 100;
+		expire_tv.tv_usec = (tmout.exp100ths % 100) * 10000L;
 		tvptr = &expire_tv;
 	    }
+	    else
+		tvptr = NULL;
+
+	    selret = select(fdmax+1, (SELECT_ARG_2_T) & foofd,
+			    NULL, NULL, tvptr);
 # endif
-	    do {
-# ifdef HAVE_POLL
-		selret = poll(fds, errtry ? 1 : nfds, poll_timeout);
-# else
-		FD_ZERO(&foofd);
-		FD_SET(SHTTY, &foofd);
-		if (!keytmout && !errtry) {
-		    for (i = 0; i < nwatch; i++) {
-			int fd = watch_fds[i];
-			FD_SET(fd, &foofd);
-			if (fd > fdmax)
-			    fdmax = fd;
-		    }
-		}
-		selret = select(fdmax+1, (SELECT_ARG_2_T) & foofd,
-				NULL, NULL, tvptr);
-# endif
-		/*
-		 * Make sure a user interrupt gets passed on straight away.
-		 */
-		if (selret < 0 && errflag)
-		    break;
+	    /*
+	     * Make sure a user interrupt gets passed on straight away.
+	     */
+	    if (selret < 0 && errflag)
+		break;
+	    /*
+	     * Try to avoid errors on our special fd's from
+	     * messing up reads from the terminal.  Try first
+	     * with all fds, then try unsetting the special ones.
+	     */
+	    if (selret < 0 && !errtry) {
+		errtry = 1;
+		continue;
+	    }
+	    if (selret == 0) {
 		/*
-		 * Try to avoid errors on our special fd's from
-		 * messing up reads from the terminal.  Try first
-		 * with all fds, then try unsetting the special ones.
+		 * Nothing ready and no error, so we timed out.
 		 */
-		if (selret < 0 && !keytmout && !errtry) {
-		    errtry = 1;
-		    continue;
-		}
-		if (selret == 0) {
+		switch (tmout.tp) {
+		case ZTM_NONE:
+		    /* keeps compiler happy if not debugging */
+#ifdef DEBUG
+		    dputs("BUG: timeout fired with no timeout set.");
+#endif
+		    /* treat as if a key timeout triggered */
+		    /*FALLTHROUGH*/
+		case ZTM_KEY:
 		    /* Special value -2 signals nothing ready */
 		    selret = -2;
-		}
-		if (selret < 0)
 		    break;
-		if (!keytmout && nwatch) {
+
+		case ZTM_FUNC:
+		    while (firstnode(timedfns)) {
+			Timedfn tfdat = (Timedfn)getdata(firstnode(timedfns));
+			/*
+			 * It's possible a previous function took
+			 * a long time to run (though it can't
+			 * call zle recursively), so recalculate
+			 * the time on each iteration.
+			 */
+			time_t now = time(NULL);
+			if (tfdat->when > now)
+			    break;
+			tfdat->func();
+		    }
+		    /* Function may have messed up the display */
+		    if (resetneeded)
+			zrefresh();
+		    /* We need to recalculate the timeout */
+		    /*FALLTHROUGH*/
+		case ZTM_MAX:
 		    /*
-		     * Copy the details of the watch fds in case the
-		     * user decides to delete one from inside the
-		     * handler function.
+		     * Reached the limit of our range, but not the
+		     * actual timeout; recalculate the timeout.
+		     * We're cheating with the key timeout here:
+		     * if one clashed with a function timeout we
+		     * reconsider the key timeout from scratch.
+		     * The effect of this is microscopic.
 		     */
-		    int lnwatch = nwatch;
-		    int *lwatch_fds = zalloc(lnwatch*sizeof(int));
-		    char **lwatch_funcs = zarrdup(watch_funcs);
-		    memcpy(lwatch_fds, watch_fds, lnwatch*sizeof(int));
-		    for (i = 0; i < lnwatch; i++) {
-			if (
+		    calc_timeout(&tmout, do_keytmout);
+		    break;
+		}
+		/*
+		 * If we handled the timeout successfully,
+		 * carry on.
+		 */
+		if (selret == 0)
+		    continue;
+	    }
+	    /* If error or unhandled timeout, give up. */
+	    if (selret < 0)
+		break;
+	    if (nwatch && !errtry) {
+		/*
+		 * Copy the details of the watch fds in case the
+		 * user decides to delete one from inside the
+		 * handler function.
+		 */
+		int lnwatch = nwatch;
+		int *lwatch_fds = zalloc(lnwatch*sizeof(int));
+		char **lwatch_funcs = zarrdup(watch_funcs);
+		memcpy(lwatch_fds, watch_fds, lnwatch*sizeof(int));
+		for (i = 0; i < lnwatch; i++) {
+		    if (
 # ifdef HAVE_POLL
-			    (fds[i+1].revents & POLLIN)
+			(fds[i+1].revents & POLLIN)
 # else
-			    FD_ISSET(lwatch_fds[i], &foofd)
+			FD_ISSET(lwatch_fds[i], &foofd)
 # endif
-			    ) {
-			    /* Handle the fd. */
-			    LinkList funcargs = znewlinklist();
-			    zaddlinknode(funcargs, ztrdup(lwatch_funcs[i]));
-			    {
-				char buf[BDIGBUFSIZE];
-				convbase(buf, lwatch_fds[i], 10);
-				zaddlinknode(funcargs, ztrdup(buf));
-			    }
+			) {
+			/* Handle the fd. */
+			LinkList funcargs = znewlinklist();
+			zaddlinknode(funcargs, ztrdup(lwatch_funcs[i]));
+			{
+			    char buf[BDIGBUFSIZE];
+			    convbase(buf, lwatch_fds[i], 10);
+			    zaddlinknode(funcargs, ztrdup(buf));
+			}
 # ifdef HAVE_POLL
 #  ifdef POLLERR
-			    if (fds[i+1].revents & POLLERR)
-				zaddlinknode(funcargs, ztrdup("err"));
+			if (fds[i+1].revents & POLLERR)
+			    zaddlinknode(funcargs, ztrdup("err"));
 #  endif
 #  ifdef POLLHUP
-			    if (fds[i+1].revents & POLLHUP)
-				zaddlinknode(funcargs, ztrdup("hup"));
+			if (fds[i+1].revents & POLLHUP)
+			    zaddlinknode(funcargs, ztrdup("hup"));
 #  endif
 #  ifdef POLLNVAL
-			    if (fds[i+1].revents & POLLNVAL)
-				zaddlinknode(funcargs, ztrdup("nval"));
+			if (fds[i+1].revents & POLLNVAL)
+			    zaddlinknode(funcargs, ztrdup("nval"));
 #  endif
 # endif
 
 
-			    callhookfunc(lwatch_funcs[i], funcargs);
-			    if (errflag) {
-				/* No sensible way of handling errors here */
-				errflag = 0;
-				/*
-				 * Paranoia: don't run the hooks again this
-				 * time.
-				 */
-				errtry = 1;
-			    }
-			    freelinklist(funcargs, freestr);
+			callhookfunc(lwatch_funcs[i], funcargs);
+			if (errflag) {
+			    /* No sensible way of handling errors here */
+			    errflag = 0;
+			    /*
+			     * Paranoia: don't run the hooks again this
+			     * time.
+			     */
+			    errtry = 1;
 			}
+			freelinklist(funcargs, freestr);
 		    }
-		    /* Function may have invalidated the display. */
-		    if (resetneeded)
-			zrefresh();
-		    zfree(lwatch_fds, lnwatch*sizeof(int));
-		    freearray(lwatch_funcs);
 		}
-	    } while (!
+		/* Function may have invalidated the display. */
+		if (resetneeded)
+		    zrefresh();
+		zfree(lwatch_fds, lnwatch*sizeof(int));
+		freearray(lwatch_funcs);
+	    }
+	} while (!
 # ifdef HAVE_POLL
-		     (fds[0].revents & POLLIN)
+		 (fds[0].revents & POLLIN)
 # else
-		     FD_ISSET(SHTTY, &foofd)
+		 FD_ISSET(SHTTY, &foofd)
 # endif
-		);
+		 );
 # ifdef HAVE_POLL
-	    zfree(fds, sizeof(struct pollfd) * nfds);
+	zfree(fds, sizeof(struct pollfd) * nfds);
 # endif
-	    if (selret < 0)
-		return selret;
-	}
+	if (selret < 0)
+	    return selret;
 #else
 # ifdef HAS_TIO
 	ti = shttyinfo;
 	ti.tio.c_lflag &= ~ICANON;
 	ti.tio.c_cc[VMIN] = 0;
-	ti.tio.c_cc[VTIME] = exp100ths / 10;
+	ti.tio.c_cc[VTIME] = tmout.exp100ths / 10;
 #  ifdef HAVE_TERMIOS_H
 	tcsetattr(SHTTY, TCSANOW, &ti.tio);
 #  else
@@ -623,7 +776,7 @@
 
 /**/
 mod_export int
-getbyte(int keytmout, int *timeout)
+getbyte(int do_keytmout, int *timeout)
 {
     char cc;
     unsigned int ret;
@@ -656,7 +809,7 @@
 	for (;;) {
 	    int q = queue_signal_level();
 	    dont_queue_signals();
-	    r = raw_getbyte(keytmout, &cc);
+	    r = raw_getbyte(do_keytmout, &cc);
 	    restore_queue_signals(q);
 	    if (r == -2) {
 		/* timeout */
@@ -733,9 +886,9 @@
 
 /**/
 mod_export ZLE_INT_T
-getfullchar(int keytmout)
+getfullchar(int do_keytmout)
 {
-    int inchar = getbyte(keytmout, NULL);
+    int inchar = getbyte(do_keytmout, NULL);
 
 #ifdef MULTIBYTE_SUPPORT
     return getrestchar(inchar);
@@ -938,7 +1091,7 @@
 	return shingetline();
     }
 
-    keytimeout = getiparam("KEYTIMEOUT");
+    keytimeout = (time_t)getiparam("KEYTIMEOUT");
     if (!shout) {
 	if (SHTTY != -1)
 	    init_shout();
@@ -1551,7 +1704,7 @@
 mod_export void
 trashzle(void)
 {
-    if (zleactive) {
+    if (zleactive && !trashedzle) {
 	/* This zrefresh() is just to get the main editor display right and *
 	 * get the cursor in the right place.  For that reason, we disable  *
 	 * list display (which would otherwise result in infinite           *
Index: Src/Zle/zle_thingy.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_thingy.c,v
retrieving revision 1.29
diff -u -r1.29 zle_thingy.c
--- Src/Zle/zle_thingy.c	30 May 2006 22:35:04 -0000	1.29
+++ Src/Zle/zle_thingy.c	7 Sep 2006 18:48:26 -0000
@@ -721,8 +721,7 @@
      * true if a completion widget is active.
      */
     if (zleactive) {
-	if (!trashedzle)
-	    trashzle();
+	trashzle();
 	return 0;
     } else
 	return 1;

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug#335481: zsh: zsh/sched waits for next return to prompt even with NOTIFY set

[Bart Schaefer]

On Sep 7,  8:25pm, Peter Stephenson wrote:
}
} I got a chance to look at it.  As we already do quite sophisticated
} things while the line editor is waiting for shell input (key timeouts
} and polling of file descriptors) it's not so difficult to expand this so
} that a sched timer can fire at this point.  I haven't looked at
} extending this to be fully asynchronous: it's not clear you want that

It's pretty clear to me that one *doesn't*, actually.  What would be a
useful alternative is to be able to do something like

	(sched 01:05 print "It's 1:05 AM, go to bed")&

and have the subshell hang around until either the parent shell exits
or the scheduled time arrives.  That probably needs an option to the
sched command, perhaps -w (for wait) or -s (for sleep).  If this were
possible, maybe a lot of the other changes you made wouldn't be
necessary, but ...

} sched events will now fire while waiting for user input in zle.

Presumably this means they behave as if prefixed with "zle -I".  I
see that you've made "print" intelligent in this way, but what if the
sched job invokes an external command that outputs to the terminal?
Why shouldn't "sched" be what takes care of "zle -I" rather than
having it built into "print"?  (Given a choice of directions, I'd
rather make the main shell less aware of ZLE rather than more.)

I realize this might mean that "sched" itself needs a -t option to
turn off the implicit "zle -I".

Jumping ahead a bit:

} The only problem with this I can see is if the "print" is only being
} used, say, to update a terminal emulator title bar, and the likely of
} doing that at a point where you want to make it invisible to the user
} that a function is being run isn't that great anyway so I can't see a
} reason why this shouldn't be the default.

I update a clock in my title bar with a TRAPALRM function, and in fact
I'd suspect that this is the sort of thing for which lots of people
would rather use sched if it had always worked the way this patch
changes it to work.  I confess I'm not precisely clear whether that
falls under the category of "print"s that would or would not need -t,
but I'd be unhappy to have the default stop behaving the way I expect
(and the way it has behaved for me now for lo-these-many-years).

Also, it's unclear whether "echo" is affected by the change to "print"?

} As a side effect of the new mechanism I've upped the arbitrary limit on
} key timeouts, which used to be 5 seconds, to (typically) 24 days, which
} ought to be long enough

Long enough for what?  I'm stumped as to the rationale for this.  If
it's because the scheduled task might begin doing something for a
long time right in the middle of key sequence, I think it would make
much more sense to keep the key timeout short and prevent processing
sched items during the wait for the "rest" of a sequence.

} sched now accepts a number of seconds (when there's no ":" in the time)

This is nice, so why not also allow it to accept HH:MM:SS too?

} Internally, I've done the following:

Those all sound like worthwhile improvements to me.

Re: Bug#335481: zsh: zsh/sched waits for next return to prompt even with NOTIFY set

[Peter Stephenson]

Bart Schaefer wrote:
> } sched events will now fire while waiting for user input in zle.
> 
> Presumably this means they behave as if prefixed with "zle -I".  I
> see that you've made "print" intelligent in this way, but what if the
> sched job invokes an external command that outputs to the terminal?
> Why shouldn't "sched" be what takes care of "zle -I" rather than
> having it built into "print"?  (Given a choice of directions, I'd
> rather make the main shell less aware of ZLE rather than more.)

sched doesn't know if you're doing output to the terminal at all.

> I realize this might mean that "sched" itself needs a -t option to
> turn off the implicit "zle -I".

Or on.  That was my first thought, but I wondered if it could be made
more intelligent.

> Jumping ahead a bit:
> 
> } The only problem with this I can see is if the "print" is only being
> } used, say, to update a terminal emulator title bar, and the likely of
> } doing that at a point where you want to make it invisible to the user
> } that a function is being run isn't that great anyway so I can't see a
> } reason why this shouldn't be the default.
> 
> I update a clock in my title bar with a TRAPALRM function, and in fact
> I'd suspect that this is the sort of thing for which lots of people
> would rather use sched if it had always worked the way this patch
> changes it to work.  I confess I'm not precisely clear whether that
> falls under the category of "print"s that would or would not need -t,
> but I'd be unhappy to have the default stop behaving the way I expect
> (and the way it has behaved for me now for lo-these-many-years).

That looks like a problem.  Maybe it needs to be an option to sched, in
that case; that's the simplest thing to do and I'll change it.  Of
course you can use a function with zle -I in it, but that makes using
sched for simple output a little too difficult.

> Also, it's unclear whether "echo" is affected by the change to "print"?

Currently it is (and this is documented under "print -t", but not for
"echo" itself), but it doesn't need to be.  This will go away if I move
the behaviour to sched.

> } As a side effect of the new mechanism I've upped the arbitrary limit on
> } key timeouts, which used to be 5 seconds, to (typically) 24 days, which
> } ought to be long enough
> 
> Long enough for what?  I'm stumped as to the rationale for this.  If
> it's because the scheduled task might begin doing something for a
> long time right in the middle of key sequence, I think it would make
> much more sense to keep the key timeout short and prevent processing
> sched items during the wait for the "rest" of a sequence.

I haven't change any key timeout at all, I've changed the *limit*.  It's
now long enough for any conceivable use of a key timeout.  The 5 second
limit was arbitrary but arguably not long enough for any *conceivable*
use.  The change doesn't show up unless you set KEYTIMEOUT to something
more than 500.  This saves me from having two different limits built
into the code since I need one for restarting the timeout for sched
events long in the future (which are probably atypical but there's no
good reason why they shouldn't work).

> } sched now accepts a number of seconds (when there's no ":" in the time)
> 
> This is nice, so why not also allow it to accept HH:MM:SS too?

I didn't see a good use for this, but it's not a very difficult change.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug#335481: zsh: zsh/sched waits for next return to prompt even with NOTIFY set

[Peter Stephenson]

Since there seem to be no more comments I've committed the sched changes
in the modified form: print is unchanged and you need to specify "sched
-o" to make the event clear and redraw a zle command line around output.
Maybe there's a better option, but o for output seemed an obvious one.
"sched HH:MM:SS cmd..." is allowed as well.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

PATCH: $! on bg

[Peter Stephenson]

A thread on the Austin Group mailing list suggests we should be setting
$! when a process is put into the background, not just when a process
is started in the background.  Bash already does this.  (There's some
other grungy stuff about $! on the list I'm less concerned with.)

It's hard to see any compatilibity problems arising with this since bg
is mostly used at the command line, but maybe Bart or someone else can
think of problems.

This updates the documentation, too.  It was probably intended to mean
what the shell actually did, i.e. set $! on "... &" only, although one
could interpret running "bg" as "invoking a background command".

I haven't attempted to lever this into Sven's hack whereby the
shell can fork and allow you to put a job that was running
into the current shell into the background; however, it "might just
work", since the fudged job should already be in the job table by the
time you do "bg".

Index: Doc/Zsh/params.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/params.yo,v
retrieving revision 1.34
diff -u -r1.34 params.yo
--- Doc/Zsh/params.yo	5 Oct 2006 13:28:24 -0000	1.34
+++ Doc/Zsh/params.yo	26 Oct 2006 17:24:12 -0000
@@ -477,7 +477,8 @@
 startitem()
 vindex(!)
 item(tt(!) <S>)(
-The process ID of the last background command invoked.
+The process ID of the last command started in the background with tt(&),
+or put into the background with the tt(bg) builtin.
 )
 vindex(#)
 item(tt(#) <S>)(
Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.47
diff -u -r1.47 jobs.c
--- Src/jobs.c	28 Jun 2006 13:12:55 -0000	1.47
+++ Src/jobs.c	26 Oct 2006 17:24:12 -0000
@@ -1789,9 +1789,21 @@
 	case BIN_WAIT:
 	    if (func == BIN_BG)
 		jobtab[job].stat |= STAT_NOSTTY;
-	    if ((stopped = (jobtab[job].stat & STAT_STOPPED)))
+	    if ((stopped = (jobtab[job].stat & STAT_STOPPED))) {
 		makerunning(jobtab + job);
-	    else if (func == BIN_BG) {
+		if (func == BIN_BG) {
+		    /* Set $! to indicate this was backgrounded */
+		    Process pn = jobtab[job].procs;
+		    for (;;) {
+			Process next = pn->next;
+			if (!next) {
+			    lastpid = (zlong) pn->pid;
+			    break;
+			}
+			pn = next;
+		    }
+		}
+	    } else if (func == BIN_BG) {
 		/* Silly to bg a job already running. */
 		zwarnnam(name, "job already in background");
 		thisjob = ocj;

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: PATCH: $! on bg

[Bart Schaefer]

On Oct 26,  6:39pm, Peter Stephenson wrote:
}
} It's hard to see any compatilibity problems arising with this since bg
} is mostly used at the command line, but maybe Bart or someone else can
} think of problems.

The only case where this could possibly cause confusion is if someone
has deliberately started a non-interactive shell with "setopt monitor".
I don't think that's frequent enough, or the resulting problem severe
enough, to worry about.
 
} I haven't attempted to lever this into Sven's hack whereby the
} shell can fork and allow you to put a job that was running
} into the current shell into the background; however, it "might just
} work", since the fudged job should already be in the job table by the
} time you do "bg".

No, it does not appear to work.  Try e.g.

zsh% while sleep 10; do : ; done
^Z
zsh: suspended  while sleep 10; do; :; done
zsh% bg ; echo $!
0
zsh% 

Re: PATCH: $! on bg

[Peter Stephenson]

Bart Schaefer wrote:
> } I haven't attempted to lever this into Sven's hack whereby the
> } shell can fork and allow you to put a job that was running
> } into the current shell into the background; however, it "might just
> } work", since the fudged job should already be in the job table by the
> } time you do "bg".
> 
> No, it does not appear to work.

I meant after the patch, though I didn't make that clear.  It works for me:

% while sleep 10; do : ; done
[1]  + 3070 suspended  while sleep 10; do; :; done
% bg
[1]  + 3070 continued  while sleep 10; do; :; done
% print $!
3070

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: $! on bg

[Bart Schaefer]

On Oct 29, 12:58pm, Peter Stephenson wrote:
} Subject: Re: PATCH: $! on bg
}
} Bart Schaefer wrote:
} > No, it does not appear to work.
} 
} I meant after the patch

I thought I did, too, but I just realized you haven't yet committed the
patch, so I wasn't testing what I thought I was.

Re: PATCH: $! on bg

[Peter Stephenson]

Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Oct 29, 12:58pm, Peter Stephenson wrote:
> } Subject: Re: PATCH: $! on bg
> }
> } Bart Schaefer wrote:
> } > No, it does not appear to work.
> } 
> } I meant after the patch
> 
> I thought I did, too, but I just realized you haven't yet committed the
> patch, so I wasn't testing what I thought I was.

I was waiting for comments.  I've committed it now.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

error code for failure to execute

[Clint Adams]

Does this make sense?

$ touch /tmp/nonex
$ chmod 0 /tmp/nonex
$ (/tmp/nonex) 2>/dev/null || echo $?
127
$ (/tmp/nonex)
zsh: permission denied: /tmp/nonex
$ echo $?
126
$ (/tmp/nonex) 2>/dev/null
$ echo $?
127
$ 

Re: error code for failure to execute

[Peter Stephenson]

Clint Adams wrote:
> Does this make sense?
> 
> $ touch /tmp/nonex
> $ chmod 0 /tmp/nonex
> $ (/tmp/nonex) 2>/dev/null || echo $?
> 127
> $ (/tmp/nonex)
> zsh: permission denied: /tmp/nonex
> $ echo $?
> 126
> $ (/tmp/nonex) 2>/dev/null
> $ echo $?
> 127
> $ 

Yes, I think so, but I don't really understand the question.
Are you asking about the following?

http://www.opengroup.org/onlinepubs/009695399/toc.htm

 2.8.2 Exit Status for Commands

...

If a command is not found, the exit status shall be 127. If the command
name is found, but it is not an executable utility, the exit status
shall be 126. Applications that invoke utilities without using the shell
should use these exit status values to report similar errors.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: error code for failure to execute

[Clint Adams]

> Yes, I think so, but I don't really understand the question.
> Are you asking about the following?

No, I'm confused by why it changes from 126 to 127 when stderr is
redirected to /dev/null.

Re: error code for failure to execute

[Peter Stephenson]

Clint Adams wrote:
> > Yes, I think so, but I don't really understand the question.
> > Are you asking about the following?
> 
> No, I'm confused by why it changes from 126 to 127 when stderr is
> redirected to /dev/null.

Ah.

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.105
diff -u -u -r1.105 exec.c
--- Src/exec.c	6 Nov 2006 12:49:21 -0000	1.105
+++ Src/exec.c	19 Nov 2006 21:32:03 -0000
@@ -522,12 +522,12 @@
     }
     for (s = arg0; *s; s++)
 	if (*s == '/') {
-	    errno = zexecve(arg0, argv);
+	    int lerrno = zexecve(arg0, argv);
 	    if (arg0 == s || unset(PATHDIRS) ||
 		(arg0[0] == '.' && (arg0 + 1 == s ||
 				    (arg0[1] == '.' && arg0 + 2 == s)))) {
-		zerr("%e: %s", errno, arg0);
-		_exit((errno == EACCES || errno == ENOEXEC) ? 126 : 127);
+		zerr("%e: %s", lerrno, arg0);
+		_exit((lerrno == EACCES || lerrno == ENOEXEC) ? 126 : 127);
 	    }
 	    break;
 	}

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

PATCH: _firefox

[Peter Stephenson]

Not sure why we don't have this already, except that the command line
options aren't very well documented.

Index: Completion/Unix/Command/.distfiles
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Command/.distfiles,v
retrieving revision 1.80
diff -u -r1.80 .distfiles
--- Completion/Unix/Command/.distfiles	25 Oct 2006 13:13:23 -0000	1.80
+++ Completion/Unix/Command/.distfiles	8 Dec 2006 21:15:27 -0000
@@ -10,6 +10,7 @@
 _dict           _diff           _diffstat       _du             _dvi
 _ecasound       _elinks         _elm            _enscript       _env
 _fakeroot       _fetchmail      _figlet         _find           _finger
+_firefox
 _flasher        _flex           _fortune        _fsh            _fuser
 _gcc            _gdb            _getconf        _getent         _git
 _getmail
Index: Completion/Unix/Command/_firefox
===================================================================
RCS file: Completion/Unix/Command/_firefox
diff -N Completion/Unix/Command/_firefox
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Completion/Unix/Command/_firefox	8 Dec 2006 21:15:27 -0000
@@ -0,0 +1,46 @@
+#compdef firefox
+
+local context state line expl
+local -A opt_args
+
+_arguments \
+    '--display[pick X display]:display:_x_display' \
+    '--sync[make X calls synchronous]' \
+    '--no-xshm[no X shared memory extension]' \
+    '--xim-preedit=:style: ' \
+    '--xim-status=:style: ' \
+    '--g-fatal-warnings[make all warnings fatal]' \
+    '-browser[open browser window]' \
+    '-chrome[specify chrome]:load specified chrome:_urls' \
+    '-console[start with debugging console]' \
+    '-contentLocale=[select local for content]:locale:_locales' \
+    '-CreateProfile[create a profile]:profile name: ' \
+    '-h[output help message]' \
+    '-height[set initial window height]:height: ' \
+    '-help[output help message]' \
+    '-inspector[start with DOM Inspector]:URL to inspect:->location' \
+    '-install-global-extension[install an extension]:extension to install:_files' \
+    '-jsconsole[start with Javascript console]' \
+    '-new-window[load URL in new window]:URL to load:->location' \
+    '-new-tab[load URL in new tab]:URL to load:->location' \
+    '-no-remote[run with multiple profiles]' \
+    '-P[start with given profile]:profile:->profile' \
+    '-profile[specify profile file]:profile file:_files' \
+    '-ProfileManager[start with profile manager]' \
+    '-UILocale[select local for UI]:locale:_locales' \
+    '-v[show version]' \
+    '-version[show version]' \
+    '-width[set initial window width]:width: ' \
+    '*:location:->location' && return
+
+case $state in
+    (location)
+    _alternative 'files:file:_files' 'urls:URL:_urls'
+    ;;
+
+    (profile)
+    _wanted profiles expl profile compadd --
+    $(sed -n -e 's/^\[General\]//' -e 's/^\[\(.*\)\]$/\1/p' \
+	~/.mozilla/firefox/profiles.ini)
+    ;;
+esac
Index: Completion/Unix/Command/_webbrowser
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Command/_webbrowser,v
retrieving revision 1.7
diff -u -r1.7 _webbrowser
--- Completion/Unix/Command/_webbrowser	21 May 2005 06:55:35 -0000	1.7
+++ Completion/Unix/Command/_webbrowser	8 Dec 2006 21:15:27 -0000
@@ -1,3 +1,3 @@
-#compdef amaya arena chimera dillo express firefox galeon grail gzilla hotjava konqueror light mmm Mosaic netrik opera phoenix retawq skipstone www xmosaic zen
+#compdef amaya arena chimera dillo express galeon grail gzilla hotjava konqueror light mmm Mosaic netrik opera phoenix retawq skipstone www xmosaic zen
 
 _alternative 'files:file:_files' 'urls:URL:_urls'

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: _firefox

[Peter Stephenson]

Peter Stephenson wrote:
> Not sure why we don't have this already, except that the command line
> options aren't very well documented.

Ah, I see.  _webbrowser shouldn't claim to handle firefox since
X/Command/_mozilla does, so the fix is simply to remove it from _webbrowser.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: _firefox

[Clint Adams]

> Ah, I see.  _webbrowser shouldn't claim to handle firefox since
> X/Command/_mozilla does, so the fix is simply to remove it from _webbrowser.

In that case,

Index: Completion/X/Command/_mozilla
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/X/Command/_mozilla,v
retrieving revision 1.7
diff -u -r1.7 _mozilla
--- Completion/X/Command/_mozilla	15 Aug 2005 14:39:54 -0000	1.7
+++ Completion/X/Command/_mozilla	8 Dec 2006 21:52:15 -0000
@@ -1,4 +1,4 @@
-#compdef mozilla mozilla-firefox mozilla-xremote-client firefox
+#compdef mozilla mozilla-firefox mozilla-xremote-client firefox iceweasel=firefox
 
 local curcontext="$curcontext" state line expl ret=1 suf
 typeset -A opt_args

Is wait not interruptable?

[Dave Yost]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

Hi.

This program isn't working as I would expect:

#!/bin/zsh

(sleep 3 ; echo A) &
proc1=$!
(sleep 4 ; echo B) &
proc2=$!
trap 'echo kill $proc2 $proc1 ; kill $proc2 $proc1' INT
wait $proc1
kill $proc2
#ps

Here's what it does:

315 Z% wait-kill-test
A
316 Z% wait-kill-test
^Ckill 4099 4098
317 Z% echo $ZSH_VERSION
4.2.3
318 Z% uname -a
Darwin ip2 8.8.2 Darwin Kernel Version 8.8.2: Thu Sep 28 20:43:26 PDT 
2006; root:xnu-792.14.14.obj~1/RELEASE_I386 i386 i386
319 Z%

In the second run, the ^C was typed 1 second after the command was 
started, and the first sleep plodded along to completion for its full 
3 seconds.  Why wasn't the first sleep killed by the kill command in 
the trap?  This is on Mac OS X 10.4.8.

When I run this code with Bash 2.05b.0(1)-release, the ^C does cause 
the first sleep to be killed instantly.

I need this to work correctly for an enhancement to
   http://Yost.com/computers/d
That will allow you to say that you want to tail -f the output until 
the backgrounded completes or until you type ^C.

Thanks

Dave

[-- Attachment #2: Type: text/html, Size: 1994 bytes --]

Re: Is wait not interruptable?

[Peter Stephenson]

Dave Yost <Dave@Yost.com> wrote:
> This program isn't working as I would expect:
> 
> #!/bin/zsh
> 
> (sleep 3 ; echo A) &
> proc1=$!
> (sleep 4 ; echo B) &
> proc2=$!
> trap 'echo kill $proc2 $proc1 ; kill $proc2 $proc1' INT
> wait $proc1
> kill $proc2
> #ps
> 
> 316 Z% wait-kill-test
> ^Ckill 4099 4098
>
> The ^C was typed 1 second after the command was 
> started, and the first sleep plodded along to completion for its full 
> 3 seconds.

There are two things here, I think.  Unfortunately I'm not a signal expert;
we could do with one to maintain this part of the code.  Bart will probably
be able to shed some more light on the following.

The first is that the subshell itself (the stuff in parentheses) doesn't
get killed because it's waiting for the child process.  This seems to be a
feature of SIGTERM.  If you "setopt trapsasync" or send a different signal
such as SIGHUP it does get killed.  I'm not really sure why SIGTERM is
different.

The second is that even if the subshell gets killed, the sleep processes
themselves don't.  This seems to be because (a) the NOHUP option is on by
default (b) even with "setopt hup" we (i.e. in this case the subshell that
is exiting after the first problem is worked around) don't send HUP to
processes when exiting unless the exiting shell is interactive.

I make no warranty that any of the behaviour I've found is correct, since
signal handling confuses me.

I *think* the patch below fixes (b) by sending SIGHUP to our own process
group when exiting from a non-interactive shell, but I'll take advice on
this.  (I can at least then get the test program to work with "setopt hup
trapsasync".)  The manual isn't clear over whether this is supposed to
happen non-interactively, though it would probably be fair to assume that
if HUP is set in a non-interactive shell you'd expect the SIGHUP to be
sent.  I don't know if I should signal_ignore(SIGHUP) before killpg(0,
SIGHUP) but it didn't seem to be necessary.

The Src/jobs.c hunk is a separate thing I noticed and will commit anyway:
the error message for "kill -" (no option or signal) is confusing.


Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.171
diff -u -r1.171 builtin.c
--- Src/builtin.c	10 Dec 2006 23:27:03 -0000	1.171
+++ Src/builtin.c	15 Dec 2006 11:44:38 -0000
@@ -4423,10 +4423,9 @@
      */
     errflag = 0;
 
-    if (isset(MONITOR)) {
-	/* send SIGHUP to any jobs left running  */
-	killrunjobs(from_where == 1);
-    }
+    /* send SIGHUP to any jobs left running  */
+    killrunjobs(from_where == 1);
+
     if (isset(RCS) && interact) {
 	if (!nohistsave) {
 	    int writeflags = HFILE_USE_OPTIONS;
Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.50
diff -u -r1.50 jobs.c
--- Src/jobs.c	6 Nov 2006 12:49:21 -0000	1.50
+++ Src/jobs.c	15 Dec 2006 11:44:38 -0000
@@ -2027,6 +2027,10 @@
 		    return 1;
 		} else
 		    signame = *argv;
+		if (!*signame) {
+		    zwarnnam(nam, "-: signal name expected");
+		    return 1;
+		}
 		signame = casemodify(signame, CASMOD_UPPER);
 		if (!strncmp(signame, "SIG", 3))
 		    signame+=3;
Index: Src/signals.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/signals.c,v
retrieving revision 1.41
diff -u -r1.41 signals.c
--- Src/signals.c	30 May 2006 22:35:03 -0000	1.41
+++ Src/signals.c	15 Dec 2006 11:44:38 -0000
@@ -636,9 +636,15 @@
 killrunjobs(int from_signal)
 {
     int i, killed = 0;
- 
+
     if (unset(HUP))
         return;
+
+    if (!isset(MONITOR))
+    {
+	killpg(0, SIGHUP);
+	return;
+    }
     for (i = 1; i <= maxjob; i++)
         if ((from_signal || i != thisjob) && (jobtab[i].stat & STAT_LOCKED) &&
             !(jobtab[i].stat & STAT_NOPRINT) &&




To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Is wait not interruptable?

[Peter Stephenson]

Peter Stephenson wrote:
> The first is that the subshell itself (the stuff in parentheses) doesn't
> get killed because it's waiting for the child process.  This seems to be a
> feature of SIGTERM.  If you "setopt trapsasync" or send a different signal
> such as SIGHUP it does get killed.  I'm not really sure why SIGTERM is
> different.

I got a chance to look at the code in more detail.

It's not SIGTERM that's different, it's SIGHUP.  The code is:

    if (isset(TRAPSASYNC)) {
	sigemptyset(set);
    } else {
	sigfillset(set);
	sigdelset(set, sig);
#ifdef POSIX_SIGNALS
	sigdelset(set, SIGHUP);  /* still don't know why we add this? */
#endif
	if (wait_cmd)
	{
	    /*
	     * Using "wait" builtin.  We should allow SIGINT and
	     * execute traps delivered to the shell.
	     */
	    int sigtr;
	    sigdelset(set, SIGINT);
	    for (sigtr = 1; sigtr <= NSIG; sigtr++) {
		if (sigtrapped[sigtr] & ~ZSIG_IGNORED)
		    sigdelset(set, sigtr);
	    }
	}
    }

So there's not really any mystery about how it behaves, only why it's
the way it is.  POSIX is fairly light on what to do with signals:
http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_11

TRAPSASYNC was really introduced, as the name suggests, to affect trap
execution (see second paragraph of the section in the URL), not the
point at which the signal is delivered to the shell, but in practice it
does the former by altering the latter.  The may be a case for always
handling the signal immediately and altering the code only to delay any
traps.  But I'm still not sure why it's done the way it is.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Is wait not interruptable?

[Bart Schaefer]

On Dec 15, 12:05pm, Peter Stephenson wrote:
}
} > The ^C was typed 1 second after the command was 
} > started, and the first sleep plodded along to completion for its full 
} > 3 seconds.
} 
} I *think* the patch below fixes (b) by sending SIGHUP to our own process
} group when exiting from a non-interactive shell, but I'll take advice on
} this.  (I can at least then get the test program to work with "setopt hup
} trapsasync".)  The manual isn't clear over whether this is supposed to
} happen non-interactively, though it would probably be fair to assume

This is a conflict between subshells and other non-interactive shells.  I
believe a subshell whose parent is an interactive shell should propagate
HUP signals to any tasks that it backgrounded, but shells that are non-
interactive from the get-go should not.

This has to do with the defnition of SIGHUP, which is that it's a signal
sent by a controlling TTY to the foreground process group leader when the
TTY is disconnected.  Shells that are not a foreground group leader are
not responsible for propagating HUP signals.

This is of course all a mishmash from the way shells work.  If a shell
has started a foreground job and given that control of the TTY, then
when the line drops the foreground job will get HUP'd but the shell
will not -- it'll just get EOF after it reaps the foreground job and
goes back to read from the terminal again.  Generating a HUP to all its
other jobs in that circumstance is just to mimic the behavior of the
OS signalling the whole process group, which would have happened if the
shell had been the foreground job at the time of the hangup.

} if HUP is set in a non-interactive shell you'd expect the SIGHUP to be
} sent.

Well, yes and no.  In a non-interactive shell there is theoretically no
controlling TTY, so there's no reason to expect to receive a HUP in the
first place, so no reason to expect it to be sent either.

[In another message]

} TRAPSASYNC was really introduced, as the name suggests, to affect trap
} execution (see second paragraph of the section in the URL), not the
} point at which the signal is delivered to the shell, but in practice it
} does the former by altering the latter.  The may be a case for always
} handling the signal immediately and altering the code only to delay any
} traps.  But I'm still not sure why it's done the way it is.

The signals are handled the way they are mainly because of non-re-entrant
libraries (particularly malloc) that can be confused or broken by having
signal code executed at an arbitrary point.  The "wait" builtin should
probably set TRAPS_ASYNC implicitly during its execution, based on the
text in 2.11 at that POSIX URL you indicated.

Re: PATCH: _firefox

[arno.]

[-- Attachment #1: Type: text/plain, Size: 1611 bytes --]

Le vendredi 08 décembre 2006, à 21:28:25 +0000, Peter a écrit :
> Peter Stephenson wrote:
> > Not sure why we don't have this already, except that the command line
> > options aren't very well documented.

Hi,
this is part of an update I submitted a few weeks ago :
http://www.zsh.org/mla/workers/2006/msg00755.html
http://www.zsh.org/mla/workers/2006/msg00756.html

It's nice that firefox completion is updated now, but -CreateProfile
option has not been updated. That option is really useful when
developing firefox extensions, and I'd like to known if it is possible to
include completion for -CreateProfile for Firefox. So, I join a patch
suggestion for that :

arno.

diff -Nur zsh/Completion/X/Command/_mozilla zsh~/Completion/X/Command/_mozilla
--- zsh/Completion/X/Command/_mozilla   2006-12-17 14:08:38.000000000 +0100
+++ zsh~/Completion/X/Command/_mozilla  2006-12-17 14:24:02.000000000 +0100
@@ -27,7 +27,6 @@
   else
     mozopts=(
       "($popts)-installer[start with 4.x migration window]"
-      "($popts)-CreateProfile:profile"
       "($popts)-ProfileWizard[start with profile wizard]"
       "($popts)-SelectProfile[start with profile selection dialog]"
       '-splash[enable splash screen]'
@@ -42,6 +41,7 @@
   _x_arguments -C  $mozopts \
     '-height[height of startup window]:height' \
     '(-)'{-h,-help}'[show usage message]' \
+      "($popts)-CreateProfile:profile" \
     '-width[width of startup window]:width' \
     '(-)'{-v,-version}'[show the version number and build date]' \
     "($popts)-P[start with profile]:profile:->profile" \


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Is wait not interruptable?

[Peter Stephenson]

Bart Schaefer wrote:
> } if HUP is set in a non-interactive shell you'd expect the SIGHUP to be
> } sent.
> 
> Well, yes and no.  In a non-interactive shell there is theoretically no
> controlling TTY, so there's no reason to expect to receive a HUP in the
> first place, so no reason to expect it to be sent either.

Hmm... there's may still be a process group, with the shell executing
the commands as leader, right?  Although that will depend how the script
itself was started.  Potentially, I suppose it might also kill what
started the script, if that wasn't using job control.

It seems from the original post that bash is doing something to kill its
children when it gets a signal during execution of a script, so there
seems to be some discrepancy I haven't investigated further.

I see the external command kill on Linux allows you to send a signal to
a process group, but I don't think there's a way of doing that within
zsh.

> } TRAPSASYNC was really introduced, as the name suggests, to affect trap
> } execution (see second paragraph of the section in the URL), not the
> } point at which the signal is delivered to the shell, but in practice it
> } does the former by altering the latter.  The may be a case for always
> } handling the signal immediately and altering the code only to delay any
> } traps.  But I'm still not sure why it's done the way it is.
> 
> The signals are handled the way they are mainly because of non-re-entrant
> libraries (particularly malloc) that can be confused or broken by having
> signal code executed at an arbitrary point.

But that doesn't apply here, I don't think.  The code in question is in
signal_suspend_setup().  We only execute that when the shell is about to
suspend execution to wait for a child to exit; the signal mask thus
prepared is only used during sigsuspend() or equivalent.  After the
SIGCHLD (or possibly some other signal) has been received, we then
immediately undo the signal mask (or ideally leave it to the OS when
sigsuspend() returns).  So there's no question of interrupting arbitrary
code if the signal handler is executed at that point.

Indeed, in the Linux manual entry for signal(2), sigsuspend() is listed
as a "safe function"; if a signal handler executes while the shell is
here, it's explicitly OK to call general OS stuff like malloc().  We use
dont_queue_signals() around the code that calls signal_suspend() in
waitforpid(), presumably because of this.

I wondered if it was because receiving the "wrong" signal when waiting
for SIGCHLD would confuse the shell logic.  But the loops surrounding
both calls to signal_suspend() seem to contradict that.

Surely the right thing to do would be to use a more lenient signal mask
in signal_suspend(), since we know that's a good place for signal
handling---although we might have to queue traps for later execution
unless one of TRAPSASYNC or the wait builtin is in use.

So is the delayed execution of traps the only issue?  What have I
missed?  Signal handlers theselves don't need to guard against recursive
calling, do they?

> The "wait" builtin should
> probably set TRAPS_ASYNC implicitly during its execution, based on the
> text in 2.11 at that POSIX URL you indicated.

I think you're right that the wait code (which is currently special) and
the TRAPSASYNC code should probably do the same thing.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Is wait not interruptable?

[Bart Schaefer]

On Dec 17,  4:00pm, Peter Stephenson wrote:
}
} I see the external command kill on Linux allows you to send a signal to
} a process group, but I don't think there's a way of doing that within
} zsh.

Sure there is.  The negative PID thing is implemented at the OS level by
the kill(2) system call, so "kill -HUP -12345" even from the zsh builtin
will kill the entire process group of 12345.  Or at least it's intended
to do so -- has something changed such that zsh's kill won't handle a
negative PID any more?

} I wondered if it was because receiving the "wrong" signal when waiting
} for SIGCHLD would confuse the shell logic.  But the loops surrounding
} both calls to signal_suspend() seem to contradict that.
} 
} Surely the right thing to do would be to use a more lenient signal mask
} in signal_suspend(), since we know that's a good place for signal
} handling---although we might have to queue traps for later execution
} unless one of TRAPSASYNC or the wait builtin is in use.

This may be an artifact from before trap queuing was possible, then.  I
generally concur with your assessment here.

} So is the delayed execution of traps the only issue?  What have I
} missed?  Signal handlers theselves don't need to guard against recursive
} calling, do they?

The handler functions themselves don't, no, unless (IIRC) the handler is
explicitly reset from within the function.  Which could happen if the
handler calls a shell function which calls "trap", I suppose.

Re: Is wait not interruptable?

[Peter Stephenson]

Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Dec 17,  4:00pm, Peter Stephenson wrote:
> }
> } I see the external command kill on Linux allows you to send a signal to
> } a process group, but I don't think there's a way of doing that within
> } zsh.
> 
> Sure there is.  The negative PID thing is implemented at the OS level by
> the kill(2) system call, so "kill -HUP -12345" even from the zsh builtin
> will kill the entire process group of 12345.  Or at least it's intended
> to do so -- has something changed such that zsh's kill won't handle a
> negative PID any more?

It does work; it could probably do with being mentioned in the kill manual
entry since that specifically mentions "jobs" and "processes" at the
moment.  However, I suspect it's not quite what's wanted here, after all,
since without job control the subshells are in the same process group as
the main shell.

> } I wondered if it was because receiving the "wrong" signal when waiting
> } for SIGCHLD would confuse the shell logic.  But the loops surrounding
> } both calls to signal_suspend() seem to contradict that.
> } 
> } Surely the right thing to do would be to use a more lenient signal mask
> } in signal_suspend(), since we know that's a good place for signal
> } handling---although we might have to queue traps for later execution
> } unless one of TRAPSASYNC or the wait builtin is in use.
> 
> This may be an artifact from before trap queuing was possible, then.  I
> generally concur with your assessment here.

I think some rewriting may be in order... There's the vestiges of code
(trap_queue* variables in signals.c) for queueing traps (as opposed to the
current method of queueing signals, which necessarily means delaying traps,
too).  I suggest using that in this case so that signals are handled
immediately, but traps are queued except for the cases of wait or
TRAPSASYNC.  This will allow the shell to exit quickly if the signal isn't
trapped, which is the real problem in this case (and presumably the reason
for that sigdelset(set, SIGHUP) in signal_suspend_setup()).  To be clear:
nothing would change except for the handling of signals around the point
where we're doing nothing but waiting for a child.  (In particular, no
fiddling with the signal queueing code is needed.)

> } So is the delayed execution of traps the only issue?  What have I
> } missed?  Signal handlers theselves don't need to guard against recursive
> } calling, do they?
> 
> The handler functions themselves don't, no, unless (IIRC) the handler is
> explicitly reset from within the function.  Which could happen if the
> handler calls a shell function which calls "trap", I suppose.

Yes, that rings a bell, now you mention it.  If traps are queued at this
point, that won't happen, at least.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Is wait not interruptable?

[Bart Schaefer]

On Dec 18, 11:39am, Peter Stephenson wrote:
}
} I think some rewriting may be in order... (In particular, no
} fiddling with the signal queueing code is needed.)

This suggestion sounds right to me.

Re: Is wait not interruptable?

[Peter Stephenson]

Peter Stephenson <pws@csr.com> wrote:
> > } Surely the right thing to do would be to use a more lenient signal mask
> > } in signal_suspend(), since we know that's a good place for signal
> > } handling---although we might have to queue traps for later execution
> > } unless one of TRAPSASYNC or the wait builtin is in use.
> > 
> > This may be an artifact from before trap queuing was possible, then.  I
> > generally concur with your assessment here.
> 
> I think some rewriting may be in order... There's the vestiges of code
> (trap_queue* variables in signals.c) for queueing traps (as opposed to the
> current method of queueing signals, which necessarily means delaying traps,
> too).  I suggest using that in this case so that signals are handled
> immediately, but traps are queued except for the cases of wait or
> TRAPSASYNC.  This will allow the shell to exit quickly if the signal isn't
> trapped, which is the real problem in this case (and presumably the reason
> for that sigdelset(set, SIGHUP) in signal_suspend_setup()).  To be clear:
> nothing would change except for the handling of signals around the point
> where we're doing nothing but waiting for a child.  (In particular, no
> fiddling with the signal queueing code is needed.)

Here is the implementation, attempting to be minimally invasive.  I won't
check it in immediately (but I'm on holiday from Wednesday so would like
to sort it out by Tuesday evening UK time).  Tests suggest it does what I
think it should.

As I noted in a comment, there's presumably a race when queuing traps since
the signal may already have been delivered by the time we start the trap
queuing.  That's not new (we only queue traps at roughly the point where we
previously blocked signals).  I could move the new queue_traps() before
dont_queue_signals() in the two applicable functions: then traps would be
queued when the previously blocked signals arrive.  I suspect that's a good
thing to do, but (i) it only just occurred to me (ii) it's a slightly
larger change of behaviour, so I haven't done it here.  Discussion welcome,
obviously.

One slight difference is that in zwaitjob() we'll delay traps to wait for
the entire foreground job, not just the first process to exit.  Before, if
there were multiple processes to wait for, we'd have returned from
sigsuspend() after each, and potentially handled signals there.  That seems
a bit hit and miss---it didn't matter which process was actually
finished---so I'm happy with the new behaviour.

The first hunk is separate, to indicate that additional uses of PIDs by
kill may be used if handled by the OS.

I think that with this patch Dave's original problem should go away: even though
the "sleep"s in his example hang on, as discussed in the other half of this thread,
they won't have any effect since the shell will exit immediately when it gets the
(untrapped) SIGTERM.

Index: Doc/Zsh/builtins.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/builtins.yo,v
retrieving revision 1.89
diff -u -r1.89 builtins.yo
--- Doc/Zsh/builtins.yo	13 Dec 2006 22:30:38 -0000	1.89
+++ Doc/Zsh/builtins.yo	18 Dec 2006 15:54:06 -0000
@@ -692,6 +692,9 @@
 show if the alternative form corresponds to a signal number.  For example,
 under Linux tt(kill -l IO) and tt(kill -l POLL) both output 29, hence
 tt(kill -IO) and tt(kill -POLL) have the same effect.
+
+Many systems will allow process IDs to be negative to kill a process
+group or zero to kill the current process group.
 )
 findex(let)
 item(tt(let) var(arg) ...)(
Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.51
diff -u -r1.51 jobs.c
--- Src/jobs.c	18 Dec 2006 11:16:03 -0000	1.51
+++ Src/jobs.c	18 Dec 2006 15:54:06 -0000
@@ -1141,6 +1141,7 @@
     /* child_block() around this loop in case #ifndef WNOHANG */
     dont_queue_signals();
     child_block();		/* unblocked in signal_suspend() */
+    queue_traps(wait_cmd);
     while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) {
 	if (first)
 	    first = 0;
@@ -1148,7 +1149,7 @@
 	    kill(pid, SIGCONT);
 
 	last_signal = -1;
-	signal_suspend(SIGCHLD, wait_cmd);
+	signal_suspend(SIGCHLD);
 	if (last_signal != SIGCHLD && wait_cmd) {
 	    /* wait command interrupted, but no error: return */
 	    restore_queue_signals(q);
@@ -1156,6 +1157,7 @@
 	}
 	child_block();
     }
+    unqueue_traps();
     child_unblock();
     restore_queue_signals(q);
 
@@ -1177,6 +1179,7 @@
 
     dont_queue_signals();
     child_block();		 /* unblocked during signal_suspend() */
+    queue_traps(wait_cmd);
     if (jn->procs || jn->auxprocs) { /* if any forks were done         */
 	jn->stat |= STAT_LOCKED;
 	if (jn->stat & STAT_CHANGED)
@@ -1184,7 +1187,7 @@
 	while (!errflag && jn->stat &&
 	       !(jn->stat & STAT_DONE) &&
 	       !(interact && (jn->stat & STAT_STOPPED))) {
-	    signal_suspend(SIGCHLD, wait_cmd);
+	    signal_suspend(SIGCHLD);
 	    if (last_signal != SIGCHLD && wait_cmd)
 	    {
 		/* builtin wait interrupted by trapped signal */
@@ -1211,6 +1214,7 @@
 	pipestats[0] = lastval;
 	numpipestats = 1;
     }
+    unqueue_traps();
     child_unblock();
     restore_queue_signals(q);
 
Index: Src/signals.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/signals.c,v
retrieving revision 1.41
diff -u -r1.41 signals.c
--- Src/signals.c	30 May 2006 22:35:03 -0000	1.41
+++ Src/signals.c	18 Dec 2006 15:54:06 -0000
@@ -346,39 +346,10 @@
 static int suspend_longjmp = 0;
 static signal_jmp_buf suspend_jmp_buf;
 #endif
- 
-#if defined(POSIX_SIGNALS) || defined(BSD_SIGNALS)
-static void
-signal_suspend_setup(sigset_t *set, int sig, int wait_cmd)
-{
-    if (isset(TRAPSASYNC)) {
-	sigemptyset(set);
-    } else {
-	sigfillset(set);
-	sigdelset(set, sig);
-#ifdef POSIX_SIGNALS
-	sigdelset(set, SIGHUP);  /* still don't know why we add this? */
-#endif
-	if (wait_cmd)
-	{
-	    /*
-	     * Using "wait" builtin.  We should allow SIGINT and
-	     * execute traps delivered to the shell.
-	     */
-	    int sigtr;
-	    sigdelset(set, SIGINT);
-	    for (sigtr = 1; sigtr <= NSIG; sigtr++) {
-		if (sigtrapped[sigtr] & ~ZSIG_IGNORED)
-		    sigdelset(set, sigtr);
-	    }
-	}
-    }
-}
-#endif
 
 /**/
 int
-signal_suspend(int sig, int wait_cmd)
+signal_suspend(int sig)
 {
     int ret;
  
@@ -388,7 +359,7 @@
     sigset_t oset;
 #endif /* BROKEN_POSIX_SIGSUSPEND */
 
-    signal_suspend_setup(&set, sig, wait_cmd);
+    sigemptyset(&set);
 #ifdef BROKEN_POSIX_SIGSUSPEND
     sigprocmask(SIG_SETMASK, &set, &oset);
     pause();
@@ -400,7 +371,7 @@
 # ifdef BSD_SIGNALS
     sigset_t set;
 
-    signal_suspend_setup(&set, sig, wait_cmd);
+    sigemptyset(&set);
     ret = sigpause(set);
 # else
 #  ifdef SYSV_SIGNALS
@@ -419,7 +390,7 @@
 #  endif /* SYSV_SIGNALS  */
 # endif  /* BSD_SIGNALS   */
 #endif   /* POSIX_SIGNALS */
- 
+
     return ret;
 }
 
@@ -561,18 +532,14 @@
         break;
  
     case SIGHUP:
-        if (sigtrapped[SIGHUP])
-            dotrap(SIGHUP);
-        else {
+        if (!handletrap(SIGHUP)) {
             stopmsg = 1;
             zexit(SIGHUP, 1);
         }
         break;
  
     case SIGINT:
-        if (sigtrapped[SIGINT])
-            dotrap(SIGINT);
-        else {
+        if (!handletrap(SIGINT)) {
 	    if ((isset(PRIVILEGED) || isset(RESTRICTED)) &&
 		isset(INTERACTIVE) && noerrexit < 0)
 		zexit(SIGINT, 1);
@@ -587,19 +554,12 @@
 #ifdef SIGWINCH
     case SIGWINCH:
         adjustwinsize(1);  /* check window size and adjust */
-	if (sigtrapped[SIGWINCH])
-	    dotrap(SIGWINCH);
+	(void) handletrap(SIGWINCH);
         break;
 #endif
 
     case SIGALRM:
-        if (sigtrapped[SIGALRM]) {
-	    int tmout;
-            dotrap(SIGALRM);
-
-	    if ((tmout = getiparam("TMOUT")))
-		alarm(tmout);           /* reset the alarm */
-        } else {
+        if (!handletrap(SIGALRM)) {
 	    int idle = ttyidlegetfn(NULL);
 	    int tmout = getiparam("TMOUT");
 	    if (idle >= 0 && idle < tmout)
@@ -614,7 +574,7 @@
         break;
  
     default:
-        dotrap(sig);
+        (void) handletrap(sig);
         break;
     }   /* end of switch(sig) */
  
@@ -1000,6 +960,96 @@
 	  "BUG: still saved traps outside all function scope");
 }
 
+
+/*
+ * Decide whether a trap needs handling.
+ * If so, see if the trap should be run now or queued.
+ * Return 1 if the trap has been or will be handled.
+ * This only needs to be called in place of dotrap() in the
+ * signal handler, since it's only while waiting for children
+ * to exit that we queue traps.
+ */
+/**/
+static int
+handletrap(int sig)
+{
+    if (!sigtrapped[sig])
+	return 0;
+
+    if (trap_queueing_enabled)
+    {
+	/* Code borrowed from signal queueing */
+	int temp_rear = ++trap_queue_rear % MAX_QUEUE_SIZE;
+
+	DPUTS(temp_rear == trap_queue_front, "BUG: trap queue full");
+	/* If queue is not full... */
+	if (temp_rear != trap_queue_front) {
+	    trap_queue_rear = temp_rear;
+	    trap_queue[trap_queue_rear] = sig;
+	}
+	return 1;
+    }
+
+    dotrap(sig);
+
+    if (sig == SIGALRM)
+    {
+	int tmout;
+	/*
+	 * Reset the alarm.
+	 * It seems slightly more natural to do this when the
+	 * trap is run, rather than when it's queued, since
+	 * the user doesn't see the latter.
+	 */
+	if ((tmout = getiparam("TMOUT")))
+	    alarm(tmout);
+    }
+
+    return 1;
+}
+
+
+/*
+ * Queue traps if they shouldn't be run asynchronously, i.e.
+ * we're not in the wait builtin and TRAPSASYNC isn't set, when
+ * waiting for children to exit.
+ *
+ * Note that unlike signal queuing this should only be called
+ * in single matching pairs and can't be nested.  It is
+ * only needed when waiting for a job or process to finish.
+ *
+ * There is presumably a race setting this up: we shouldn't be running
+ * traps between forking a foreground process and this point, either.
+ */
+/**/
+void
+queue_traps(int wait_cmd)
+{
+    if (!isset(TRAPSASYNC) && !wait_cmd) {
+	/*
+	 * Traps need to be handled synchronously, so
+	 * enable queueing.
+	 */
+	trap_queueing_enabled = 1;
+    }
+}
+
+
+/*
+ * Disable trap queuing and run the traps.
+ */
+/**/
+void
+unqueue_traps(void)
+{
+    trap_queueing_enabled = 0;
+    while (trap_queue_front != trap_queue_rear) {
+	trap_queue_front = (trap_queue_front + 1) % MAX_QUEUE_SIZE;
+	(void) handletrap(trap_queue[trap_queue_front]);
+    }
+}
+
+
 /* Execute a trap function for a given signal, possibly
  * with non-standard sigtrapped & siglists values
  */

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Is wait not interruptable?

[Bart Schaefer]

On Dec 18,  4:12pm, Peter Stephenson wrote:
}
} As I noted in a comment, there's presumably a race when queuing traps since
} the signal may already have been delivered by the time we start the trap
} queuing.  That's not new (we only queue traps at roughly the point where we
} previously blocked signals).  I could move the new queue_traps() before
} dont_queue_signals() in the two applicable functions: then traps would be
} queued when the previously blocked signals arrive.

As long as it's "safe" (in terms of shell function re-entrancy) and not
semantically incorrect for traps to be executed at that point, I think
what you have is fine.

Note that the important thing through this whole section is that
last_signal is sane, so that we don't end up waiting for the wrong
child if both a SIGCHLD and some other signal arrive during the
signal_suspend().  (I may be missing a detail that makes that comment
irrelevant.)

} One slight difference is that in zwaitjob() we'll delay traps to wait
} for the entire foreground job, not just the first process to exit.

Is that affected by TRAPS_ASYNC?

Re: Is wait not interruptable?

[Peter Stephenson]

Bart Schaefer wrote:
> Note that the important thing through this whole section is that
> last_signal is sane, so that we don't end up waiting for the wrong
> child if both a SIGCHLD and some other signal arrive during the
> signal_suspend().  (I may be missing a detail that makes that comment
> irrelevant.)

I don't think there should be a problem with signal ordering.  It's not
affected by the changes to traps --- none of the code I touched affects
last_signal, except, obviously, the fact that we may now handle some
signals earlier; but we had to be prepared to handle signals at that
point anyway.

> } One slight difference is that in zwaitjob() we'll delay traps to wait
> } for the entire foreground job, not just the first process to exit.
> 
> Is that affected by TRAPS_ASYNC?

Yes, if TRAPS_ASYNC is set (or we're at the wait builtin) we'd have
handled the trap already.  The difference is only if we'd otherwise have
been delaying signal delivery.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

4.3.2/20061219 -> 4.3.2/20070126 very broken

[Alexey Tourbin]

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

Hello,
4.3.2/20061219 -> 4.3.2/20070126

The prompt cannot be displayed correctly and input is sort of broken
(after pressing enter, the command is executed but the prompt is not
displayed; pressing enter again gets the prompt back).

git-bisect blames
23097: splitting of $'...' strings in completion

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 4.3.2/20061219 -> 4.3.2/20070126 very broken

[Alexey Tourbin]

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

On Sat, Jan 27, 2007 at 08:07:26PM +0300, Alexey Tourbin wrote:
> Hello,
> 4.3.2/20061219 -> 4.3.2/20070126
> 
> The prompt cannot be displayed correctly and input is sort of broken
> (after pressing enter, the command is executed but the prompt is not
> displayed; pressing enter again gets the prompt back).
> 
> git-bisect blames
> 23097: splitting of $'...' strings in completion

PS: here is what my prompt is:

myprompt()
{       
        autoload colors -U && colors
        local COLOR=green
        local WINDOW="%{$fg_bold[$COLOR]%}${WINDOW:--}%{$fg_no_bold[$COLOR]%}"
        local NULL=$'%{\ek\e\\%}'
        PROMPT="$NULL%{$fg[$COLOR]%}%n@%m %3~ $WINDOW %(!.#.$) %{$fg[default]%}"
}
myprompt   

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 4.3.2/20061219 -> 4.3.2/20070126 very broken

[Peter Stephenson]

On Sat, 27 Jan 2007 20:15:01 +0300
Alexey Tourbin <at@altlinux.ru> wrote:
> On Sat, Jan 27, 2007 at 08:07:26PM +0300, Alexey Tourbin wrote:
> > Hello,
> > 4.3.2/20061219 -> 4.3.2/20070126
> > 
> > The prompt cannot be displayed correctly and input is sort of broken
> > (after pressing enter, the command is executed but the prompt is not
> > displayed; pressing enter again gets the prompt back).
> > 
> > git-bisect blames
> > 23097: splitting of $'...' strings in completion
> 
> PS: here is what my prompt is:

Thanks, this combination allowed we to find it quite quickly.  It was a
major oversight in the the change to the way backslashes in $'...' are
handled.  At least, I presume this is the same problem you're
seeing... the results of the bug could be unpredictable.

Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.150
diff -u -r1.150 utils.c
--- Src/utils.c	21 Jan 2007 22:47:41 -0000	1.150
+++ Src/utils.c	27 Jan 2007 18:54:37 -0000
@@ -4914,23 +4914,34 @@
 	} else if (*s == Meta)
 	    *t++ = *++s ^ 32;
 	else {
-	    *t++ = *s;
 	    if (itok(*s)) {
 		if (meta || control) {
 		    /*
 		     * Presumably we should be using meta or control
 		     * on the character representing the token.
 		     */
-		    *s = ztokens[*s - Pound];
+		    *t++ = ztokens[*s - Pound];
 		} else if (how & GETKEY_DOLLAR_QUOTE) {
-		    /*
-		     * We don't want to metafy this, it's a real
-		     * token.
-		     */
-		    *tdest++ = *s;
+		    if (*s == Bnull) {
+			/*
+			 * Bnull is a backslash which quotes a couple
+			 * of special characters that always appear
+			 * literally next.  See strquote handling
+			 * in gettokstr() in lex.c.
+			 */
+			*tdest++ = *++s;
+		    } else {
+			/*
+			 * We don't want to metafy this, it's a real
+			 * token.
+			 */
+			*tdest++ = *s;
+		    }
 		    continue;
-		}
-	    }
+		} else
+		    *t++ = *s;
+	    } else
+		*t++ = *s;
 	}
 	if (meta == 2) {
 	    t[-1] |= 0x80;
Index: Test/A03quoting.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A03quoting.ztst,v
retrieving revision 1.1
diff -u -r1.1 A03quoting.ztst
--- Test/A03quoting.ztst	2 Apr 2001 12:30:47 -0000	1.1
+++ Test/A03quoting.ztst	27 Jan 2007 18:54:37 -0000
@@ -13,6 +13,10 @@
 >'ut queant laxis'
 >"resonare fibris"
 
+  print -r $'\'a \\\' is \'a backslash\' is \'a \\\''
+0:$'-style quotes with backslashed backslashes
+>'a \' is 'a backslash' is 'a \'
+
   print -r ''''
   setopt rcquotes
 # We need to set rcquotes here for the next example since it is

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: 4.3.2/20061219 -> 4.3.2/20070126 very broken

[Bart Schaefer]

On Jan 27,  8:15pm, Alexey Tourbin wrote:
}
} > The prompt cannot be displayed correctly and input is sort of broken
} > (after pressing enter, the command is executed but the prompt is not
} > displayed; pressing enter again gets the prompt back).
} 
}         local NULL=$'%{\ek\e\\%}'

It appears that \\ is not being interpreted the same way it used to be.
Here's 4.2.6:

schaefer[501] NULL=$'%{\ek\e\\%}'
schaefer[502] print -RP $NULL | cat -v
^[k^[\

Here's the latest 4.3.2:

schaefer<501> NULL=$'%{\ek\e\\%}'
schaefer<502> print -RP $NULL | cat -v
^[k^[

Re: 4.3.2/20061219 -> 4.3.2/20070126 very broken

[Peter Stephenson]

Here's a further improvement... the previous patch didn't pass on the
Bnull, which I added specially so that the completion code could see it
when trying to complete words inside $'...', and I missed the admittedly
rather special case of an escaped backslash that had \M- or \C- before
it.


Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.151
diff -u -r1.151 utils.c
--- Src/utils.c	27 Jan 2007 19:01:11 -0000	1.151
+++ Src/utils.c	27 Jan 2007 23:45:36 -0000
@@ -4915,27 +4915,45 @@
 	    *t++ = *++s ^ 32;
 	else {
 	    if (itok(*s)) {
+		/*
+		 * We need to be quite careful here.  We haven't
+		 * necessarily got an input stream with all tokens
+		 * removed, so the majority of tokens need passing
+		 * through untouched and without Meta handling.
+		 * However, me may need to handle tokenized
+		 * backslashes.
+		 */
 		if (meta || control) {
 		    /*
 		     * Presumably we should be using meta or control
 		     * on the character representing the token.
+		     *
+		     * Special case: $'\M-\\' where the token is a Bnull.
+		     * This time we dump the Bnull since we're
+		     * replacing the whole thing.  The lexer
+		     * doesn't know about the meta or control modifiers.
 		     */
-		    *t++ = ztokens[*s - Pound];
+		    if ((how & GETKEY_DOLLAR_QUOTE) && *s == Bnull)
+			*t++ = *++s;
+		    else
+			*t++ = ztokens[*s - Pound];
 		} else if (how & GETKEY_DOLLAR_QUOTE) {
+		    /*
+		     * We don't want to metafy this, it's a real
+		     * token.
+		     */
+		    *tdest++ = *s;
 		    if (*s == Bnull) {
 			/*
 			 * Bnull is a backslash which quotes a couple
 			 * of special characters that always appear
 			 * literally next.  See strquote handling
-			 * in gettokstr() in lex.c.
+			 * in gettokstr() in lex.c.  We need
+			 * to retain the Bnull (as above) so that quote
+			 * handling in completion can tell where the
+			 * backslash was.
 			 */
 			*tdest++ = *++s;
-		    } else {
-			/*
-			 * We don't want to metafy this, it's a real
-			 * token.
-			 */
-			*tdest++ = *s;
 		    }
 		    continue;
 		} else
Index: Test/A03quoting.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A03quoting.ztst,v
retrieving revision 1.2
diff -u -r1.2 A03quoting.ztst
--- Test/A03quoting.ztst	27 Jan 2007 19:01:11 -0000	1.2
+++ Test/A03quoting.ztst	27 Jan 2007 23:45:36 -0000
@@ -17,6 +17,20 @@
 0:$'-style quotes with backslashed backslashes
 >'a \' is 'a backslash' is 'a \'
 
+  chars=$(print -r $'BS\\MBS\M-\\')
+  for (( i = 1; i <= $#chars; i++ )); do
+    char=$chars[$i]
+    print $(( [#16] #char ))
+  done
+0:$'-style quote with metafied backslash
+>16#42
+>16#53
+>16#5C
+>16#4D
+>16#42
+>16#53
+>16#DC
+
   print -r ''''
   setopt rcquotes
 # We need to set rcquotes here for the next example since it is

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: 4.3.2/20061219 -> 4.3.2/20070126 very broken

[Alexey Tourbin]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

On Sat, Jan 27, 2007 at 11:47:03PM +0000, Peter Stephenson wrote:
> Here's a further improvement... the previous patch didn't pass on the
> Bnull, which I added specially so that the completion code could see it
> when trying to complete words inside $'...', and I missed the admittedly
> rather special case of an escaped backslash that had \M- or \C- before
> it.

Thanks, the upgrade is now smooth.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Quoting problem and crashes with ${(#)var}

[Bart Schaefer]

With zsh -f from the latest CVS as of 2007-02-10 10:28 AM PST:

torch% for x in {1..255}; echo -n ${(#)x}; echo ''


123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ÿÿÿÿÿÿ-n#$^BUILD Completion Config config.cache config.h config.h-xx config.log config.modules config.modules.sh config.status core.6537 core.6551 core.6555 core.6571 core.6573 core.6575 core.6577 core.6583 core.6587 Doc Etc Functions Makefile Src stamp-h stamp-h.in Test ulzsh: bad pattern: (
torch% for x in {1..255}; echo -n ${(V#)x}; echo ''
^A^B^C^D^E^F^G^H
^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_ !"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~^? #$^*()$=|{}[]`<>?~`,'"\\torch% 

Note that with the ${(V#)x} incantation, the newline that should
be output by the final echo is not printed.  In fact, if I replace that
last echo with "cat config.h" there is STILL nothing output, so it's not
just that a single line is somehow being swallowed.  But if I replace the
final echo with "sleep 30" the shell does pause for 30 seconds, so the
command is in fact being executed.  Where is the output going?

On we go:

torch% for x in {1..255}; echo -n ${(q#)x}; echo ''

I can't show the output because at this point the terminal window vanished
as zsh dumped core:

#0  0x080bc534 in quotestring (s=0xb7d674d0 "\203", e=0x0, instring=1)
    at ../../zsh-4.0/Src/utils.c:4282
#1  0x080b1822 in paramsubst (l=0xb7d67470, n=0xb7d67494, str=0xbff167b8, 
    qt=0, ssub=0) at ../../zsh-4.0/Src/subst.c:2896
#2  0x080ac095 in stringsubst (list=0xb7d67470, node=0xb7d67494, ssub=0, 
    asssub=0) at ../../zsh-4.0/Src/subst.c:193
#3  0x080aba08 in prefork (list=0xb7d67470, flags=0)
    at ../../zsh-4.0/Src/subst.c:91
#4  0x08063189 in execcmd (state=0xbff17020, input=0, output=0, how=18, 
    last1=2) at ../../zsh-4.0/Src/exec.c:2005

Set's try again slightly differently:

torch% setopt nopromptcr nopromptsp
torch% for x in {1..255}; do echo -n ${(#)x}; done |& cat -nv; echo ''
     1  ^A^B^C^D^E^F^G^H
     2  ^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_ !"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~^?M-^@M-^AM-^B M-^?M-^?M-^?M-^?M-^?M-^?#$^BUILD Completion Config config.cache config.h config.h-xx config.log config.modules config.modules.sh config.status core.6537 core.6551 core.6555 core.6571 core.6573 core.6575 core.6577 core.6583 core.6587 Doc Etc Functions Makefile Src stamp-h stamp-h.in Test ulzsh: bad pattern: (

torch% for x in {1..255}; do echo -n ${(V#)x}; done | cat -nv; echo ''
     1  ^A^B^C^D^E^F^G^H
     2  ^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_ !"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~^?M-^@M-^AM-^B #$^*()$=|{}[]`<>?~`,'"\\M-^]M-^^M-^_M- M-!M-"M-#M-$M-%M-&M-'M-(M-)M-*M-+M-,M--M-.M-/M-0M-1M-2M-3M-4M-5M-6M-7M-8M-9M-:M-;M-<M-=M->M-?M-@M-AM-BM-CM-DM-EM-FM-GM-HM-IM-JM-KM-LM-MM-NM-OM-PM-QM-RM-SM-TM-UM-VM-WM-XM-YM-ZM-[M-\M-]M-^M-_M-`M-aM-bM-cM-dM-eM-fM-gM-hM-iM-jM-kM-lM-mM-nM-oM-pM-qM-rM-sM-tM-uM-vM-wM-xM-yM-zM-{M-|M-}M-~M-^?
torch% for x in {1..255}; do echo -n ${(q#)x}; done | cat -nv; echo ''
     1  ^A^B^C^D^E^F^G^H\       '
     2  '^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_\ \!\"\#\$%\&\'\(\)\*+,./0123456789:\;\<\=\>\?@ABCDEFGHIJKLMNOPQRSTUVWXYZ\[\\]\^_\`abcdefghijklmnopqrstuvwxyz\{\|\}\~^?M-^@M-^AM-^Bzsh: segmentation fault (core dumped)  for x in {1..255}; do; echo -n ${(q#)x}; done | 
zsh: done                              cat -nv

torch% 

OK, there was a "bad pattern" error so:

torch% setopt noglob
torch% for x in {1..255}; do echo -n ${(#)x}; done |& cat -nv; echo ''
     1  ^A^B^C^D^E^F^G^H
     2  ^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_ !"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~^?M-^@M-^AM-^B M-^?M-^?M-^?M-^?M-^?M-^?#$^*()$=|{}[] ../../zsh-4.0/Src/subst.c:221: BUG: parse error in command substitution
     3   ../../zsh-4.0/Src/subst.c:221: BUG: parse error in command substitution
    <<thousands of lines of output suppressed>>
  9394   ../../zsh-4.0/Src/subst.c:221: BUG: parse error in command substitution
  9395   ../../zsh-4.0/Src/subst.c:221: BUG: parse error in command substitution
zsh: segmentation fault (core dumped)  for x in {1..255}; do; echo -n ${(#)x}; done 2>&1 | 
zsh: done                              cat -nv

torch% setopt glob
torch% setopt nobadpattern
torch% for x in {1..255}; do echo -n ${(#)x}; done |& cat -nv; echo ''
     1  ^A^B^C^D^E^F^G^H
     2  ^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_ !"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~^?M-^@M-^AM-^B M-^?M-^?M-^?M-^?M-^?M-^?#$^BUILD Completion Config config.cache config.h config.h-xx config.log config.modules config.modules.sh config.status core.6587 Doc Etc Functions Makefile Src stamp-h stamp-h.in Test ul()$=zsh: no matches found: |

torch% 

So part of the problem appears to be attempting to interpet an unmatched
backtick as the beginning of a command substitution, but we never get that
far if globbing is enabled.

Re: Quoting problem and crashes with ${(#)var}

[Peter Stephenson]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]

Bart Schaefer wrote:
> With zsh -f from the latest CVS as of 2007-02-10 10:28 AM PST:
> 
> torch% for x in {1..255}; echo -n ${(#)x}; echo ''

I think this is it, but shout if you can see any problems left...
This was always a bit broken, since the character was never metafied
properly, but UTF-8 gave exciting new was for bogus tokens to get into
the input.  I also forgot the MULTIBYTE_SUPPORT test.

Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.74
diff -u -r1.74 subst.c
--- Src/subst.c	2 Feb 2007 21:42:15 -0000	1.74
+++ Src/subst.c	10 Feb 2007 22:07:29 -0000
@@ -1193,21 +1193,25 @@
 substevalchar(char *ptr)
 {
     zlong ires = mathevali(ptr);
+    int len;
 
     if (errflag)
 	return NULL;
+#ifdef MULTIBYTE_SUPPORT
     if (isset(MULTIBYTE) && ires > 127) {
 	char buf[10];
-	int dummy;
 
 	/* inefficient: should separate out \U handling from getkeystring */
 	sprintf(buf, "\\U%.8x", (unsigned int)ires);
-	return getkeystring(buf, &dummy, GETKEYS_BINDKEY, NULL);
-    } else {
+	ptr = getkeystring(buf, &len, GETKEYS_BINDKEY, NULL);
+    } else
+#endif
+    {
 	ptr = zhalloc(2);
+	len = 1;
 	sprintf(ptr, "%c", (int)ires);
-	return ptr;
     }
+    return metafy(ptr, len, META_USEHEAP);
 }
 
 /* parameter substitution */
Index: Test/D07multibyte.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D07multibyte.ztst,v
retrieving revision 1.14
diff -u -r1.14 D07multibyte.ztst
--- Test/D07multibyte.ztst	22 Jan 2007 14:35:14 -0000	1.14
+++ Test/D07multibyte.ztst	10 Feb 2007 22:07:29 -0000
@@ -326,3 +326,35 @@
 0:Multibyte characters in print sorting
 >HAH HEH HÉH HÈH HUH
 >HAH HEH HUH HÈH HÉH
+
+# These are control characters in Unicode, so don't show up.
+# We just want to check they're not being treated as tokens.
+  for x in {128..150}; do
+     print ${(#)x}
+  done | while read line; do
+    print ${#line} $(( #line ))
+  done
+0:evaluated character number with multibyte characters
+>1 128
+>1 129
+>1 130
+>1 131
+>1 132
+>1 133
+>1 134
+>1 135
+>1 136
+>1 137
+>1 138
+>1 139
+>1 140
+>1 141
+>1 142
+>1 143
+>1 144
+>1 145
+>1 146
+>1 147
+>1 148
+>1 149
+>1 150

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Quoting problem and crashes with ${(#)var}

[Peter Stephenson]

Peter Stephenson wrote:
> I think this is it

... except I didn't mark it as UTF-8 encoded.  Sorry.  Ignore the
hieroglyphs, I'm about to commit it anyway.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Quoting problem and crashes with ${(#)var}

[Bart Schaefer]

On Feb 10, 10:08pm, Peter Stephenson wrote:
}
} Bart Schaefer wrote:
} > With zsh -f from the latest CVS as of 2007-02-10 10:28 AM PST:
} > 
} > torch% for x in {1..255}; echo -n ${(#)x}; echo ''
} 
} I think this is it, but shout if you can see any problems left...

It doesn't crash any more.  I'm a bit puzzled, given this test ...

}      if (isset(MULTIBYTE) && ires > 127) {

... why ${(V)x} for x in 128 through 159 display as \u0080 through
\u009f, but then 160 through 255 are treated as directly printable.

Furthermore, if I run with LANG=C I get

% for x in {1..254}; h[x]=${(V#)x}
zsh: character not in range

That seems wrong.  It does the right thing if "unsetopt multibyte"
is also in effect, but why should I have to explicitly do so?

Re: Quoting problem and crashes with ${(#)var}

[Peter Stephenson]

Bart Schaefer wrote:
> I'm a bit puzzled, given this test ...
> 
> }      if (isset(MULTIBYTE) && ires > 127) {
> 
> ... why ${(V)x} for x in 128 through 159 display as \u0080 through
> \u009f, but then 160 through 255 are treated as directly printable.

On my terminal, I've got different effects, which worries me more: if I
assign the UTF-8 representation of character 128 to a variable, ${(V)x}
tries to print it out directly (and it only shows up if send it through
xxd or equivalent).  Quite possibly the shell is linked with different
libraries.  (However, the ZLE function insert-unicode-char correctly
shows it as control character, ^ followed by A with a grave accent.)

Anyway, 128 to 159 aren't printable, 160 on are: in Unicode:

0080	<control>
...
009F	<control>
	= APPLICATION PROGRAM COMMAND
00A0	NO-BREAK SPACE
	= NBSP
	x (space - 0020)
	x (figure space - 2007)
	x (narrow no-break space - 202F)
	x (word joiner - 2060)
	x (zero width no-break space - FEFF)
	# <noBreak> 0020

(V) is documented as "make special characters visible".  That's exactly
what you're getting (but I'm not, for some reason---I'd be interested in
knowing where on your system the printability test is taking place).

> Furthermore, if I run with LANG=C I get
> 
> % for x in {1..254}; h[x]=${(V#)x}
> zsh: character not in range
> 
> That seems wrong.  It does the right thing if "unsetopt multibyte"
> is also in effect, but why should I have to explicitly do so?

Well, because you've (explicitly or otherwise) got it set to a locale
with no knowledge of characters beyond 127; it only knows about the
portable character set.  It's simply telling you it doesn't know what to
do with them.  It can't guess, because there's nothing really for it to
guess; locale C is a statement of ignorance about the non-portable,
post-ASCII world.  You're probably expecting 128 to print out a single octet
corresponding to the value of a C unsigned char with 128 in it.  Yet for
all the computer's been told, character 128 on the terminal you're using
is "really" the symbol for a deity worshipped by a Venusian cargo cult
which is represented by a string of 17 0xff's followed by 0x73 0x57, the
magic number indicating the transfer of energy between worshippers when
the Earth is high in the sky.  (All right, maybe this particular example
wasn't very realistic.)

What you're asking is for some kludged special case for LANG=C
(presumably we shouldn't second-guess any other character set).  It's
doable, I suppose, but I can't see the gain.  MULTIBYTE mode was never
intended to be backward compatible; that's exactly why NO_MULTIBYTE
exists.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Quoting problem and crashes with ${(#)var}

[Bart Schaefer]

On Feb 13,  9:11pm, Peter Stephenson wrote:
}
} Bart Schaefer wrote:
} > I'm a bit puzzled, given this test ...
} > 
} > }      if (isset(MULTIBYTE) && ires > 127) {
} > 
} > ... why ${(V)x} for x in 128 through 159 display as \u0080 through
} > \u009f, but then 160 through 255 are treated as directly printable.
} 
} On my terminal, I've got different effects, which worries me more: if I
} assign the UTF-8 representation of character 128 to a variable, ${(V)x}
} tries to print it out directly (and it only shows up if send it through
} xxd or equivalent).

Did you remember to use "print -R"?  If I do

	print ${(V)x}

then print interprets the \u0080 sequence and send a raw byte.  That
doesn't happen with

	print -R ${(V)x}

} (However, the ZLE function insert-unicode-char correctly
} shows it as control character, ^ followed by A with a grave accent.)

That's what I expected ${(V)x} to do, but instead it displays it as a
\u escape.

} > % for x in {1..254}; h[x]=${(V#)x}
} > zsh: character not in range
} > 
} > That seems wrong.
} 
} Well, because you've (explicitly or otherwise) got it set to a locale
} with no knowledge of characters beyond 127; it only knows about the
} portable character set.  It's simply telling you it doesn't know what to
} do with them.
} 
} What you're asking is for some kludged special case for LANG=C

Well, no, I'm not.  I'm asking for two things:

(1) when "character not in range" we don't treat it as a fatal error
and bail out of the whole surrounding loop; and

(2) regardless of the locale, single-byte values should always be
convertible to something "viewable", either \u00xy or \M-c.

There might be cases where "character not in range" is a fatal error,
but this doesn't seem as though it ought to be one of them.


--- /tmp/subst.c	2007-02-13 23:44:46.000000000 -0800
+++ /tmp/subst.c5229YwW	2007-02-13 23:44:46.000000000 -0800
@@ -1193,18 +1193,22 @@
 substevalchar(char *ptr)
 {
     zlong ires = mathevali(ptr);
-    int len;
+    int len = 0;
 
     if (errflag)
 	return NULL;
 #ifdef MULTIBYTE_SUPPORT
     if (isset(MULTIBYTE) && ires > 127) {
+	int one = noerrs;
 	char buf[10];
 
 	/* inefficient: should separate out \U handling from getkeystring */
 	sprintf(buf, "\\U%.8x", (unsigned int)ires);
+	noerrs = 1;
 	ptr = getkeystring(buf, &len, GETKEYS_BINDKEY, NULL);
-    } else
+	noerrs = one, errflag = 0;
+    }
+    if (len == 0)
 #endif
     {
 	ptr = zhalloc(2);

-- 

Re: Quoting problem and crashes with ${(#)var}

[Peter Stephenson]

> } (However, the ZLE function insert-unicode-char correctly
> } shows it as control character, ^ followed by A with a grave accent.)
> 
> That's what I expected ${(V)x} to do, but instead it displays it as a
> \u escape.
>...
> I'm asking for two things:
> 
> (1) when "character not in range" we don't treat it as a fatal error
> and bail out of the whole surrounding loop; and
> 
> (2) regardless of the locale, single-byte values should always be
> convertible to something "viewable", either \u00xy or \M-c.
> 
> There might be cases where "character not in range" is a fatal error,
> but this doesn't seem as though it ought to be one of them.

OK, I misinterpreted... I've certainly no objection to cleverer use of
escape codes for things that can't otherwise be output (where we can
test this, obviously).  I thought you were saying "treat this as a
directly printable character" in both cases.

Passing the character through the appropriate version of nicechar()
ought to help.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

Re: Quoting problem and crashes with ${(#)var}

[Bart Schaefer]

On Feb 14, 10:16am, Peter Stephenson wrote:
} Subject: Re: Quoting problem and crashes with ${(#)var}
}
} > } (However, the ZLE function insert-unicode-char correctly
} > } shows it as control character, ^ followed by A with a grave accent.)
} > 
} > That's what I expected ${(V)x} to do, but instead it displays it as a
} > \u escape.
} >...
} > I'm asking for two things:
} > 
} > (1) when "character not in range" we don't treat it as a fatal error
} > and bail out of the whole surrounding loop; and

On further thought, it ought to be possible to test whether a character
is within range, so how about this:  Extend the (X) flag to cover (#)
as well as (Qqe).  Then ${(X#):-129} throws an error when LANG=C and
multibyte support is enabled, but ${(#):-129} does not.

Code (but not yet doc) patch below (ignore the diff in 23173).

} > (2) regardless of the locale, single-byte values should always be
} > convertible to something "viewable", either \u00xy or \M-c.
} 
} Passing the character through the appropriate version of nicechar()
} ought to help.

Unfortunately that doesn't seem to be enough:  the (V) flag is already
calling nicedupstring().  It looks like wcs_nicechar() only uses the
carat-character format for control characters less than 0x80?  ZLE
must be doing its own printable conversion somewhere else.


--- Src/subst.c	2007-02-14 07:59:29.000000000 -0800
+++ /tmp/subst.c	2007-02-14 07:59:29.000000000 -0800
@@ -1193,7 +1193,7 @@
 substevalchar(char *ptr)
 {
     zlong ires = mathevali(ptr);
-    int len;
+    int len = 0;
 
     if (errflag)
 	return NULL;
@@ -1204,7 +1204,8 @@
 	/* inefficient: should separate out \U handling from getkeystring */
 	sprintf(buf, "\\U%.8x", (unsigned int)ires);
 	ptr = getkeystring(buf, &len, GETKEYS_BINDKEY, NULL);
-    } else
+    }
+    if (len == 0)
 #endif
     {
 	ptr = zhalloc(2);
@@ -2658,6 +2659,10 @@
     if (errflag)
 	return NULL;
     if (evalchar) {
+	int one = noerrs, oef = errflag, haserr;
+
+	if (!quoteerr)
+	    noerrs = 1;
 	/*
 	 * Evaluate the value numerically and output the result as
 	 * a character.
@@ -2669,15 +2674,24 @@
 
 	    for (avptr = aval, av2ptr = aval2; *avptr; avptr++, av2ptr++)
 	    {
-		if (!(*av2ptr = substevalchar(*avptr)))
-		    return NULL;
+		/* When noerrs = 1, the only error is out-of-memory */
+		if (!(*av2ptr = substevalchar(*avptr))) {
+		    haserr = 1;
+		    break;
+		}
 	    }
 	    *av2ptr = NULL;
 	    aval = aval2;
 	} else {
+	    /* When noerrs = 1, the only error is out-of-memory */
 	    if (!(val = substevalchar(val)))
-		return NULL;
+		haserr = 1;
 	}
+	noerrs = one;
+	if (!quoteerr)
+	    errflag = oef;
+	if (haserr || errflag)
+	    return NULL;
     }
     /*
      * This handles taking a length with ${#foo} and variations.

Re: Quoting problem and crashes with ${(#)var}

[Peter Stephenson]

Bart Schaefer <schaefer@brasslantern.com> wrote:
> } Passing the character through the appropriate version of nicechar()
> } ought to help.
> 
> Unfortunately that doesn't seem to be enough:  the (V) flag is already
> calling nicedupstring().  It looks like wcs_nicechar() only uses the
> carat-character format for control characters less than 0x80?  ZLE
> must be doing its own printable conversion somewhere else.

Ah.  Try this.

Yes, I wasn't using "print -r", so the \u code got converted back to a raw
character.

Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.156
diff -u -r1.156 utils.c
--- Src/utils.c	12 Feb 2007 16:43:42 -0000	1.156
+++ Src/utils.c	14 Feb 2007 16:18:14 -0000
@@ -514,10 +514,12 @@
 	    sprintf(buf, "\\U%.8x", (unsigned int)c);
 	    if (widthp)
 		*widthp = 10;
-	} else {
+	} else if (c >= 0x100) {
 	    sprintf(buf, "\\u%.4x", (unsigned int)c);
 	    if (widthp)
 		*widthp = 6;
+	} else {
+	    return nicechar((int)c);
 	}
 	if (swidep)
 	    *swidep = buf + *widthp;


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

Re: Quoting problem and crashes with ${(#)var}

[Bart Schaefer]

(Just back from a week-long vacation trip.)

On Feb 14,  4:19pm, Peter Stephenson wrote:
} Subject: Re: Quoting problem and crashes with ${(#)var}
}
} Ah.  Try this.

It's broken -- it doesn't set *widthp and *swidep when returning
nicechar().

I'm not certain the following is the right way to fix it; is something
more wide-char-aware than strcpy/strlen needed?  
 

--- ../zsh-forge/current/Src/utils.c	2007-02-25 14:06:38.000000000 -0800
+++ Src/utils.c	2007-02-25 15:01:14.000000000 -0800
@@ -519,7 +519,9 @@
 	    if (widthp)
 		*widthp = 6;
 	} else {
-	    return nicechar((int)c);
+	    strcpy(buf, nicechar((int)c));
+	    if (widthp)
+		*widthp = strlen(buf);
 	}
 	if (swidep)
 	    *swidep = buf + *widthp;

Re: Quoting problem and crashes with ${(#)var}

[Peter Stephenson]

Bart Schaefer wrote:
> I'm not certain the following is the right way to fix it; is something
> more wide-char-aware than strcpy/strlen needed?  
>
> -	    return nicechar((int)c);
> +	    strcpy(buf, nicechar((int)c));
> +	    if (widthp)
> +		*widthp = strlen(buf);

If printeightbit is set, then nicechar() may pass 8-bit characters,
which may be metafied, straight through, so ztrlen() is probably better.
printeightbit doesn't guaranteed to get character lengths right, but
counting metafied characters as 1 is more consistent.

I don't think we need to take account of wide characters here; we don't
have the information for that in this branch of the code.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

Re: Quoting problem and crashes with ${(#)var}

[Bart Schaefer]

On Feb 26, 10:34am, Peter Stephenson wrote:
} Subject: Re: Quoting problem and crashes with ${(#)var}
}
} Bart Schaefer wrote:
} > +	    strcpy(buf, nicechar((int)c));
} > +	    if (widthp)
} > +		*widthp = strlen(buf);
} 
} If printeightbit is set, then nicechar() may pass 8-bit characters,
} which may be metafied, straight through, so ztrlen() is probably better.
} printeightbit doesn't guaranteed to get character lengths right, but
} counting metafied characters as 1 is more consistent.

That may be more consistent, but a bit later on the same function computes
(*swidep = buf + *widthp) so I think what's wanted here is the actual
width rather than the visible width.

} I don't think we need to take account of wide characters here; we don't
} have the information for that in this branch of the code.

I'm more worried about embedded nuls, really, but that should be OK.  I'll
go ahead and commit it as above.

Re: Quoting problem and crashes with ${(#)var}

[Peter Stephenson]

Bart Schaefer wrote:
> } > +		*widthp = strlen(buf);
> } 
> } If printeightbit is set, then nicechar() may pass 8-bit characters,
> } which may be metafied, straight through, so ztrlen() is probably better.
> } printeightbit doesn't guaranteed to get character lengths right, but
> } counting metafied characters as 1 is more consistent.
> 
> That may be more consistent, but a bit later on the same function computes
> (*swidep = buf + *widthp) so I think what's wanted here is the actual
> width rather than the visible width.

That's true for swidep, it needs to point (in this case) to the NULL at
then end (as there's not wide character following).  However, *widthp
does need to be ztrlen().  So I think it they need to be separate.
(Before that nicechar() was added there couldn't be any metafied
characters at this point.)

pws


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

Re: Quoting problem and crashes with ${(#)var}

[Bart Schaefer]

On Feb 26,  4:24pm, Peter Stephenson wrote:
}
} (Before that nicechar() was added there couldn't be any metafied
} characters at this point.)

Is that going to cause problems in calling scopes?  That is, do we need
to avoid nicechar() here and instead copy some portion of it (or the
equivalent) into wcs_nicechar() directly?

Re: Quoting problem and crashes with ${(#)var}

[Peter Stephenson]

Bart Schaefer wrote:
> On Feb 26,  4:24pm, Peter Stephenson wrote:
> }
> } (Before that nicechar() was added there couldn't be any metafied
> } characters at this point.)
> 
> Is that going to cause problems in calling scopes?  That is, do we need
> to avoid nicechar() here and instead copy some portion of it (or the
> equivalent) into wcs_nicechar() directly?

It shouldn't cause problems: as wcs_nicechar() is already supposed to
handle printable wide characters cleanly, the return values may already
contain metafied characters.  See the end of the function where a normal
printable multibyte character is metafied.  By "at this point", I meant
specifically in the branch handling of non-printable wide characters or
other parse failures.

pws


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

PATCH: zsh/regex and =~

[Phil Pennock]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]

[ Sorry for not having one diff which creates new files, but "cvs diff -N"
  seems to be ignoring that little 'N' ]

The attached patch and files, which includes documentation, adds a new
loadable module, zsh/regex.  I've not examined widechar issues and which
regex libraries actually do handle these.  I've not looked at linkage
issues on platforms where regex (the POSIX interface, not regexp) is not
a part of libc.

This also includes my previous =~ work, replacing the previous patch.
I'm not sure that auto-unsetting REMATCH_PCRE is a good idea, so invite
comments; also as to which should be the default value; I suppose that
if pcre is not the default, then the warning can be put back in ...

My only test platform has been freebsd/amd64.

I've also cleaned up various memory leaks in zsh/pcre.
zsh/pcre now also sets $MATCH, not just $match.

I went with having $BASH_REMATCH be set instead of, rather than in
addition to, $MATCH and $match.  I'm again very open to persuasion here.

Oh, and the copyright notice in regex.c seems a bit disjointed, with
multiple names.  What's the copyright policy on newly contributed files?

zsh/regex provides the -regex-match conditional operator, the knowledge
of -regex-match and -pcre-match remains in cond.c with the COND_REGEX
handling for =~.

Also, I've decided that I much prefer the PCRE API to the POSIX regex
API.  :-)  I'm off to drink more wine to recover.

-Phil

[-- Attachment #2: regex-both.patch --]
[-- Type: text/x-diff, Size: 16747 bytes --]

Index: Doc/Zsh/cond.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/cond.yo,v
retrieving revision 1.3
diff -p -u -r1.3 cond.yo
--- Doc/Zsh/cond.yo	22 May 2000 15:01:35 -0000	1.3
+++ Doc/Zsh/cond.yo	28 Apr 2007 07:42:51 -0000
@@ -109,6 +109,11 @@ backward compatibility and should be con
 item(var(string) tt(!=) var(pattern))(
 true if var(string) does not match var(pattern).
 )
+item(var(string) tt(=~) var(regexp))(
+true if var(string) matches the PCRE regular expression
+var(regexp).  Requires the tt(zsh/pcre) module to be present,
+which is a compile-time option.
+)
 item(var(string1) tt(<) var(string2))(
 true if var(string1) comes before var(string2)
 based on ASCII value of their characters.
Index: Doc/Zsh/mod_pcre.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/mod_pcre.yo,v
retrieving revision 1.5
diff -p -u -r1.5 mod_pcre.yo
--- Doc/Zsh/mod_pcre.yo	20 Jun 2004 22:47:18 -0000	1.5
+++ Doc/Zsh/mod_pcre.yo	28 Apr 2007 07:42:51 -0000
@@ -22,14 +22,17 @@ Studies the previously-compiled PCRE whi
 matching.
 )
 findex(pcre_match)
-item(tt(pcre_match) [ tt(-a) var(arr) ] var(string))(
+item(tt(pcre_match) [ tt(-v) var(var) ] [ tt(-a) var(arr) ] var(string))(
 Returns successfully if tt(string) matches the previously-compiled
 PCRE.
 
 If the expression captures substrings within parentheses,
 tt(pcre_match) will set the array var($match) to those
 substrings, unless the tt(-a) option is given, in which
-case it will set the array var(arr).
+case it will set the array var(arr).  Similarly, the variable
+var($MATCH) will be set to the entire matched portion of the
+string, unless the tt(-v) option is given, in which var it will
+set the variable var(var).
 )
 enditem()
 
Index: Doc/Zsh/options.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/options.yo,v
retrieving revision 1.53
diff -p -u -r1.53 options.yo
--- Doc/Zsh/options.yo	5 Mar 2007 17:35:18 -0000	1.53
+++ Doc/Zsh/options.yo	28 Apr 2007 07:42:51 -0000
@@ -478,6 +478,19 @@ var(xx) is set to tt(LPAR())var(a b c)tt
 `var(fooabar foobbar foocbar)' instead of the default
 `var(fooa b cbar)'.
 )
+pindex(REMATCH_PCRE)
+cindex(regexp, PCRE)
+cindex(PCRE, regexp)
+item(tt(REMATCH_PCRE) <Z>)(
+If set, regular expression matching with the tt(=~) operator will use
+Perl-Compatible Regular Expressions from the PCRE library, if available.
+If not set, regular expressions will use the extended regexp syntax
+provided by the system libraries.
+Experimental:
+When zsh is invoked as tt(zsh), this option is initially set, but may be
+unset if the tt(zsh/pcre) module can not be loaded.  This behaviour, as
+well as the default status, is subject to change.
+)
 pindex(SH_GLOB)
 cindex(sh, globbing style)
 cindex(globbing style, sh)
@@ -1131,6 +1144,20 @@ enditem()
 
 subsect(Shell Emulation)
 startitem()
+pindex(BASH_REMATCH)
+cindex(bash, BASH_REMATCH variable)
+cindex(regexp, bash BASH_REMATCH variable)
+item(tt(BASH_REMATCH))(
+When set, matches performed with the tt(=~) operator will set the
+tt(BASH_REMATCH) array variable, instead of the default tt(MATCH) and
+tt(match) variables.  The first element of the tt(BASH_REMATCH) array
+will contain the entire matched text and subsequent elements will contain
+extracted substrings.  This option makes more sense when tt(KSH_ARRAYS) is
+also set, so that the entire matched portion is stored at index 0 and the
+first substring is at index 1.  Without this option, the tt(MATCH) variable
+contains the entire matched text and the tt(match) array variable will
+the substrings.
+)
 pindex(BSD_ECHO)
 cindex(echo, BSD compatible)
 item(tt(BSD_ECHO) <S>)(
Index: Src/cond.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/cond.c,v
retrieving revision 1.8
diff -p -u -r1.8 cond.c
--- Src/cond.c	30 May 2006 22:35:03 -0000	1.8
+++ Src/cond.c	28 Apr 2007 07:42:51 -0000
@@ -34,7 +34,7 @@ int tracingcond;
 
 static char *condstr[COND_MOD] = {
     "!", "&&", "||", "==", "!=", "<", ">", "-nt", "-ot", "-ef", "-eq",
-    "-ne", "-lt", "-gt", "-le", "-ge"
+    "-ne", "-lt", "-gt", "-le", "-ge", "=~"
 };
 
 /*
@@ -53,14 +53,14 @@ int
 evalcond(Estate state, char *fromtest)
 {
     struct stat *st;
-    char *left, *right;
+    char *left, *right, *overridename;
     Wordcode pcode;
     wordcode code;
     int ctype, htok = 0, ret;
 
  rec:
 
-    left = right = NULL;
+    left = right = overridename = NULL;
     pcode = state->pc++;
     code = *pcode;
     ctype = WC_COND_TYPE(code);
@@ -92,13 +92,42 @@ evalcond(Estate state, char *fromtest)
 	    state->pc = pcode + (WC_COND_SKIP(code) + 1);
 	    return ret;
 	}
+    case COND_REGEX:
+	{
+	    int loaded = 0;
+	    if (isset(REMATCHPCRE)) {
+		loaded = load_module_silence("zsh/pcre", 1);
+		if (loaded) {
+		    overridename = "-pcre-match";
+		} else {
+		    dosetopt(REMATCHPCRE, 0, 1);
+#if 0
+		    zwarnnam(fromtest, "zsh/pcre not available for regex");
+		    return 2;
+#endif
+		}
+	    }
+	    if (!loaded) {
+		loaded = load_module_silence("zsh/regex", 1);
+		if (loaded) {
+		    overridename = "-regex-match";
+		} else {
+		    zwarnnam(fromtest, "zsh/regex not available for regex");
+		    return 2;
+		}
+	    }
+	    ctype = COND_MODI;
+	}
     case COND_MOD:
     case COND_MODI:
 	{
 	    Conddef cd;
-	    char *name = ecgetstr(state, EC_NODUP, NULL), **strs;
+	    char *name = overridename;
+	    char **strs;
 	    int l = WC_COND_SKIP(code);
 
+	    if (name == NULL)
+		name = ecgetstr(state, EC_NODUP, NULL);
 	    if (ctype == COND_MOD)
 		strs = ecgetarr(state, l, EC_DUP, NULL);
 	    else {
@@ -139,7 +168,8 @@ evalcond(Estate state, char *fromtest)
 		    return !cd->handler(strs, cd->condid);
 		} else {
 		    zwarnnam(fromtest,
-			     "unrecognized condition: `%s'", name);
+			     "unrecognized condition: `%s'",
+			     name ? name : "<null>");
 		}
 	    }
 	    /* module not found, error */
Index: Src/options.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/options.c,v
retrieving revision 1.35
diff -p -u -r1.35 options.c
--- Src/options.c	15 Mar 2007 15:16:58 -0000	1.35
+++ Src/options.c	28 Apr 2007 07:42:51 -0000
@@ -88,6 +88,7 @@ static struct optname optns[] = {
 {{NULL, "banghist",	      OPT_NONBOURNE},		 BANGHIST},
 {{NULL, "bareglobqual",       OPT_EMULATE|OPT_ZSH},      BAREGLOBQUAL},
 {{NULL, "bashautolist",	      0},                        BASHAUTOLIST},
+{{NULL, "bashrematch",	      0},			 BASHREMATCH},
 {{NULL, "beep",		      OPT_ALL},			 BEEP},
 {{NULL, "bgnice",	      OPT_EMULATE|OPT_NONBOURNE},BGNICE},
 {{NULL, "braceccl",	      OPT_EMULATE},		 BRACECCL},
@@ -201,6 +202,7 @@ static struct optname optns[] = {
 {{NULL, "rcquotes",	      OPT_EMULATE},		 RCQUOTES},
 {{NULL, "rcs",		      OPT_ALL},			 RCS},
 {{NULL, "recexact",	      0},			 RECEXACT},
+{{NULL, "rematchpcre",	      OPT_ZSH},			 REMATCHPCRE},
 {{NULL, "restricted",	      OPT_SPECIAL},		 RESTRICTED},
 {{NULL, "rmstarsilent",	      OPT_BOURNE},		 RMSTARSILENT},
 {{NULL, "rmstarwait",	      0},			 RMSTARWAIT},
Index: Src/parse.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
retrieving revision 1.64
diff -p -u -r1.64 parse.c
--- Src/parse.c	23 Apr 2007 17:24:23 -0000	1.64
+++ Src/parse.c	28 Apr 2007 07:42:52 -0000
@@ -2124,6 +2124,12 @@ par_cond_triple(char *a, char *b, char *
 	ecstr(a);
 	ecstr(c);
 	ecadd(ecnpats++);
+    } else if ((b[0] == Equals || b[0] == '=') &&
+               (b[1] == '~' || b[1] == Tilde) && ~b[2]) {
+	ecadd(WCB_COND(COND_REGEX, 0));
+	ecstr(a);
+	ecstr(c);
+	ecadd(ecnpats++);
     } else if (b[0] == '-') {
 	if ((t0 = get_cond_num(b + 1)) > -1) {
 	    ecadd(WCB_COND(t0 + COND_NT, 0));
Index: Src/text.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/text.c,v
retrieving revision 1.19
diff -p -u -r1.19 text.c
--- Src/text.c	23 Apr 2007 15:24:00 -0000	1.19
+++ Src/text.c	28 Apr 2007 07:42:52 -0000
@@ -640,7 +640,7 @@ gettext2(Estate state)
 	    {
 		static char *c1[] = {
 		    "=", "!=", "<", ">", "-nt", "-ot", "-ef", "-eq",
-		    "-ne", "-lt", "-gt", "-le", "-ge"
+		    "-ne", "-lt", "-gt", "-le", "-ge", "=~"
 		};
 
 		int ctype;
@@ -724,7 +724,7 @@ gettext2(Estate state)
 			}
 			break;
 		    default:
-			if (ctype <= COND_GE) {
+			if (ctype < COND_MOD) {
 			    /* Binary test: `a = b' etc. */
 			    taddstr(ecgetstr(state, EC_NODUP, NULL));
 			    taddstr(" ");
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.112
diff -p -u -r1.112 zsh.h
--- Src/zsh.h	29 Mar 2007 21:35:39 -0000	1.112
+++ Src/zsh.h	28 Apr 2007 07:42:53 -0000
@@ -519,8 +519,9 @@ struct timedfn {
 #define COND_GT    13
 #define COND_LE    14
 #define COND_GE    15
-#define COND_MOD   16
-#define COND_MODI  17
+#define COND_REGEX 16
+#define COND_MOD   17
+#define COND_MODI  18
 
 typedef int (*CondHandler) _((char **, int));
 
@@ -1588,6 +1589,7 @@ enum {
     BANGHIST,
     BAREGLOBQUAL,
     BASHAUTOLIST,
+    BASHREMATCH,
     BEEP,
     BGNICE,
     BRACECCL,
@@ -1695,6 +1697,7 @@ enum {
     RCQUOTES,
     RCS,
     RECEXACT,
+    REMATCHPCRE,
     RESTRICTED,
     RMSTARSILENT,
     RMSTARWAIT,
Index: Src/Modules/pcre.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/pcre.c,v
retrieving revision 1.11
diff -p -u -r1.11 pcre.c
--- Src/Modules/pcre.c	5 Apr 2007 16:20:15 -0000	1.11
+++ Src/Modules/pcre.c	28 Apr 2007 07:42:53 -0000
@@ -3,7 +3,7 @@
  *
  * This file is part of zsh, the Z shell.
  *
- * Copyright (c) 2001, 2002, 2003, 2004 Clint Adams
+ * Copyright (c) 2001, 2002, 2003, 2004, 2007 Clint Adams
  * All rights reserved.
  *
  * Permission is hereby granted, without written agreement and without
@@ -42,6 +42,37 @@ static pcre_extra *pcre_hints;
 
 /**/
 static int
+zpcre_utf8_enabled(void)
+{
+#if defined(MULTIBYTE_SUPPORT) && defined(HAVE_NL_LANGINFO) && defined(CODESET)
+    static int have_utf8_pcre = -1;
+
+    /* value can toggle based on MULTIBYTE, so don't
+     * be too eager with caching */
+    if (have_utf8_pcre < -1)
+	return 0;
+
+    if (!isset(MULTIBYTE))
+	return 0;
+
+    if ((have_utf8_pcre == -1) &&
+        (!strcmp(nl_langinfo(CODESET), "UTF-8"))) {
+
+	if (pcre_config(PCRE_CONFIG_UTF8, &have_utf8_pcre))
+	    have_utf8_pcre = -2; /* erk, failed to ask */
+    }
+
+    if (have_utf8_pcre < 0)
+	return 0;
+    return have_utf8_pcre;
+
+#else
+    return 0;
+#endif
+}
+
+/**/
+static int
 bin_pcre_compile(char *nam, char **args, Options ops, UNUSED(int func))
 {
     int pcre_opts = 0, pcre_errptr;
@@ -52,8 +83,14 @@ bin_pcre_compile(char *nam, char **args,
     if(OPT_ISSET(ops,'m')) pcre_opts |= PCRE_MULTILINE;
     if(OPT_ISSET(ops,'x')) pcre_opts |= PCRE_EXTENDED;
     
+    if (zpcre_utf8_enabled())
+	pcre_opts |= PCRE_UTF8;
+
     pcre_hints = NULL;  /* Is this necessary? */
     
+    if (pcre_pattern)
+	pcre_free(pcre_pattern);
+
     pcre_pattern = pcre_compile(*args, pcre_opts, &pcre_error, &pcre_errptr, NULL);
     
     if (pcre_pattern == NULL)
@@ -100,37 +137,52 @@ bin_pcre_study(char *nam, UNUSED(char **
 
 /**/
 static int
-zpcre_get_substrings(char *arg, int *ovec, int ret, char *receptacle)
+zpcre_get_substrings(char *arg, int *ovec, int ret, char *matchvar, char *substravar, int matchedinarr)
 {
-    char **captures, **matches;
+    char **captures, **match_all, **matches;
+    int capture_start = 1;
 
-	if(!pcre_get_substring_list(arg, ovec, ret, (const char ***)&captures)) {
-	    
-	    matches = zarrdup(&captures[1]); /* first one would be entire string */
-	    if (receptacle == NULL)
-		setaparam("match", matches);
-	    else
-		setaparam(receptacle, matches);
-	    
-	    pcre_free_substring_list((const char **)captures);
-	}
+    if (matchedinarr)
+	capture_start = 0;
+    if (matchvar == NULL)
+	matchvar = "MATCH";
+    if (substravar == NULL)
+	substravar = "match";
+
+    /* captures[0] will be entire matched string, [1] first substring */
+    if(!pcre_get_substring_list(arg, ovec, ret, (const char ***)&captures)) {
+	match_all = ztrdup(captures[0]);
+	setsparam(matchvar, match_all);
+	matches = zarrdup(&captures[capture_start]);
+	setaparam(substravar, matches);
+	pcre_free_substring_list((const char **)captures);
+    }
 
-	return 0;
+    return 0;
 }
 
 /**/
 static int
 bin_pcre_match(char *nam, char **args, Options ops, UNUSED(int func))
 {
-    int ret, capcount, *ovec, ovecsize;
+    int ret, capcount, *ovec, ovecsize, c;
+    char *matched_portion = NULL;
     char *receptacle = NULL;
+    int return_value = 1;
+
+    if (pcre_pattern == NULL) {
+	zwarnnam(nam, "no pattern has been compiled");
+	return 1;
+    }
     
-    if(OPT_ISSET(ops,'a')) {
-	receptacle = *args++;
-	if(!*args) {
-	    zwarnnam(nam, "not enough arguments");
-	    return 1;
-	}
+    if(OPT_HASARG(ops,c='a')) {
+	receptacle = OPT_ARG(ops,c);
+    }
+    if(OPT_HASARG(ops,c='v')) {
+	matched_portion = OPT_ARG(ops,c);
+    }
+    if(!*args) {
+	zwarnnam(nam, "not enough arguments");
     }
     
     if ((ret = pcre_fullinfo(pcre_pattern, pcre_hints, PCRE_INFO_CAPTURECOUNT, &capcount)))
@@ -144,18 +196,20 @@ bin_pcre_match(char *nam, char **args, O
     
     ret = pcre_exec(pcre_pattern, pcre_hints, *args, strlen(*args), 0, 0, ovec, ovecsize);
     
-    if (ret==0) return 0;
-    else if (ret==PCRE_ERROR_NOMATCH) return 1; /* no match */
+    if (ret==0) return_value = 0;
+    else if (ret==PCRE_ERROR_NOMATCH) /* no match */;
     else if (ret>0) {
-	zpcre_get_substrings(*args, ovec, ret, receptacle);
-	return 0;
+	zpcre_get_substrings(*args, ovec, ret, matched_portion, receptacle, 0);
+	return_value = 0;
     }
     else {
 	zwarnnam(nam, "error in pcre_exec");
-	return 1;
     }
     
-    return 1;
+    if (ovec)
+	zfree(ovec, ovecsize*sizeof(int));
+
+    return return_value;
 }
 
 /**/
@@ -164,33 +218,63 @@ cond_pcre_match(char **a, int id)
 {
     pcre *pcre_pat;
     const char *pcre_err;
-    char *lhstr, *rhre;
+    char *lhstr, *rhre, *avar=NULL;
     int r = 0, pcre_opts = 0, pcre_errptr, capcnt, *ov, ovsize;
+    int return_value = 0;
+
+    if (zpcre_utf8_enabled())
+	pcre_opts |= PCRE_UTF8;
 
     lhstr = cond_str(a,0,0);
     rhre = cond_str(a,1,0);
+    pcre_pat = ov = NULL;
+
+    if (isset(BASHREMATCH))
+	avar="BASH_REMATCH";
 
     switch(id) {
 	 case CPCRE_PLAIN:
-		 pcre_pat = pcre_compile(rhre, pcre_opts, &pcre_err, &pcre_errptr, NULL);
-                 pcre_fullinfo(pcre_pat, NULL, PCRE_INFO_CAPTURECOUNT, &capcnt);
-    		 ovsize = (capcnt+1)*3;
-		 ov = zalloc(ovsize*sizeof(int));
-    		 r = pcre_exec(pcre_pat, NULL, lhstr, strlen(lhstr), 0, 0, ov, ovsize);
-    		if (r==0) return 1;
+		pcre_pat = pcre_compile(rhre, pcre_opts, &pcre_err, &pcre_errptr, NULL);
+		if (pcre_pat == NULL) {
+		    zwarn("failed to compile regexp /%s/: %s", rhre, pcre_err);
+		    break;
+		}
+                pcre_fullinfo(pcre_pat, NULL, PCRE_INFO_CAPTURECOUNT, &capcnt);
+    		ovsize = (capcnt+1)*3;
+		ov = zalloc(ovsize*sizeof(int));
+    		r = pcre_exec(pcre_pat, NULL, lhstr, strlen(lhstr), 0, 0, ov, ovsize);
+		/* r < 0 => error; r==0 match but not enough size in ov
+		 * r > 0 => (r-1) substrings found; r==1 => no substrings
+		 */
+    		if (r==0) {
+		    zwarn("reportable zsh problem: pcre_exec() returned 0");
+		    return_value = 1;
+		    break;
+		}
 	        else if (r==PCRE_ERROR_NOMATCH) return 0; /* no match */
+		else if (r<0) {
+		    zwarn("pcre_exec() error: %d", r);
+		    break;
+		}
                 else if (r>0) {
-		    zpcre_get_substrings(lhstr, ov, r, NULL);
-		    return 1;
+		    zpcre_get_substrings(lhstr, ov, r, NULL, avar, isset(BASHREMATCH));
+		    return_value = 1;
+		    break;
 		}
 		break;
     }
 
-    return 0;
+    if (pcre_pat)
+	pcre_free(pcre_pat);
+    if (ov)
+	zfree(ov, ovsize*sizeof(int));
+
+    return return_value;
 }
 
 static struct conddef cotab[] = {
     CONDDEF("pcre-match", CONDF_INFIX, cond_pcre_match, 0, 0, CPCRE_PLAIN)
+    /* CONDDEF can register =~ but it won't be found */
 };
 
 /**/
@@ -206,7 +290,7 @@ static struct conddef cotab[] = {
 static struct builtin bintab[] = {
     BUILTIN("pcre_compile", 0, bin_pcre_compile, 1, 1, 0, "aimx",  NULL),
     BUILTIN("pcre_study",   0, bin_pcre_study,   0, 0, 0, NULL,    NULL),
-    BUILTIN("pcre_match",   0, bin_pcre_match,   1, 2, 0, "a",    NULL)
+    BUILTIN("pcre_match",   0, bin_pcre_match,   1, 1, 0, "a:v:",    NULL)
 };
 
 

[-- Attachment #3: regex.mdd --]
[-- Type: text/plain, Size: 68 bytes --]

name=zsh/regex
link=dynamic
load=no

autobins=""

objects="regex.o"

[-- Attachment #4: regex.c --]
[-- Type: text/x-csrc, Size: 3879 bytes --]

/*
 * regex.c
 *
 * This file is part of zsh, the Z shell.
 *
 * Copyright (c) 2007 Phil Pennock
 * All Rights Reserved.
 *
 * Permission is hereby granted, without written agreement and without
 * license or royalty fees, to use, copy, modify, and distribute this
 * software and to distribute modified versions of this software for any
 * purpose, provided that the above copyright notice and the following
 * two paragraphs appear in all copies of this software.
 *
 * In no event shall Clint Adams or the Zsh Development Group be liable
 * to any party for direct, indirect, special, incidental, or consequential
 * damages arising out of the use of this software and its documentation,
 * even if Andrew Main and the Zsh Development Group have been advised of
 * the possibility of such damage.
 *
 * Clint Adams and the Zsh Development Group specifically disclaim any
 * warranties, including, but not limited to, the implied warranties of
 * merchantability and fitness for a particular purpose.  The software
 * provided hereunder is on an "as is" basis, and Andrew Main and the
 * Zsh Development Group have no obligation to provide maintenance,
 * support, updates, enhancements, or modifications.
 *
 */

#include "regex.mdh"
#include "regex.pro"

#include <regex.h>

/* we default to a vaguely modern syntax and set of capabilities */
#define ZREGEX_EXTENDED 0
/* if you want Basic syntax, make it an alternative options */

static void
zregex_regerrwarn(int r, regex_t *re, char *msg)
{
    char *errbuf;
    size_t errbufsz;

    errbufsz = regerror(r, re, NULL, 0);
    errbuf = zalloc(errbufsz*sizeof(char));
    regerror(r, re, errbuf, errbufsz);
    zwarn("%s: %s", msg, errbuf);
    zfree(errbuf, errbufsz);
}

/**/
static int
zcond_regex_match(char **a, int id)
{
    regex_t re;
    regmatch_t *m, *matches = NULL;
    size_t matchessz;
    char *lhstr, *rhre, *s, **arr, **x;
    int r, n, return_value, rcflags, reflags, nelem, start;

    lhstr = cond_str(a,0,0);
    rhre = cond_str(a,1,0);
    rcflags = reflags = 0;
    return_value = 0; /* 1 => matched successfully */

    switch(id) {
    case ZREGEX_EXTENDED:
	rcflags |= REG_EXTENDED;
	r = regcomp(&re, rhre, rcflags);
	if (r) {
	    zregex_regerrwarn(r, &re, "failed to compile regex");
	    break;
	}
	/* re.re_nsub is number of parenthesized groups, we also need
	 * 1 for the 0 offset, which is the entire matched portion
	 */
	matchessz = (re.re_nsub + 1) * sizeof(regmatch_t);
	matches = zalloc(matchessz);
	r = regexec(&re, lhstr, re.re_nsub+1, matches, reflags);
	if (r == REG_NOMATCH) /**/;
	else if (r == 0) {
	    return_value = 1;
	    if (isset(BASHREMATCH)) {
		start = 0;
		nelem = re.re_nsub + 1;
	    } else {
		start = 1;
		nelem = re.re_nsub;
	    }
	    /* entire matched portion + re_nsub substrings + NULL */
	    if (nelem) {
		arr = x = (char **) zalloc(sizeof(char *) * (nelem + 1));
		for (m = matches + start, n = start; n <= re.re_nsub; ++n, ++m, ++x) {
		    *x = ztrduppfx(lhstr + m->rm_so, m->rm_eo - m->rm_so);
		}
		*x = NULL;
	    }
	    if (isset(BASHREMATCH)) {
		setaparam("BASH_REMATCH", arr);
	    } else {
		m = matches;
		s = ztrduppfx(lhstr + m->rm_so, m->rm_eo - m->rm_so);
		setsparam("MATCH", s);
		if (nelem)
		    setaparam("match", arr);
	    }
	}
	else zregex_regerrwarn(r, &re, "regex matching error");
	break;
    }

    if (matches)
	zfree(matches, matchessz);
    regfree(&re);
    return return_value;
}

static struct conddef cotab[] = {
    CONDDEF("regex-match", CONDF_INFIX, zcond_regex_match, 0, 0, ZREGEX_EXTENDED)
};

/**/
int
setup_(UNUSED(Module m))
{
    return 0;
}

/**/
int
boot_(Module m)
{
    return !addconddefs(m->nam, cotab, sizeof(cotab)/sizeof(*cotab));
}

/**/
int
cleanup_(Module m)
{
    deleteconddefs(m->nam, cotab, sizeof(cotab)/sizeof(*cotab));
    return 0;
}

/**/
int
finish_(UNUSED(Module m))
{
    return 0;
}

Re: PATCH: zsh/regex and =~

[Phil Pennock]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

On 2007-04-28 at 00:56 -0700, Phil Pennock wrote:
> The attached patch and files, which includes documentation,

*sigh*  I say these things, I tinker with documentation for the options
etc, but do I remember to write documentation for the new module?

See attached, relative to the source tree as patched by the previous
mail.

[-- Attachment #2: zsh-regex-doc.patch --]
[-- Type: text/x-diff, Size: 1703 bytes --]

diff -pNur zsh-regexp/Doc/Makefile.in zsh-regex-doc/Doc/Makefile.in
--- zsh-regexp/Doc/Makefile.in	Sun Dec 17 08:02:02 2006
+++ zsh-regex-doc/Doc/Makefile.in	Sat Apr 28 01:08:36 2007
@@ -61,7 +61,7 @@ Zsh/mod_computil.yo \
 Zsh/mod_datetime.yo Zsh/mod_deltochar.yo \
 Zsh/mod_example.yo Zsh/mod_files.yo \
 Zsh/mod_mapfile.yo Zsh/mod_mathfunc.yo Zsh/mod_newuser.yo \
-Zsh/mod_parameter.yo Zsh/mod_pcre.yo \
+Zsh/mod_parameter.yo Zsh/mod_pcre.yo Zsh/mod_regex.yo \
 Zsh/mod_sched.yo Zsh/mod_socket.yo \
 Zsh/mod_stat.yo  Zsh/mod_system.yo Zsh/mod_tcp.yo \
 Zsh/mod_termcap.yo Zsh/mod_terminfo.yo \
diff -pNur zsh-regexp/Doc/Zsh/mod_regex.yo zsh-regex-doc/Doc/Zsh/mod_regex.yo
--- zsh-regexp/Doc/Zsh/mod_regex.yo	Wed Dec 31 16:00:00 1969
+++ zsh-regex-doc/Doc/Zsh/mod_regex.yo	Sat Apr 28 01:15:20 2007
@@ -0,0 +1,24 @@
+COMMENT(!MOD!zsh/regex
+Interface to the POSIX regex library.
+!MOD!)
+cindex(regular expressions, REGEX)
+The tt(zsh/regex) module makes available the following test condition:
+startitem()
+findex(regex-match)
+item(expr tt(-regex-match) regex)(
+Matches a string against a POSIX extended regular expression.
+The matched portion of the string will normally be placed in the tt($MATCH)
+variable.  If there are any capturing parentheses within the regex, then
+the tt($match) array variable will contain those.
+
+For example,
+
+[[ alphabetical -regex-match ^a([^a]+)a([^a]+)a ]] && print -l $MATCH X $match
+
+If tt(REGMATCH_PCRE) is not set, then the tt(=~) operator will automatically
+load this module as needed and will invoke the tt(-regex-match) operator.
+
+If tt(BASH_REMATCH) is set, then tt($BASH_REMATCH) will be set instead of
+tt($MATCH) and tt($match).
+)
+enditem()

Re: PATCH: zsh/regex and =~

[Phil Pennock]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

On 2007-04-28 at 00:56 -0700, Phil Pennock wrote:
> The attached patch and files, which includes documentation, adds a new
> loadable module, zsh/regex.  I've not examined widechar issues and which
> regex libraries actually do handle these.  I've not looked at linkage
> issues on platforms where regex (the POSIX interface, not regexp) is not
> a part of libc.

I noticed a gcc complaint that a variable might be used uninitialised;
this was bogus, but understandable.  The first patch below fixes it.
The second patch is a spelling correction in the docs.  Alternatively,
it might be a sign that the option needs to be renamed ...

arr is only initialised if nelem.  Later there are two references to it.
The second is guarded by "if (nelem)"; the first is guarded by
"if (isset(BASHREMATCH))".  If BASHREMATCH is set, nelem is always at
least 1.  Assuming re.re_nsub is never negative, which it isn't.  And
the patch also affirms this for sheer paranoia's sake.

-Phil

[-- Attachment #2: zsh-regex-bogus-unused.patch --]
[-- Type: text/x-diff, Size: 880 bytes --]

--- Src/Modules/regex.c.old	Sat Apr 28 17:40:12 2007
+++ Src/Modules/regex.c	Sat Apr 28 17:43:27 2007
@@ -75,6 +75,11 @@ zcond_regex_match(char **a, int id)
 	/* re.re_nsub is number of parenthesized groups, we also need
 	 * 1 for the 0 offset, which is the entire matched portion
 	 */
+	if (re.re_nsub < 0) {
+	    zwarn("INTERNAL ERROR: regcomp() returned "
+		    "negative subpattern count %d", re.re_nsub);
+	    break;
+	}
 	matchessz = (re.re_nsub + 1) * sizeof(regmatch_t);
 	matches = zalloc(matchessz);
 	r = regexec(&re, lhstr, re.re_nsub+1, matches, reflags);
@@ -88,6 +93,7 @@ zcond_regex_match(char **a, int id)
 		start = 1;
 		nelem = re.re_nsub;
 	    }
+	    arr = NULL; /* bogus gcc warning of used uninitialised */
 	    /* entire matched portion + re_nsub substrings + NULL */
 	    if (nelem) {
 		arr = x = (char **) zalloc(sizeof(char *) * (nelem + 1));

[-- Attachment #3: zsh-regex-speling.patch --]
[-- Type: text/x-diff, Size: 513 bytes --]

--- Doc/Zsh/mod_regex.yo.old	Sat Apr 28 17:45:57 2007
+++ Doc/Zsh/mod_regex.yo	Sat Apr 28 17:45:35 2007
@@ -15,7 +15,7 @@
 
 [[ alphabetical -regex-match ^a([^a]+)a([^a]+)a ]] && print -l $MATCH X $match
 
-If tt(REGMATCH_PCRE) is not set, then the tt(=~) operator will automatically
+If tt(REMATCH_PCRE) is not set, then the tt(=~) operator will automatically
 load this module as needed and will invoke the tt(-regex-match) operator.
 
 If tt(BASH_REMATCH) is set, then tt($BASH_REMATCH) will be set instead of

Re: PATCH: zsh/regex and =~

[Peter Stephenson]

Phil Pennock wrote:
> This also includes my previous =~ work, replacing the previous patch.

As I said, I want to do =~ with standard POSIX regular expressions and
no add-ons.  I've already got this working.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: zsh/regex and =~

[Peter Stephenson]

Peter Stephenson wrote:
> Phil Pennock wrote:
> > This also includes my previous =~ work, replacing the previous patch.
> 
> As I said, I want to do =~ with standard POSIX regular expressions and
> no add-ons.  I've already got this working.

Having looked further, I see that's the effect of what you've done (with
other stuff)... however, (unless I missed something in one of the
subsequent patches) the documentation for =~ still claims it's (always)
based on pcre.

There's no harm in the regular expression stuff being a separate module,
but if it's going to be it might as well be loadable if and only
if configure detects the POSIX regexp stuff.  That's easy---I can
patch that later.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: zsh/regex and =~

[Phil Pennock]

On 2007-04-29 at 16:28 +0100, Peter Stephenson wrote:
> Having looked further, I see that's the effect of what you've done (with
> other stuff)... however, (unless I missed something in one of the
> subsequent patches) the documentation for =~ still claims it's (always)
> based on pcre.

Meh.  I knew I was missing something.
Sorry.

Re: PATCH: zsh/regex and =~

[Peter Stephenson]

On Sat, 28 Apr 2007 00:56:35 -0700
Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> The attached patch and files, which includes documentation, adds a new
> loadable module, zsh/regex.  I've not examined widechar issues and which
> regex libraries actually do handle these.  I've not looked at linkage
> issues on platforms where regex (the POSIX interface, not regexp) is not
> a part of libc.
> 
> This also includes my previous =~ work, replacing the previous patch.
> I'm not sure that auto-unsetting REMATCH_PCRE is a good idea, so invite
> comments; also as to which should be the default value; I suppose that
> if pcre is not the default, then the warning can be put back in ...

Here's what I've ended up with after putting the patches together and
tweaking them.  The tweaks include (some of these deal with points
above):

I didn't put RE_MATCH_PCRE on by default; it seems to me to be a
user choice and having it depend on the shell emulation is more
confusing than useful.  Similarly, I've made the condition code
behave identically (apart, obviously, from the module =~ uses)
whether or not the option is set; if the module you didn't
ask for is not available, you get an error message rather than
a different sort of regular expression.

I've fixed the doc for =~ and tweaked at least one other minor typo.

I've added the option NO_CASE_MATCH to zsh/regex handling (like bash).
I don't know enough about the PCRE library to decide whether it's
sensible to have the same effect there, but if it is that's fine by me.

I've made regex.mdd only compile regex.c based on locating all
four POSIX functions.  This is the best way of handling conditional
support for modules: if it's not supported, there's nothing there
at all, so anything that tests will find out straight away it's
not supported and the disk isn't cluttered with unusable junk.

I've added some debugging code to test for a bad id passed to
the regex-match handler.  This doesn't do a heck of a lot at
the moment, but the case statement was looking a bit lonely
with just one entry.

regex.c is substantially Phil's work so he's mentioned in the copyright.
(As the licence makes clear, no one has actually transferred their
copyright anyway, so if there's ever any argy-bargy it has to be
sorted out line by line or even character by character---the copyrights
at the top of the files are rather less legally meaningful than they look.)


I'll commit this even if there are quibbles, to establish either a stick
in the ground or a line in the sand.  (I asked what the difference
was and was told one was horizontal and the other vertical.)

Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.61
diff -u -r1.61 configure.ac
--- configure.ac	5 Jan 2007 13:58:04 -0000	1.61
+++ configure.ac	1 May 2007 21:54:50 -0000
@@ -1135,7 +1135,8 @@
 	       erand48 open_memstream \
 	       wctomb iconv \
 	       grantpt unlockpt ptsname \
-	       htons ntohs)
+	       htons ntohs \
+	       regcomp regexec regerror regfree)
 AC_FUNC_STRCOLL
 
 if test x$enable_cap = xyes; then
Index: Doc/Makefile.in
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Makefile.in,v
retrieving revision 1.35
diff -u -r1.35 Makefile.in
--- Doc/Makefile.in	17 Dec 2006 16:02:02 -0000	1.35
+++ Doc/Makefile.in	1 May 2007 21:54:52 -0000
@@ -61,7 +61,7 @@
 Zsh/mod_datetime.yo Zsh/mod_deltochar.yo \
 Zsh/mod_example.yo Zsh/mod_files.yo \
 Zsh/mod_mapfile.yo Zsh/mod_mathfunc.yo Zsh/mod_newuser.yo \
-Zsh/mod_parameter.yo Zsh/mod_pcre.yo \
+Zsh/mod_parameter.yo Zsh/mod_pcre.yo Zsh/mod_regex.yo \
 Zsh/mod_sched.yo Zsh/mod_socket.yo \
 Zsh/mod_stat.yo  Zsh/mod_system.yo Zsh/mod_tcp.yo \
 Zsh/mod_termcap.yo Zsh/mod_terminfo.yo \
Index: Doc/Zsh/cond.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/cond.yo,v
retrieving revision 1.3
diff -u -r1.3 cond.yo
--- Doc/Zsh/cond.yo	22 May 2000 15:01:35 -0000	1.3
+++ Doc/Zsh/cond.yo	1 May 2007 21:54:52 -0000
@@ -109,6 +109,20 @@
 item(var(string) tt(!=) var(pattern))(
 true if var(string) does not match var(pattern).
 )
+item(var(string) tt(=~) var(regexp))(
+true if var(string) matches the regular expression
+var(regexp).  If the option tt(RE_MATCH_PCRE) is set
+var(regexp) is tested as a PCRE regular expression using
+the tt(zsh/pcre) module, else it is tested as a POSIX
+regular expression using the tt(zsh/regex) module.
+If the option tt(BASH_REMATCH) is set the array
+tt(BASH_REMATCH) is set to the substring that matched the pattern
+followed by the substrings that matched parenthesised
+subexpressions within the pattern; otherwise, the scalar parameter
+tt(MATCH) is set to the substring that matched the pattern and
+and the array tt(match) to the substrings that matched parenthesised
+subexpressions.
+)
 item(var(string1) tt(<) var(string2))(
 true if var(string1) comes before var(string2)
 based on ASCII value of their characters.
Index: Doc/Zsh/mod_pcre.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/mod_pcre.yo,v
retrieving revision 1.5
diff -u -r1.5 mod_pcre.yo
--- Doc/Zsh/mod_pcre.yo	20 Jun 2004 22:47:18 -0000	1.5
+++ Doc/Zsh/mod_pcre.yo	1 May 2007 21:54:52 -0000
@@ -22,14 +22,17 @@
 matching.
 )
 findex(pcre_match)
-item(tt(pcre_match) [ tt(-a) var(arr) ] var(string))(
+item(tt(pcre_match) [ tt(-v) var(var) ] [ tt(-a) var(arr) ] var(string))(
 Returns successfully if tt(string) matches the previously-compiled
 PCRE.
 
 If the expression captures substrings within parentheses,
 tt(pcre_match) will set the array var($match) to those
 substrings, unless the tt(-a) option is given, in which
-case it will set the array var(arr).
+case it will set the array var(arr).  Similarly, the variable
+var(MATCH) will be set to the entire matched portion of the
+string, unless the tt(-v) option is given, in which case the variable
+var(var) will be set.
 )
 enditem()
 
Index: Doc/Zsh/options.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/options.yo,v
retrieving revision 1.53
diff -u -r1.53 options.yo
--- Doc/Zsh/options.yo	5 Mar 2007 17:35:18 -0000	1.53
+++ Doc/Zsh/options.yo	1 May 2007 21:54:52 -0000
@@ -319,6 +319,13 @@
 can match the directory tt(CVS) owing to the presence of the globbing flag
 (unless the option tt(BARE_GLOB_QUAL) is unset).
 )
+pindex(CASE_MATCH)
+cindex(case-insensitive regular expression matches, option)
+cindex(regular expressions, case-insensitive matching, option)
+item(tt(CASE_MATCH) <D>)(
+Make regular expressions using the tt(zsh/regex) module (including
+matches with tt(=~)) sensitive to case.
+)
 pindex(CSH_NULL_GLOB)
 cindex(csh, null globbing style)
 cindex(null globbing style, csh)
@@ -478,6 +485,15 @@
 `var(fooabar foobbar foocbar)' instead of the default
 `var(fooa b cbar)'.
 )
+pindex(REMATCH_PCRE)
+cindex(regexp, PCRE)
+cindex(PCRE, regexp)
+item(tt(REMATCH_PCRE) <Z>)(
+If set, regular expression matching with the tt(=~) operator will use
+Perl-Compatible Regular Expressions from the PCRE library, if available.
+If not set, regular expressions will use the extended regexp syntax
+provided by the system libraries.
+)
 pindex(SH_GLOB)
 cindex(sh, globbing style)
 cindex(globbing style, sh)
@@ -1131,6 +1147,20 @@
 
 subsect(Shell Emulation)
 startitem()
+pindex(BASH_REMATCH)
+cindex(bash, BASH_REMATCH variable)
+cindex(regexp, bash BASH_REMATCH variable)
+item(tt(BASH_REMATCH))(
+When set, matches performed with the tt(=~) operator will set the
+tt(BASH_REMATCH) array variable, instead of the default tt(MATCH) and
+tt(match) variables.  The first element of the tt(BASH_REMATCH) array
+will contain the entire matched text and subsequent elements will contain
+extracted substrings.  This option makes more sense when tt(KSH_ARRAYS) is
+also set, so that the entire matched portion is stored at index 0 and the
+first substring is at index 1.  Without this option, the tt(MATCH) variable
+contains the entire matched text and the tt(match) array variable contains
+substrings.
+)
 pindex(BSD_ECHO)
 cindex(echo, BSD compatible)
 item(tt(BSD_ECHO) <S>)(
Index: Src/cond.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/cond.c,v
retrieving revision 1.8
diff -u -r1.8 cond.c
--- Src/cond.c	30 May 2006 22:35:03 -0000	1.8
+++ Src/cond.c	1 May 2007 21:54:52 -0000
@@ -34,7 +34,7 @@
 
 static char *condstr[COND_MOD] = {
     "!", "&&", "||", "==", "!=", "<", ">", "-nt", "-ot", "-ef", "-eq",
-    "-ne", "-lt", "-gt", "-le", "-ge"
+    "-ne", "-lt", "-gt", "-le", "-ge", "=~"
 };
 
 /*
@@ -53,14 +53,14 @@
 evalcond(Estate state, char *fromtest)
 {
     struct stat *st;
-    char *left, *right;
+    char *left, *right, *overridename, overridebuf[13];
     Wordcode pcode;
     wordcode code;
     int ctype, htok = 0, ret;
 
  rec:
 
-    left = right = NULL;
+    left = right = overridename = NULL;
     pcode = state->pc++;
     code = *pcode;
     ctype = WC_COND_TYPE(code);
@@ -92,13 +92,28 @@
 	    state->pc = pcode + (WC_COND_SKIP(code) + 1);
 	    return ret;
 	}
+    case COND_REGEX:
+	{
+	    char *modname = isset(REMATCHPCRE) ? "zsh/pcre" : "zsh/regex";
+	    if (!load_module_silence(modname, 1)) {
+		zwarnnam(fromtest, "%s not available for regex",
+			 modname);
+		return 2;
+	    }
+	    sprintf(overridename = overridebuf, "-%s-match", modname+4);
+	    ctype = COND_MODI;
+	}
+	/*FALLTHROUGH*/
     case COND_MOD:
     case COND_MODI:
 	{
 	    Conddef cd;
-	    char *name = ecgetstr(state, EC_NODUP, NULL), **strs;
+	    char *name = overridename;
+	    char **strs;
 	    int l = WC_COND_SKIP(code);
 
+	    if (name == NULL)
+		name = ecgetstr(state, EC_NODUP, NULL);
 	    if (ctype == COND_MOD)
 		strs = ecgetarr(state, l, EC_DUP, NULL);
 	    else {
@@ -139,7 +154,8 @@
 		    return !cd->handler(strs, cd->condid);
 		} else {
 		    zwarnnam(fromtest,
-			     "unrecognized condition: `%s'", name);
+			     "unrecognized condition: `%s'",
+			     name ? name : "<null>");
 		}
 	    }
 	    /* module not found, error */
Index: Src/options.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/options.c,v
retrieving revision 1.35
diff -u -r1.35 options.c
--- Src/options.c	15 Mar 2007 15:16:58 -0000	1.35
+++ Src/options.c	1 May 2007 21:54:54 -0000
@@ -88,11 +88,13 @@
 {{NULL, "banghist",	      OPT_NONBOURNE},		 BANGHIST},
 {{NULL, "bareglobqual",       OPT_EMULATE|OPT_ZSH},      BAREGLOBQUAL},
 {{NULL, "bashautolist",	      0},                        BASHAUTOLIST},
+{{NULL, "bashrematch",	      0},			 BASHREMATCH},
 {{NULL, "beep",		      OPT_ALL},			 BEEP},
 {{NULL, "bgnice",	      OPT_EMULATE|OPT_NONBOURNE},BGNICE},
 {{NULL, "braceccl",	      OPT_EMULATE},		 BRACECCL},
 {{NULL, "bsdecho",	      OPT_EMULATE|OPT_SH},	 BSDECHO},
 {{NULL, "caseglob",	      OPT_ALL},			 CASEGLOB},
+{{NULL, "casematch",	      OPT_ALL},			 CASEMATCH},
 {{NULL, "cbases",	      0},			 CBASES},
 {{NULL, "cdablevars",	      OPT_EMULATE},		 CDABLEVARS},
 {{NULL, "chasedots",	      OPT_EMULATE},		 CHASEDOTS},
@@ -201,6 +203,7 @@
 {{NULL, "rcquotes",	      OPT_EMULATE},		 RCQUOTES},
 {{NULL, "rcs",		      OPT_ALL},			 RCS},
 {{NULL, "recexact",	      0},			 RECEXACT},
+{{NULL, "rematchpcre",	      0},			 REMATCHPCRE},
 {{NULL, "restricted",	      OPT_SPECIAL},		 RESTRICTED},
 {{NULL, "rmstarsilent",	      OPT_BOURNE},		 RMSTARSILENT},
 {{NULL, "rmstarwait",	      0},			 RMSTARWAIT},
Index: Src/parse.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
retrieving revision 1.64
diff -u -r1.64 parse.c
--- Src/parse.c	23 Apr 2007 17:24:23 -0000	1.64
+++ Src/parse.c	1 May 2007 21:54:54 -0000
@@ -2124,6 +2124,12 @@
 	ecstr(a);
 	ecstr(c);
 	ecadd(ecnpats++);
+    } else if ((b[0] == Equals || b[0] == '=') &&
+               (b[1] == '~' || b[1] == Tilde) && ~b[2]) {
+	ecadd(WCB_COND(COND_REGEX, 0));
+	ecstr(a);
+	ecstr(c);
+	ecadd(ecnpats++);
     } else if (b[0] == '-') {
 	if ((t0 = get_cond_num(b + 1)) > -1) {
 	    ecadd(WCB_COND(t0 + COND_NT, 0));
Index: Src/text.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/text.c,v
retrieving revision 1.19
diff -u -r1.19 text.c
--- Src/text.c	23 Apr 2007 15:24:00 -0000	1.19
+++ Src/text.c	1 May 2007 21:54:54 -0000
@@ -640,7 +640,7 @@
 	    {
 		static char *c1[] = {
 		    "=", "!=", "<", ">", "-nt", "-ot", "-ef", "-eq",
-		    "-ne", "-lt", "-gt", "-le", "-ge"
+		    "-ne", "-lt", "-gt", "-le", "-ge", "=~"
 		};
 
 		int ctype;
@@ -724,7 +724,7 @@
 			}
 			break;
 		    default:
-			if (ctype <= COND_GE) {
+			if (ctype < COND_MOD) {
 			    /* Binary test: `a = b' etc. */
 			    taddstr(ecgetstr(state, EC_NODUP, NULL));
 			    taddstr(" ");
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.112
diff -u -r1.112 zsh.h
--- Src/zsh.h	29 Mar 2007 21:35:39 -0000	1.112
+++ Src/zsh.h	1 May 2007 21:54:57 -0000
@@ -519,8 +519,9 @@
 #define COND_GT    13
 #define COND_LE    14
 #define COND_GE    15
-#define COND_MOD   16
-#define COND_MODI  17
+#define COND_REGEX 16
+#define COND_MOD   17
+#define COND_MODI  18
 
 typedef int (*CondHandler) _((char **, int));
 
@@ -1588,11 +1589,13 @@
     BANGHIST,
     BAREGLOBQUAL,
     BASHAUTOLIST,
+    BASHREMATCH,
     BEEP,
     BGNICE,
     BRACECCL,
     BSDECHO,
     CASEGLOB,
+    CASEMATCH,
     CBASES,
     CDABLEVARS,
     CHASEDOTS,
@@ -1695,6 +1698,7 @@
     RCQUOTES,
     RCS,
     RECEXACT,
+    REMATCHPCRE,
     RESTRICTED,
     RMSTARSILENT,
     RMSTARWAIT,
Index: Src/Modules/pcre.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/pcre.c,v
retrieving revision 1.11
diff -u -r1.11 pcre.c
--- Src/Modules/pcre.c	5 Apr 2007 16:20:15 -0000	1.11
+++ Src/Modules/pcre.c	1 May 2007 21:54:57 -0000
@@ -3,7 +3,7 @@
  *
  * This file is part of zsh, the Z shell.
  *
- * Copyright (c) 2001, 2002, 2003, 2004 Clint Adams
+ * Copyright (c) 2001, 2002, 2003, 2004, 2007 Clint Adams
  * All rights reserved.
  *
  * Permission is hereby granted, without written agreement and without
@@ -42,6 +42,37 @@
 
 /**/
 static int
+zpcre_utf8_enabled(void)
+{
+#if defined(MULTIBYTE_SUPPORT) && defined(HAVE_NL_LANGINFO) && defined(CODESET)
+    static int have_utf8_pcre = -1;
+
+    /* value can toggle based on MULTIBYTE, so don't
+     * be too eager with caching */
+    if (have_utf8_pcre < -1)
+	return 0;
+
+    if (!isset(MULTIBYTE))
+	return 0;
+
+    if ((have_utf8_pcre == -1) &&
+        (!strcmp(nl_langinfo(CODESET), "UTF-8"))) {
+
+	if (pcre_config(PCRE_CONFIG_UTF8, &have_utf8_pcre))
+	    have_utf8_pcre = -2; /* erk, failed to ask */
+    }
+
+    if (have_utf8_pcre < 0)
+	return 0;
+    return have_utf8_pcre;
+
+#else
+    return 0;
+#endif
+}
+
+/**/
+static int
 bin_pcre_compile(char *nam, char **args, Options ops, UNUSED(int func))
 {
     int pcre_opts = 0, pcre_errptr;
@@ -52,8 +83,14 @@
     if(OPT_ISSET(ops,'m')) pcre_opts |= PCRE_MULTILINE;
     if(OPT_ISSET(ops,'x')) pcre_opts |= PCRE_EXTENDED;
     
+    if (zpcre_utf8_enabled())
+	pcre_opts |= PCRE_UTF8;
+
     pcre_hints = NULL;  /* Is this necessary? */
     
+    if (pcre_pattern)
+	pcre_free(pcre_pattern);
+
     pcre_pattern = pcre_compile(*args, pcre_opts, &pcre_error, &pcre_errptr, NULL);
     
     if (pcre_pattern == NULL)
@@ -100,37 +137,52 @@
 
 /**/
 static int
-zpcre_get_substrings(char *arg, int *ovec, int ret, char *receptacle)
+zpcre_get_substrings(char *arg, int *ovec, int ret, char *matchvar, char *substravar, int matchedinarr)
 {
-    char **captures, **matches;
+    char **captures, **match_all, **matches;
+    int capture_start = 1;
 
-	if(!pcre_get_substring_list(arg, ovec, ret, (const char ***)&captures)) {
-	    
-	    matches = zarrdup(&captures[1]); /* first one would be entire string */
-	    if (receptacle == NULL)
-		setaparam("match", matches);
-	    else
-		setaparam(receptacle, matches);
-	    
-	    pcre_free_substring_list((const char **)captures);
-	}
+    if (matchedinarr)
+	capture_start = 0;
+    if (matchvar == NULL)
+	matchvar = "MATCH";
+    if (substravar == NULL)
+	substravar = "match";
+
+    /* captures[0] will be entire matched string, [1] first substring */
+    if(!pcre_get_substring_list(arg, ovec, ret, (const char ***)&captures)) {
+	match_all = ztrdup(captures[0]);
+	setsparam(matchvar, match_all);
+	matches = zarrdup(&captures[capture_start]);
+	setaparam(substravar, matches);
+	pcre_free_substring_list((const char **)captures);
+    }
 
-	return 0;
+    return 0;
 }
 
 /**/
 static int
 bin_pcre_match(char *nam, char **args, Options ops, UNUSED(int func))
 {
-    int ret, capcount, *ovec, ovecsize;
+    int ret, capcount, *ovec, ovecsize, c;
+    char *matched_portion = NULL;
     char *receptacle = NULL;
+    int return_value = 1;
+
+    if (pcre_pattern == NULL) {
+	zwarnnam(nam, "no pattern has been compiled");
+	return 1;
+    }
     
-    if(OPT_ISSET(ops,'a')) {
-	receptacle = *args++;
-	if(!*args) {
-	    zwarnnam(nam, "not enough arguments");
-	    return 1;
-	}
+    if(OPT_HASARG(ops,c='a')) {
+	receptacle = OPT_ARG(ops,c);
+    }
+    if(OPT_HASARG(ops,c='v')) {
+	matched_portion = OPT_ARG(ops,c);
+    }
+    if(!*args) {
+	zwarnnam(nam, "not enough arguments");
     }
     
     if ((ret = pcre_fullinfo(pcre_pattern, pcre_hints, PCRE_INFO_CAPTURECOUNT, &capcount)))
@@ -144,18 +196,20 @@
     
     ret = pcre_exec(pcre_pattern, pcre_hints, *args, strlen(*args), 0, 0, ovec, ovecsize);
     
-    if (ret==0) return 0;
-    else if (ret==PCRE_ERROR_NOMATCH) return 1; /* no match */
+    if (ret==0) return_value = 0;
+    else if (ret==PCRE_ERROR_NOMATCH) /* no match */;
     else if (ret>0) {
-	zpcre_get_substrings(*args, ovec, ret, receptacle);
-	return 0;
+	zpcre_get_substrings(*args, ovec, ret, matched_portion, receptacle, 0);
+	return_value = 0;
     }
     else {
 	zwarnnam(nam, "error in pcre_exec");
-	return 1;
     }
     
-    return 1;
+    if (ovec)
+	zfree(ovec, ovecsize*sizeof(int));
+
+    return return_value;
 }
 
 /**/
@@ -164,33 +218,63 @@
 {
     pcre *pcre_pat;
     const char *pcre_err;
-    char *lhstr, *rhre;
+    char *lhstr, *rhre, *avar=NULL;
     int r = 0, pcre_opts = 0, pcre_errptr, capcnt, *ov, ovsize;
+    int return_value = 0;
+
+    if (zpcre_utf8_enabled())
+	pcre_opts |= PCRE_UTF8;
 
     lhstr = cond_str(a,0,0);
     rhre = cond_str(a,1,0);
+    pcre_pat = ov = NULL;
+
+    if (isset(BASHREMATCH))
+	avar="BASH_REMATCH";
 
     switch(id) {
 	 case CPCRE_PLAIN:
-		 pcre_pat = pcre_compile(rhre, pcre_opts, &pcre_err, &pcre_errptr, NULL);
-                 pcre_fullinfo(pcre_pat, NULL, PCRE_INFO_CAPTURECOUNT, &capcnt);
-    		 ovsize = (capcnt+1)*3;
-		 ov = zalloc(ovsize*sizeof(int));
-    		 r = pcre_exec(pcre_pat, NULL, lhstr, strlen(lhstr), 0, 0, ov, ovsize);
-    		if (r==0) return 1;
+		pcre_pat = pcre_compile(rhre, pcre_opts, &pcre_err, &pcre_errptr, NULL);
+		if (pcre_pat == NULL) {
+		    zwarn("failed to compile regexp /%s/: %s", rhre, pcre_err);
+		    break;
+		}
+                pcre_fullinfo(pcre_pat, NULL, PCRE_INFO_CAPTURECOUNT, &capcnt);
+    		ovsize = (capcnt+1)*3;
+		ov = zalloc(ovsize*sizeof(int));
+    		r = pcre_exec(pcre_pat, NULL, lhstr, strlen(lhstr), 0, 0, ov, ovsize);
+		/* r < 0 => error; r==0 match but not enough size in ov
+		 * r > 0 => (r-1) substrings found; r==1 => no substrings
+		 */
+    		if (r==0) {
+		    zwarn("reportable zsh problem: pcre_exec() returned 0");
+		    return_value = 1;
+		    break;
+		}
 	        else if (r==PCRE_ERROR_NOMATCH) return 0; /* no match */
+		else if (r<0) {
+		    zwarn("pcre_exec() error: %d", r);
+		    break;
+		}
                 else if (r>0) {
-		    zpcre_get_substrings(lhstr, ov, r, NULL);
-		    return 1;
+		    zpcre_get_substrings(lhstr, ov, r, NULL, avar, isset(BASHREMATCH));
+		    return_value = 1;
+		    break;
 		}
 		break;
     }
 
-    return 0;
+    if (pcre_pat)
+	pcre_free(pcre_pat);
+    if (ov)
+	zfree(ov, ovsize*sizeof(int));
+
+    return return_value;
 }
 
 static struct conddef cotab[] = {
     CONDDEF("pcre-match", CONDF_INFIX, cond_pcre_match, 0, 0, CPCRE_PLAIN)
+    /* CONDDEF can register =~ but it won't be found */
 };
 
 /**/
@@ -206,7 +290,7 @@
 static struct builtin bintab[] = {
     BUILTIN("pcre_compile", 0, bin_pcre_compile, 1, 1, 0, "aimx",  NULL),
     BUILTIN("pcre_study",   0, bin_pcre_study,   0, 0, 0, NULL,    NULL),
-    BUILTIN("pcre_match",   0, bin_pcre_match,   1, 2, 0, "a",    NULL)
+    BUILTIN("pcre_match",   0, bin_pcre_match,   1, 1, 0, "a:v:",    NULL)
 };
 
 
Index: Src/Modules/regex.c
===================================================================
RCS file: Src/Modules/regex.c
diff -N Src/Modules/regex.c
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Src/Modules/regex.c	1 May 2007 21:54:57 -0000
@@ -0,0 +1,161 @@
+/*
+ * regex.c
+ *
+ * This file is part of zsh, the Z shell.
+ *
+ * Copyright (c) 2007 Phil Pennock
+ * All Rights Reserved.
+ *
+ * Permission is hereby granted, without written agreement and without
+ * license or royalty fees, to use, copy, modify, and distribute this
+ * software and to distribute modified versions of this software for any
+ * purpose, provided that the above copyright notice and the following
+ * two paragraphs appear in all copies of this software.
+ *
+ * In no event shall Phil Pennock or the Zsh Development Group be liable
+ * to any party for direct, indirect, special, incidental, or consequential
+ * damages arising out of the use of this software and its documentation,
+ * even if Phil Pennock and the Zsh Development Group have been advised of
+ * the possibility of such damage.
+ *
+ * Phil Pennock and the Zsh Development Group specifically disclaim any
+ * warranties, including, but not limited to, the implied warranties of
+ * merchantability and fitness for a particular purpose.  The software
+ * provided hereunder is on an "as is" basis, and Phil Pennock and the
+ * Zsh Development Group have no obligation to provide maintenance,
+ * support, updates, enhancements, or modifications.
+ *
+ */
+
+#include "regex.mdh"
+#include "regex.pro"
+
+#include <regex.h>
+
+/* we default to a vaguely modern syntax and set of capabilities */
+#define ZREGEX_EXTENDED 0
+/* if you want Basic syntax, make it an alternative options */
+
+static void
+zregex_regerrwarn(int r, regex_t *re, char *msg)
+{
+    char *errbuf;
+    size_t errbufsz;
+
+    errbufsz = regerror(r, re, NULL, 0);
+    errbuf = zalloc(errbufsz*sizeof(char));
+    regerror(r, re, errbuf, errbufsz);
+    zwarn("%s: %s", msg, errbuf);
+    zfree(errbuf, errbufsz);
+}
+
+/**/
+static int
+zcond_regex_match(char **a, int id)
+{
+    regex_t re;
+    regmatch_t *m, *matches = NULL;
+    size_t matchessz;
+    char *lhstr, *rhre, *s, **arr, **x;
+    int r, n, return_value, rcflags, reflags, nelem, start;
+
+    lhstr = cond_str(a,0,0);
+    rhre = cond_str(a,1,0);
+    rcflags = reflags = 0;
+    return_value = 0; /* 1 => matched successfully */
+
+    switch(id) {
+    case ZREGEX_EXTENDED:
+	rcflags |= REG_EXTENDED;
+	if (!isset(CASEMATCH))
+	    rcflags |= REG_ICASE;
+	r = regcomp(&re, rhre, rcflags);
+	if (r) {
+	    zregex_regerrwarn(r, &re, "failed to compile regex");
+	    break;
+	}
+	/* re.re_nsub is number of parenthesized groups, we also need
+	 * 1 for the 0 offset, which is the entire matched portion
+	 */
+	if (re.re_nsub < 0) {
+	    zwarn("INTERNAL ERROR: regcomp() returned "
+		    "negative subpattern count %d", re.re_nsub);
+	    break;
+	}
+	matchessz = (re.re_nsub + 1) * sizeof(regmatch_t);
+	matches = zalloc(matchessz);
+	r = regexec(&re, lhstr, re.re_nsub+1, matches, reflags);
+	if (r == REG_NOMATCH) /**/;
+	else if (r == 0) {
+	    return_value = 1;
+	    if (isset(BASHREMATCH)) {
+		start = 0;
+		nelem = re.re_nsub + 1;
+	    } else {
+		start = 1;
+		nelem = re.re_nsub;
+	    }
+	    arr = NULL; /* bogus gcc warning of used uninitialised */
+	    /* entire matched portion + re_nsub substrings + NULL */
+	    if (nelem) {
+		arr = x = (char **) zalloc(sizeof(char *) * (nelem + 1));
+		for (m = matches + start, n = start; n <= re.re_nsub; ++n, ++m, ++x) {
+		    *x = ztrduppfx(lhstr + m->rm_so, m->rm_eo - m->rm_so);
+		}
+		*x = NULL;
+	    }
+	    if (isset(BASHREMATCH)) {
+		setaparam("BASH_REMATCH", arr);
+	    } else {
+		m = matches;
+		s = ztrduppfx(lhstr + m->rm_so, m->rm_eo - m->rm_so);
+		setsparam("MATCH", s);
+		if (nelem)
+		    setaparam("match", arr);
+	    }
+	}
+	else zregex_regerrwarn(r, &re, "regex matching error");
+	break;
+    default:
+	DPUTS(1, "bad regex option");
+	break;
+    }
+
+    if (matches)
+	zfree(matches, matchessz);
+    regfree(&re);
+    return return_value;
+}
+
+static struct conddef cotab[] = {
+    CONDDEF("regex-match", CONDF_INFIX, zcond_regex_match, 0, 0, ZREGEX_EXTENDED)
+};
+
+/**/
+int
+setup_(UNUSED(Module m))
+{
+    return 0;
+}
+
+/**/
+int
+boot_(Module m)
+{
+    return !addconddefs(m->nam, cotab, sizeof(cotab)/sizeof(*cotab));
+}
+
+/**/
+int
+cleanup_(Module m)
+{
+    deleteconddefs(m->nam, cotab, sizeof(cotab)/sizeof(*cotab));
+    return 0;
+}
+
+/**/
+int
+finish_(UNUSED(Module m))
+{
+    return 0;
+}
Index: Src/Modules/regex.mdd
===================================================================
RCS file: Src/Modules/regex.mdd
diff -N Src/Modules/regex.mdd
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Src/Modules/regex.mdd	1 May 2007 21:54:57 -0000
@@ -0,0 +1,10 @@
+name=zsh/regex
+link=`if test x$ac_cv_func_regcomp = xyes && \
+         test x$ac_cv_func_regexec = xyes && \
+         test x$ac_cv_func_regerror = xyes && \
+         test x$ac_cv_func_regfree = xyes; then echo dynamic; else echo no; fi`
+load=no
+
+autobins=""
+
+objects="regex.o"

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: zsh/regex and =~

[Phil Pennock]

On 2007-05-01 at 22:59 +0100, Peter Stephenson wrote:
> I didn't put RE_MATCH_PCRE on by default; it seems to me to be a
> user choice and having it depend on the shell emulation is more
> confusing than useful.  Similarly, I've made the condition code
> behave identically (apart, obviously, from the module =~ uses)
> whether or not the option is set; if the module you didn't
> ask for is not available, you get an error message rather than
> a different sort of regular expression.

Makes sense.

> I've fixed the doc for =~ and tweaked at least one other minor typo.

English really is my first language.  *cough*
Similarly, regex.c:
 /* if you want Basic syntax, make it an alternative options */

s/options/option/

> I've added the option NO_CASE_MATCH to zsh/regex handling (like bash).
> I don't know enough about the PCRE library to decide whether it's
> sensible to have the same effect there, but if it is that's fine by me.

It is.  If you look at bin_pcre_compile(), you can see that if the -i
option is passed, it ORs in the bitflag PCRE_CASELESS.  Both
bin_pcre_compile() and cond_pcre_match() need the obvious:
  if (!isset(CASEMATCH)) pcre_opts |= PCRE_CASELESS;
at a place of your choosing, before the call to pcre_compile().

You _can_ achieve the same thing using embedded options:

% [[ FoO =~ ^(?i)f([aeiou]+) ]] && print -l $MATCH $match
FoO
oO

but I see no reason to not be as compatible as possible in interpreting
basic behavioural options.  Aside: the embedded options parsing means
that the default stringification by ruby of a regexp type can be
directly parsed by zsh's pcre_compile, a happy accident for me which I
noticed yesterday.

% zrb '/^f([aeiou]+)/' ; print $zrb_value
(?-mix:^f([aeiou]+))
% pcre_compile $zrb_value ; pcre_match faui && print $match
aui

Neat.  :^)

> I've added some debugging code to test for a bad id passed to
> the regex-match handler.  This doesn't do a heck of a lot at
> the moment, but the case statement was looking a bit lonely
> with just one entry.

Doh.  Yes, of course, not something I normally skip.  Thanks.

> I'll commit this even if there are quibbles, to establish either a stick
> in the ground or a line in the sand.  (I asked what the difference
> was and was told one was horizontal and the other vertical.)

You can tell the time by the stick in the ground, on a sunny day, if you
draw lines in the sand to mark the hours.

Doing so accurately obviously requires control of both the horizontal
and the vertical.  I've reached the outer limit of what I'll say on this
topic.

Re: PATCH: zsh/regex and =~

[Bart Schaefer]

On May 1,  5:11pm, Phil Pennock wrote:
}
} > I'll commit this even if there are quibbles, to establish either a stick
} > in the ground or a line in the sand.  (I asked what the difference
} > was and was told one was horizontal and the other vertical.)
} 
} You can tell the time by the stick in the ground, on a sunny day, if you
} draw lines in the sand to mark the hours.

A stick in the ground marks your point of farthest advance, from which
you are unwilling to retreat.  A line in the sand marks the limit to
which you intend to allow your adversary to advance.  As usual, hard
upright things signify aggression, and soft recumbent things, defense.
Further exploration of these metaphors is left as an exercise for the
reader, lest even more of my zsh mail end up in Gmail's spam folder.


(Yes, after months since the last incident, zsh mail is being junked by
google again.  Sigh.)

Re: PATCH: zsh/regex and =~

[Phil Pennock]

On 2007-04-28 at 00:56 -0700, Phil Pennock wrote:
> Index: Src/parse.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
> retrieving revision 1.64
> diff -p -u -r1.64 parse.c
> --- Src/parse.c	23 Apr 2007 17:24:23 -0000	1.64
> +++ Src/parse.c	28 Apr 2007 07:42:52 -0000
> @@ -2124,6 +2124,12 @@ par_cond_triple(char *a, char *b, char *
>  	ecstr(a);
>  	ecstr(c);
>  	ecadd(ecnpats++);
> +    } else if ((b[0] == Equals || b[0] == '=') &&
> +               (b[1] == '~' || b[1] == Tilde) && ~b[2]) {
> +	ecadd(WCB_COND(COND_REGEX, 0));
> +	ecstr(a);
> +	ecstr(c);
> +	ecadd(ecnpats++);
>      } else if (b[0] == '-') {
>  	if ((t0 = get_cond_num(b + 1)) > -1) {
>  	    ecadd(WCB_COND(t0 + COND_NT, 0));

*blush*

Uhm, the third character of the sequence comprising the =~ operator
needs to be a NUL, which should be tested with a logical negation, not a
bitwise negation.

I'd wonder what I was thinking but apparently I wasn't thinking.

Could someone with commit access please fix that to be !b[2] ?

Thanks,
-Phil :^( who only noticed whilst debugging an updated viewvc install

Re: Calling a zle widget from a function

[Peter Stephenson]

Peter Stephenson wrote:
> (It would be nice if sched provided an array to access scheduled events
> but it doesn't.  That's an easy tweak I may look at.)

Here it is.  Now you can do

integer ind=${zsh_scheduled_events[(I)*my_scheduled_fn*]}
(( ind )) && sched -$ind

Index: Doc/Zsh/mod_sched.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/mod_sched.yo,v
retrieving revision 1.3
diff -u -r1.3 mod_sched.yo
--- Doc/Zsh/mod_sched.yo	11 Sep 2006 20:04:07 -0000	1.3
+++ Doc/Zsh/mod_sched.yo	12 Jun 2007 21:57:25 -0000
@@ -1,7 +1,8 @@
 COMMENT(!MOD!zsh/sched
 A builtin that provides a timed execution facility within the shell.
 !MOD!)
-The tt(zsh/sched) module makes available one builtin command:
+The tt(zsh/sched) module makes available one builtin command and one
+parameter.
 
 startitem()
 findex(sched)
@@ -40,3 +41,19 @@
 output that updates a terminal emulator's title bar.
 )
 enditem()
+
+startitem()
+vindex(zsh_scheduled_events)
+item(zsh_scheduled_events)(
+A readonly array corresponding to the events scheduled by the
+tt(sched) builtin.  The indices of the array correspond to the numbers
+shown when tt(sched) is run with no arguments (provided that the
+tt(KSH_ARRAYS) option is not set).  The value of the
+corresponding element is the same as the text shown to the right
+of the index in the tt(sched) listing.
+
+The tt(sched) builtin should be used for manipulating the events.  Note
+that this will have an immediate effect on the contents of the array,
+so that indices may become invalid.
+)
+enditem()
Index: Doc/Zsh/params.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/params.yo,v
retrieving revision 1.37
diff -u -r1.37 params.yo
--- Doc/Zsh/params.yo	21 May 2007 09:30:25 -0000	1.37
+++ Doc/Zsh/params.yo	12 Jun 2007 21:57:25 -0000
@@ -712,6 +712,10 @@
 Expands to the basename of the command used to invoke this instance
 of zsh.
 )
+item(tt(zsh_scheduled_events))(
+See ifzman(the section `The zsh/sched Module' in zmanref(zshmodules))\
+ifnzman(noderef(The zsh/sched Module)).
+)
 vindex(ZSH_VERSION)
 item(tt(ZSH_VERSION))(
 The version number of this zsh.
Index: Src/Builtins/sched.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Builtins/sched.c,v
retrieving revision 1.10
diff -u -r1.10 sched.c
--- Src/Builtins/sched.c	12 Jun 2007 15:37:20 -0000	1.10
+++ Src/Builtins/sched.c	12 Jun 2007 21:57:25 -0000
@@ -145,6 +145,40 @@
     }
 }
 
+/*
+ * Format event sch.  If sn is zero, allocate string on the heap
+ * and return it; if non-zero, print with that as scheduled event
+ * number.
+ */
+
+static
+char *schedtext(struct schedcmd *sch, int sn)
+{
+    char *str, tbuf[40], *flagstr, *endstr;
+    time_t t;
+    struct tm *tmp;
+
+    t = sch->time;
+    tmp = localtime(&t);
+    ztrftime(tbuf, 20, "%a %b %e %k:%M;%S", tmp);
+    if (sch->flags & SCHEDFLAG_TRASH_ZLE)
+	flagstr = "-o ";
+    else
+	flagstr = "";
+    if (*sch->cmd == '-')
+	endstr = "-- ";
+    else
+	endstr = "";
+    if (sn) {
+	printf("%3d %s %s%s%s\n", sn, tbuf, flagstr, endstr, sch->cmd);
+	return NULL;
+    } else {
+	str = (char *)zhalloc(48 + strlen(sch->cmd));
+	sprintf(str, "%s %s%s%s", tbuf, flagstr, endstr, sch->cmd);
+	return str;
+    }
+}
+
 /**/
 static int
 bin_sched(char *nam, char **argv, UNUSED(Options ops), UNUSED(int func))
@@ -204,22 +238,8 @@
 
     /* given no arguments, display the schedule list */
     if (!*argptr) {
-	char tbuf[40], *flagstr, *endstr;
-
-	for (sn = 1, sch = schedcmds; sch; sch = sch->next, sn++) {
-	    t = sch->time;
-	    tm = localtime(&t);
-	    ztrftime(tbuf, 20, "%a %b %e %k:%M:%S", tm);
-	    if (sch->flags & SCHEDFLAG_TRASH_ZLE)
-		flagstr = "-o ";
-	    else
-		flagstr = "";
-	    if (*sch->cmd == '-')
-		endstr = "-- ";
-	    else
-		endstr = "";
-	    printf("%3d %s %s%s%s\n", sn, tbuf, flagstr, endstr, sch->cmd);
-	}
+	for (sn = 1, sch = schedcmds; sch; sch = sch->next, sn++)
+	    (void)schedtext(sch, 1);
 	return 0;
     } else if (!argptr[1]) {
 	/* other than the two cases above, sched *
@@ -332,14 +352,43 @@
     return 0;
 }
 
+
+/**/
+static char **
+schedgetfn(UNUSED(Param pm))
+{
+    int i;
+    struct schedcmd *sch;
+    char **ret, **aptr;
+
+    for (i = 0, sch = schedcmds; sch; sch = sch->next, i++)
+	;
+
+    aptr = ret = zhalloc(sizeof(char **) * (i+1));
+    for (sch = schedcmds; sch; sch = sch->next, aptr++)
+	*aptr = schedtext(sch, 0);
+    *aptr = NULL;
+
+    return ret;
+}
+
+
 static struct builtin bintab[] = {
     BUILTIN("sched", 0, bin_sched, 0, -1, 0, NULL, NULL),
 };
 
+static const struct gsu_array sched_gsu =
+{ schedgetfn, arrsetfn, stdunsetfn };
+
+static struct paramdef partab[] = {
+    SPECIALPMDEF("zsh_scheduled_events", PM_ARRAY|PM_READONLY,
+		 &sched_gsu, NULL, NULL)
+};
+
 static struct features module_features = {
     bintab, sizeof(bintab)/sizeof(*bintab),
     NULL, 0,
-    NULL, 0,
+    partab, sizeof(partab)/sizeof(*partab),
     NULL, 0,
     0
 };
Index: Src/Builtins/sched.mdd
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Builtins/sched.mdd,v
retrieving revision 1.2
diff -u -r1.2 sched.mdd
--- Src/Builtins/sched.mdd	26 Nov 2000 20:01:03 -0000	1.2
+++ Src/Builtins/sched.mdd	12 Jun 2007 21:57:25 -0000
@@ -3,5 +3,6 @@
 load=yes
 
 autobins="sched"
+autoparams="zsh_scheduled_events"
 
 objects="sched.o"

Re: Calling a zle widget from a function

[Peter Stephenson]

Peter Stephenson wrote:
> +A readonly array corresponding to the events scheduled by the
> +tt(sched) builtin.  The indices of the array correspond to the numbers
> +shown when tt(sched) is run with no arguments (provided that the
> +tt(KSH_ARRAYS) option is not set).  The value of the
> +corresponding element is the same as the text shown to the right
> +of the index in the tt(sched) listing.

Having done this, it just occurred to me it might be more useful to have
a ":" between the date and the command string, and maybe to indicate the
"-o" in some more useful way for processing non-interactively.  That's a
task for tomorrow, I think.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Calling a zle widget from a function

[Peter Stephenson]

On Tue, 12 Jun 2007 23:01:50 +0100
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> Having done this, it just occurred to me it might be more useful to
> have a ":" between the date and the command string, and maybe to
> indicate the "-o" in some more useful way for processing
> non-interactively.  That's a task for tomorrow, I think.

This makes zsh_scheduled_events more useful for its intended purpose, and
updates calendar to use it.

Index: Doc/Zsh/mod_sched.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/mod_sched.yo,v
retrieving revision 1.4
diff -u -r1.4 mod_sched.yo
--- Doc/Zsh/mod_sched.yo	12 Jun 2007 22:01:38 -0000	1.4
+++ Doc/Zsh/mod_sched.yo	14 Jun 2007 13:09:51 -0000
@@ -48,9 +48,13 @@
 A readonly array corresponding to the events scheduled by the
 tt(sched) builtin.  The indices of the array correspond to the numbers
 shown when tt(sched) is run with no arguments (provided that the
-tt(KSH_ARRAYS) option is not set).  The value of the
-corresponding element is the same as the text shown to the right
-of the index in the tt(sched) listing.
+tt(KSH_ARRAYS) option is not set).  The value of the array
+consists of the scheduled time in seconds since the epoch
+(see ifnzman(The zsh/datetime Module)\
+ifzman(the section `The zsh/datetime Module') for facilities for
+using this number), followed by a colon, followed by any options
+(which may be empty but will be preceeded by a `tt(-)' otherwise),
+followed by a colon, followed by the command to be executed.
 
 The tt(sched) builtin should be used for manipulating the events.  Note
 that this will have an immediate effect on the contents of the array,
Index: Functions/Calendar/calendar
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Calendar/calendar,v
retrieving revision 1.3
diff -u -r1.3 calendar
--- Functions/Calendar/calendar	26 Mar 2007 14:33:33 -0000	1.3
+++ Functions/Calendar/calendar	14 Jun 2007 13:09:51 -0000
@@ -352,13 +352,8 @@
   if [[ -n $sched ]]; then
     if [[ $next -ge 0 ]]; then
       # Remove any existing calendar scheduling.
-      # Luckily sched doesn't delete its schedule in a subshell.
-      sched | while read line; do
-	if [[ $line = (#b)[[:space:]]#(<->)[[:space:]]##*[[:space:]]'calendar -s'* ]]; then
-	  # End of pipeline run in current shell, so delete directly.
-	  sched -1 $match[1]
-	fi
-      done
+      i=${"${(@)zsh_scheduled_events#*:*:}"[(I)calendar -s*]}
+      (( i )) && sched -$i
       $sched $next calendar "${calopts[@]}" $next $next
     else
       $showprog $start $stop \
Index: Src/Builtins/sched.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Builtins/sched.c,v
retrieving revision 1.11
diff -u -r1.11 sched.c
--- Src/Builtins/sched.c	12 Jun 2007 22:01:39 -0000	1.11
+++ Src/Builtins/sched.c	14 Jun 2007 13:09:55 -0000
@@ -145,40 +145,6 @@
     }
 }
 
-/*
- * Format event sch.  If sn is zero, allocate string on the heap
- * and return it; if non-zero, print with that as scheduled event
- * number.
- */
-
-static
-char *schedtext(struct schedcmd *sch, int sn)
-{
-    char *str, tbuf[40], *flagstr, *endstr;
-    time_t t;
-    struct tm *tmp;
-
-    t = sch->time;
-    tmp = localtime(&t);
-    ztrftime(tbuf, 20, "%a %b %e %k:%M;%S", tmp);
-    if (sch->flags & SCHEDFLAG_TRASH_ZLE)
-	flagstr = "-o ";
-    else
-	flagstr = "";
-    if (*sch->cmd == '-')
-	endstr = "-- ";
-    else
-	endstr = "";
-    if (sn) {
-	printf("%3d %s %s%s%s\n", sn, tbuf, flagstr, endstr, sch->cmd);
-	return NULL;
-    } else {
-	str = (char *)zhalloc(48 + strlen(sch->cmd));
-	sprintf(str, "%s %s%s%s", tbuf, flagstr, endstr, sch->cmd);
-	return str;
-    }
-}
-
 /**/
 static int
 bin_sched(char *nam, char **argv, UNUSED(Options ops), UNUSED(int func))
@@ -238,8 +204,24 @@
 
     /* given no arguments, display the schedule list */
     if (!*argptr) {
-	for (sn = 1, sch = schedcmds; sch; sch = sch->next, sn++)
-	    (void)schedtext(sch, 1);
+	for (sn = 1, sch = schedcmds; sch; sch = sch->next, sn++) {
+	    char tbuf[40], *flagstr, *endstr;
+	    time_t t;
+	    struct tm *tmp;
+
+	    t = sch->time;
+	    tmp = localtime(&t);
+	    ztrftime(tbuf, 20, "%a %b %e %k:%M:%S", tmp);
+	    if (sch->flags & SCHEDFLAG_TRASH_ZLE)
+		flagstr = "-o ";
+	    else
+		flagstr = "";
+	    if (*sch->cmd == '-')
+		endstr = "-- ";
+	    else
+		endstr = "";
+	    printf("%3d %s %s%s%s\n", sn, tbuf, flagstr, endstr, sch->cmd);
+	}
 	return 0;
     } else if (!argptr[1]) {
 	/* other than the two cases above, sched *
@@ -365,8 +347,19 @@
 	;
 
     aptr = ret = zhalloc(sizeof(char **) * (i+1));
-    for (sch = schedcmds; sch; sch = sch->next, aptr++)
-	*aptr = schedtext(sch, 0);
+    for (sch = schedcmds; sch; sch = sch->next, aptr++) {
+	char tbuf[40], *flagstr;
+	time_t t;
+
+	t = sch->time;
+	sprintf(tbuf, "%ld", t);
+	if (sch->flags & SCHEDFLAG_TRASH_ZLE)
+	    flagstr = "-o";
+	else
+	    flagstr = "";
+	*aptr = (char *)zhalloc(5 + strlen(tbuf) + strlen(sch->cmd));
+	sprintf(*aptr, "%s:%s:%s", tbuf, flagstr, sch->cmd);
+    }
     *aptr = NULL;
 
     return ret;


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

Re: ZSH (CVS) configure problem?

[Peter Stephenson]

Clint Adams wrote:
> On Thu, Oct 11, 2007 at 12:26:45PM +0100, Peter Stephenson wrote:
> > the week).  Clint thinks in that case we need -I/usr/include/ncursesw,
> > so it may not be entirely trivial.
> 
> Clint probably was mistaken, since the headers in /usr/include/ncursesw
> include <ncurses/$otherheader>.

(Moved to zsh-workers).

On my system the entire contents of /usr/include/ncurses (which it seems
to be well-established we don't need to add to the search path) are
simply links into /usr/include/ncursesw.

Shall we try this?  I'd be interested to see how it works out.
If we end up not compiling in multibyte support, should be replacing
ncursesw with ncurses, or is that unnecessary?

Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.70
diff -u -r1.70 configure.ac
--- configure.ac	8 Oct 2007 02:04:01 -0000	1.70
+++ configure.ac	11 Oct 2007 19:14:52 -0000
@@ -648,12 +648,12 @@
   termcap_curses_order="$withval"
   AC_SEARCH_LIBS(tigetstr, [$termcap_curses_order])
 else
-  termcap_curses_order="tinfo termcap ncurses curses"
+  termcap_curses_order="ncursesw tinfo termcap ncurses curses"
 fi],
 [case "$host_os" in
   hpux10.*|hpux11.*|solaris*)
       termcap_curses_order="Hcurses ncurses curses termcap" ;;
-  *)             termcap_curses_order="tinfo termcap ncurses curses" ;;
+  *)             termcap_curses_order="ncursesw tinfo termcap ncurses curses" ;;
 esac])dnl
 
 AH_TEMPLATE([HAVE_BOOLCODES],

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: ZSH (CVS) configure problem?

[Clint Adams]

On Thu, Oct 11, 2007 at 08:18:03PM +0100, Peter Stephenson wrote:
> On my system the entire contents of /usr/include/ncurses (which it seems
> to be well-established we don't need to add to the search path) are
> simply links into /usr/include/ncursesw.

This is the current state of Debian:

The left column is ncurses compiled without --enable-widec , and the
right is ncursesw (--enable-widec).

% paste <(dpkg -L libncurses5-dev | grep 'include.*h') <(dpkg -L libncursesw5-dev | grep 'include.*h')
/usr/include/unctrl.h   /usr/include/ncursesw/unctrl.h
/usr/include/termcap.h  /usr/include/ncursesw/termcap.h
/usr/include/cursesp.h  /usr/include/ncursesw/cursesp.h
/usr/include/curses.h   /usr/include/ncursesw/curses.h
/usr/include/tic.h      /usr/include/ncursesw/tic.h
/usr/include/eti.h      /usr/include/ncursesw/eti.h
/usr/include/nc_tparm.h /usr/include/ncursesw/nc_tparm.h
/usr/include/cursesw.h  /usr/include/ncursesw/cursesw.h
/usr/include/term_entry.h       /usr/include/ncursesw/term_entry.h
/usr/include/form.h     /usr/include/ncursesw/form.h
/usr/include/cursesm.h  /usr/include/ncursesw/cursesm.h
/usr/include/menu.h     /usr/include/ncursesw/menu.h
/usr/include/cursesapp.h        /usr/include/ncursesw/cursesapp.h
/usr/include/cursesf.h  /usr/include/ncursesw/cursesf.h
/usr/include/cursslk.h  /usr/include/ncursesw/cursslk.h
/usr/include/panel.h    /usr/include/ncursesw/panel.h
/usr/include/etip.h     /usr/include/ncursesw/etip.h
/usr/include/term.h     /usr/include/ncursesw/term.h
/usr/include/ncurses_dll.h      /usr/include/ncursesw/ncurses_dll.h
/usr/include/ncurses.h  /usr/include/ncursesw/ncurses.h

I've heard some vague promises that the ncursesw packages will disappear
and that we'll have only the --enable-widec headers and libraries in
libncurses$num-dev.  If there are no ABI issues with this, I'd prefer it
happens as soon as possible.

> Shall we try this?  I'd be interested to see how it works out.
> If we end up not compiling in multibyte support, should be replacing
> ncursesw with ncurses, or is that unnecessary?

Given my current understanding, if both ncurses{,w} -dev packages
installed, this will link against the wide library but use the non-wide
macros.  I assume this is harmless and equivalent to just linking
against the non-wide library.

Re: ZSH (CVS) configure problem?

[Peter Stephenson]

On Thu, 11 Oct 2007 16:23:08 -0400
Clint Adams <schizo@debian.org> wrote:
> > Shall we try this?  I'd be interested to see how it works out.
> > If we end up not compiling in multibyte support, should be replacing
> > ncursesw with ncurses, or is that unnecessary?
> 
> Given my current understanding, if both ncurses{,w} -dev packages
> installed, this will link against the wide library but use the non-wide
> macros.  I assume this is harmless and equivalent to just linking
> against the non-wide library.

Maybe I'll commit the following.

Index: INSTALL
===================================================================
RCS file: /cvsroot/zsh/zsh/INSTALL,v
retrieving revision 1.31
diff -u -r1.31 INSTALL
--- INSTALL	2 Oct 2007 10:22:26 -0000	1.31
+++ INSTALL	12 Oct 2007 09:08:42 -0000
@@ -312,12 +312,21 @@
 termcap, which is now largely outmoded, and curses, which supersedes
 termcap and typically contains the same features as well as others.
 configure will search for an appropriate library; the default search order
-is "tinfo termcap ncurses curses" except on HP-UX and Solaris where it is
-"Hcurses ncurses curses termcap".  Note that even though termcap is usually
-searched first zsh tries to make features from curses available and if the
-curses library contains both curses and termcap features, as is normal,
-the curses variant is used.  ncurses is a newer version of curses
-and tinfo is related to it.
+is "ncursesw tinfo termcap ncurses curses" except on HP-UX and Solaris
+where it is "Hcurses ncursesw ncurses curses termcap".  Note that even
+though termcap is searched before traditional forms of curses zsh tries to
+make features from curses available and if the curses library contains both
+curses and termcap features, as is normal, the curses variant is used.
+ncurses is a newer version of curses and tinfo is related to it.
+
+The library ncursesw is a variant of ncurses that supports wide characters.
+zsh attempts to use this to provide functions needed by the zsh/curses
+module; the main shell does not require the additional functions.  As the
+integration of wide character support into ncurses is continuing, it is
+possible that on some systems attempting to use ncursesw may cause problems
+during building.  If so, please report this to the developers at
+zsh-workers@sunsite.dk and attempt to recompile with --with-term-lib="tinfo
+termcap ncurses curses" (see below).
 
 On some systems a suitable development package with a name such as
 curses-devel or ncurses-devel needs to be installed before zsh can
@@ -328,7 +337,7 @@
 You can tell configure which libraries to search by passing an
 argument via --with-term-lib.  This takes a space-separated list
 of libraries to try as its argument, so the default is equivalent to
---with-term-lib="tinfo termcap ncurses curses".  It replaces the
+--with-term-lib="ncursesw tinfo termcap ncurses curses".  It replaces the
 old option --with-curses-terminfo, which altered the search order but
 didn't allow an explicit search list to be passed.
 
Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.70
diff -u -r1.70 configure.ac
--- configure.ac	8 Oct 2007 02:04:01 -0000	1.70
+++ configure.ac	12 Oct 2007 09:08:44 -0000
@@ -648,12 +648,12 @@
   termcap_curses_order="$withval"
   AC_SEARCH_LIBS(tigetstr, [$termcap_curses_order])
 else
-  termcap_curses_order="tinfo termcap ncurses curses"
+  termcap_curses_order="ncursesw tinfo termcap ncurses curses"
 fi],
 [case "$host_os" in
   hpux10.*|hpux11.*|solaris*)
-      termcap_curses_order="Hcurses ncurses curses termcap" ;;
-  *)             termcap_curses_order="tinfo termcap ncurses curses" ;;
+      termcap_curses_order="Hcurses ncursesw ncurses curses termcap" ;;
+  *)             termcap_curses_order="ncursesw tinfo termcap ncurses curses" ;;
 esac])dnl
 
 AH_TEMPLATE([HAVE_BOOLCODES],

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: ZSH (CVS) configure problem?

[Peter Stephenson]

On Fri, 12 Oct 2007 10:09:25 +0100
Peter Stephenson <pws@csr.com> wrote:
> +The library ncursesw is a variant of ncurses that supports wide characters.
> +zsh attempts to use this to provide functions needed by the zsh/curses
> +module; the main shell does not require the additional functions.

Sorry, it's me again... given Helmut's problems on Gentoo reported a couple
of days ago, I've made the final statement there rather more weaselly and
committed the result.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: precmd, preexec, and supplied prompt themes

[Clint Adams]

[moving to -workers]

>      Check out the zsh man page, section SPECIAL FUNCTIONS, or just search
>      for 'hook function'.  The long and short of it is that the prompt
>      themes should be using
>      
>      typeset -a precmd_functions
>      precmd_functions += my_prompt_precmd

Since there is no unregistration hook, how would you avoid the problem
of precmd_functions containing (my_prompt_precmd her_prompt_precmd
his_prompt_precmd my_prompt_precmd) when switching back and forth?

The clobbering below should now appear more explicit.

Index: Functions/Prompts/prompt_clint_setup
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Prompts/prompt_clint_setup,v
retrieving revision 1.6
diff -u -r1.6 prompt_clint_setup
--- Functions/Prompts/prompt_clint_setup	7 Sep 2006 18:09:44 -0000	1.6
+++ Functions/Prompts/prompt_clint_setup	13 Oct 2007 13:26:07 -0000
@@ -36,7 +36,7 @@
   [[ -n "$WINDOW" ]] && p_win="$pc['\(']%{$fg_bold[$pcc[4]]%}$WINDOW$pc['\)']"
 
   p_userpwd="$pc['<']%{$fg_no_bold[$pcc[3]]%}%n@%m$p_win%{$fg_bold[$pcc[5]]%}:%{$fg_no_bold[$pcc[4]]%}%~$pc['>']"
-  [[ -f /proc/apm ]] && p_apm="%(2v.-%2v-.)"
+  p_git="%(2v.-%U%2v%u-.)"
 
   p_shlvlhist="%{$reset_color%}zsh%(2L./$SHLVL.) %B%h%b "
   p_rc="%(?..[%?%1v] )"
@@ -46,34 +46,28 @@
 
   prompt="$p_date$p_tty$p_plat$p_ver
 $p_userpwd
-$p_shlvlhist$p_rc$p_apm$p_end"
+$p_shlvlhist$p_rc$p_git$p_end"
   PS2='%(4_.\.)%3_> %E'
 
-  [[ -f /proc/apm ]] &&
-     precmd () { prompt_clint_precmd ; prompt_clint_apm_precmd } ||
-     precmd () { prompt_clint_precmd }
+  precmd () { prompt_clint_precmd }
   preexec () { }
 }
 
 prompt_clint_precmd () {
   setopt noxtrace noksharrays localoptions
   local exitstatus=$?
+  local git_dir git_ref
 
+  psvar=()
   [[ $exitstatus -ge 128 ]] && psvar[1]=" $signals[$exitstatus-127]" ||
 	psvar[1]=""
 
   [[ -o interactive ]] && jobs -l
 
-}
-
-prompt_clint_apm_precmd () {
-  setopt noxtrace noksharrays localoptions
-  local bat ac
-
-  bat=${${="$(</proc/apm)"}[7]/%/%%}
-  [[ ${${="$(</proc/apm)"}[4]} == "0x01" ]] && ac="AC+"
+  git_dir=$(git-rev-parse --git-dir 2> /dev/null) || return
+  git_ref=$(git-symbolic-ref HEAD 2> /dev/null) || git_ref="(no branch)"
+  psvar[2]=${ref#refs/heads/}
 
-  [[ $bat == ("100%"|"-1%") ]] && psvar[2]=() || psvar[2]="$ac$bat"
 }
 
 prompt_clint_setup "$@"

Re: precmd, preexec, and supplied prompt themes

[Bart Schaefer]

On Oct 13,  9:30am, Clint Adams wrote:
}
} >      typeset -a precmd_functions
} >      precmd_functions += my_prompt_precmd
} 
} Since there is no unregistration hook, how would you avoid the problem
} of precmd_functions containing (my_prompt_precmd her_prompt_precmd
} his_prompt_precmd my_prompt_precmd) when switching back and forth?

Well, there sort of is an unregistration hook:  prompt_off_setup
(meant to be invoked as "prompt off").

However, all it does is erase the precmd and preexec functions entirely,
which is why I added an "off" switch to my theme ("prompt bart off").

*IF* we updated the themes to be clever about precmd_functions et al.,
then the "prompt" function should be updated to call prompt_off_setup
before installing a new theme.

However, that leaves the conundrum of whether/how to unregister PS1.
"prompt bart", for example, piggybacks itself on top of whatever is
already in PS1, and "prompt bart off" extricates itself to leave PS1
mostly as it was before.

Right now "prompt off" stomps PS1 and PS2 back to the "zsh -f" default.

So some discussion of how this ought to be redesigned would be useful
before someone jumps in to do it.  It's always seemed a bit clunky to
me to do build up function names by sticking the theme name in the
middle, but that's at nit-pick.  Maybe all we need is another one of
those functions, prompt_ZZZ_off (or pick another word).

-- 

Re: precmd, preexec, and supplied prompt themes

[Peter Stephenson]

Clint Adams wrote:
> >      typeset -a precmd_functions

(Would probably need to be "typeset -ga ...")

> >      precmd_functions += my_prompt_precmd
> 
> Since there is no unregistration hook, how would you avoid the problem
> of precmd_functions containing (my_prompt_precmd her_prompt_precmd
> his_prompt_precmd my_prompt_precmd) when switching back and forth?

Presumably either you would add an unregistration hook when you gave the
system the clean-up it needs, or more simply you would remove all
existing entries of the form *_prompt_precmd from precmd_functions when
you added a new one and enforce that format.  Without some form of
tidy-up the question is somewhat moot.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: precmd, preexec, and supplied prompt themes

[Peter Stephenson]

Bart Schaefer wrote:
> So some discussion of how this ought to be redesigned would be useful
> before someone jumps in to do it.  It's always seemed a bit clunky to
> me to do build up function names by sticking the theme name in the
> middle, but that's at nit-pick.  Maybe all we need is another one of
> those functions, prompt_ZZZ_off (or pick another word).

We could do it with one level of indirection...


Generically:

if [[ -z ${precmd_functions[(R)precmd_prompt]} ]]; then
  precmd_functions+=(precmd_prompt)
fi
precmd_prompt() {
   if (( ${+prompt_functions[precmd]} )); then
     $prompt_functions[precmd]
   fi
}


For each prompt:

typeset -gA prompt_function_hooks
prompt_functions=(precmd my_fantastic_precmd_function
                  setup my_absolutely_superb_setup_function
                  teardown my_truly_wonderful_teardown_function)


and so on.  As long as the functions in question are defined at the same
time as the prompt_functions associative array there isn't a
namespace clash.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: precmd, preexec, and supplied prompt themes

[Bart Schaefer]

On Oct 13,  8:10pm, Peter Stephenson wrote:
} Subject: Re: precmd, preexec, and supplied prompt themes
}
} Bart Schaefer wrote:
} > It's always seemed a bit clunky to me to do build up function names
} > by sticking the theme name in the middle, but that's at nit-pick.
} > Maybe all we need is another one of those functions, prompt_ZZZ_off
} > (or pick another word).
} 
} We could do it with one level of indirection...

That deals with the nit-pick, but doesn't really answer the question of
what to do with PS1.

Maybe promptinit should store the original values of all the {R,}PS*
variables somewhere, so they can be put back on teardown.

Re: precmd, preexec, and supplied prompt themes

[Oliver Kiddle]

On 13 Oct, Bart wrote:
> --- promptinit	1 Oct 2006 02:38:52 -0000	1.3
>  
> +  emulate -L zsh

This has broken the prompt_opts	feature of promptinit whereby prompts
could set the promptcr, promptsubst and promptpercent options if they
needed them. This means that with the "oliver" prompt, I no longer get
promptsubst set and that I end up seeing the COLUMNS bit unexpanded.

Oliver

Re: precmd, preexec, and supplied prompt themes

[Bart Schaefer]

On Oct 31, 11:07am, Oliver Kiddle wrote:
} Subject: Re: precmd, preexec, and supplied prompt themes
}
} On 13 Oct, Bart wrote:
} > --- promptinit	1 Oct 2006 02:38:52 -0000	1.3
} >  
} > +  emulate -L zsh
} 
} This has broken the prompt_opts	feature of promptinit

I think that's a bug in the way those options are initialized, then,
or in "emulate".

Even with -R, options "describing the interactive environment" are
not supposed to be reset, and without -L "only those options likely
to cause portability problems in scripts and functions are altered."

Would someone explain how promptcr, promptsubst and promptpercent
"cause portability problems in scripts and functions"?

Re: precmd, preexec, and supplied prompt themes

[Bart Schaefer]

On Oct 31,  7:49am, Bart Schaefer wrote:
}
} Even with -R, options "describing the interactive environment" are
} not supposed to be reset, and without -L "only those options likely

That should say "without -R" not -L.

In any case it appears to be OK to remove that one "emulate".

--- ../zsh-forge/current/Functions/Prompts/promptinit   2007-10-14
09:32:27.000000000 -0700
+++ Functions/Prompts/promptinit        2007-10-31 08:07:48.000000000 -0700
@@ -166,13 +166,12 @@
 }
 
 prompt () {
-  emulate -L zsh
   local prompt_opts
   
   set_prompt "$@"
  
   (( $#prompt_opts )) &&
       setopt noprompt{bang,cr,percent,subst} prompt${^prompt_opts[@]}
 
   true
 }

delete-whole-word-match fails on words starting with -, patch

[Mikael Magnusson]

-- isn't documented in the manual for the zle builtin, but seems to do
the trick anyway.

diff --git a/Functions/Zle/delete-whole-word-match
b/Functions/Zle/delete-whole-word-match
index 978b95e..51314cf 100644
--- a/Functions/Zle/delete-whole-word-match
+++ b/Functions/Zle/delete-whole-word-match
@@ -49,7 +49,7 @@ if [[ $WIDGET = *kill* ]]; then
   if [[ $LASTWIDGET = *kill* ]]; then
     CUTBUFFER="$CUTBUFFER$word"
   else
-    zle copy-region-as-kill "$word"
+    zle copy-region-as-kill -- "$word"
   fi
 fi
 BUFFER="${BUFFER[1,pos1]}${BUFFER[pos2,-1]}"

-- 
Mikael Magnusson

Re: delete-whole-word-match fails on words starting with -, patch

[Peter Stephenson]

"Mikael Magnusson" wrote:
> -- isn't documented in the manual for the zle builtin, but seems to do
> the trick anyway.

The function's been rewritten to use lower level facilities, so that
shouldn't occur.  (It looks rather strange anyway but I haven't
investigated yet; I will if I get a moment but moments are now being
scheduled for 2012...)  Could you try it from CVS and see if this and
the other problems (which I suspect are from a different source) are
still there?

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: delete-whole-word-match fails on words starting with -, patch

[Peter Stephenson]

Peter Stephenson wrote:
> "Mikael Magnusson" wrote:
> > -- isn't documented in the manual for the zle builtin, but seems to do
> > the trick anyway.
> 
> The function's been rewritten to use lower level facilities, so that
> shouldn't occur.

I'm talking nonsense:  no, it hasn't been rewritten and I must have been
looking at the wrong thing.  However, I still don't understand
how you're getting that problem.  Could you send a trace with "setopt
xtrace" both in there and in match-words-by-style (I presume it
depends on what word style is in effect since I can't se it)?

Thanks.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: delete-whole-word-match fails on words starting with -, patch

[Mikael Magnusson]

On 09/11/2007, Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> Peter Stephenson wrote:
> > "Mikael Magnusson" wrote:
> > > -- isn't documented in the manual for the zle builtin, but seems to do
> > > the trick anyway.
> >
> > The function's been rewritten to use lower level facilities, so that
> > shouldn't occur.
>
> I'm talking nonsense:  no, it hasn't been rewritten and I must have been
> looking at the wrong thing.  However, I still don't understand
> how you're getting that problem.  Could you send a trace with "setopt
> xtrace" both in there and in match-words-by-style (I presume it
> depends on what word style is in effect since I can't se it)?

By that, if you mean the problem with words starting with -, i
realized it only happens if you call it as kill-whole-word-match.

% -te<ctrl-delete>st bar
delete-whole-word-match:zle:52: unknown option: test
% bar

with xtrace set and same sequence,

+delete-whole-word-match:13> setopt extendedglob
+delete-whole-word-match:15> local 'curcontext=:zle:kill-whole-word-match'
+delete-whole-word-match:16> local -a matched_words
+delete-whole-word-match:18> integer pos1 pos2
+delete-whole-word-match:20> autoload -U match-words-by-style
+delete-whole-word-match:21> match-words-by-style
+match-words-by-style:67> emulate -L zsh
+match-words-by-style:68> setopt extendedglob
+match-words-by-style:70> local wordstyle spacepat wordpat1 wordpat2
opt charskip wordchars wordclass
+match-words-by-style:71> local match mbegin mend pat1 pat2 word1
word2 ws1 ws2 ws3 skip
+match-words-by-style:72> local nwords MATCH MBEGIN MEND
+match-words-by-style:74> local 'curcontext=:zle:kill-whole-word-match'
+match-words-by-style:76> autoload -U match-word-context
+match-words-by-style:77> match-word-context
+match-word-context:5> emulate -L zsh
+match-word-context:6> setopt extendedglob
+match-word-context:8> local -a worcon bufwords
+match-word-context:9> local pat tag lastword word
+match-word-context:10> integer iword
+match-word-context:12> zstyle -a :zle:kill-whole-word-match word-context worcon
+match-word-context:12> return 0
+match-words-by-style:79> getopts w:s:c:C: opt
+match-words-by-style:103> [[ -z '' ]]
+match-words-by-style:103> zstyle -s :zle:kill-whole-word-match
word-style wordstyle
+match-words-by-style:104> [[ -z '' ]]
+match-words-by-style:104> zstyle -s :zle:kill-whole-word-match skip-chars skip
+match-words-by-style:105> [[ -z '' ]]
+match-words-by-style:105> skip=0
+match-words-by-style:107> case shell (shell)
+match-words-by-style:108> local bufwords
+match-words-by-style:110> bufwords=( -t )
+match-words-by-style:111> nwords=1
+match-words-by-style:112> wordpat1=-t
+match-words-by-style:116> bufwords=( est bar )
+match-words-by-style:117> wordpat2=est
+match-words-by-style:118> spacepat='[[:space:]]#'
+match-words-by-style:123> bufwords=( -test bar )
+match-words-by-style:124> ((  5 > 2  ))
+match-words-by-style:127> eval 'pat1=${LBUFFER%%(#b)(-t)([[:space:]]#)}'
+(eval):1> pat1=''
+match-words-by-style:129> wordpat1=-t
+match-words-by-style:130> wordpat2='est '
+match-words-by-style:132> wordpat1=-t
+match-words-by-style:133> wordpat2='est\ '
+match-words-by-style:187> match=( )
+match-words-by-style:188> eval 'pat1=${LBUFFER%%(#b)(-t)([[:space:]]#)}'
+(eval):1> pat1=''
+match-words-by-style:189> word1=-t
+match-words-by-style:190> ws1=''
+match-words-by-style:192> match=( )
+match-words-by-style:193> charskip=''
+match-words-by-style:196> eval
'pat2=${RBUFFER##(#b)([[:space:]]#)(est\ )([[:space:]]#)}'
+(eval):1> pat2=bar
+match-words-by-style:199> ws2=''
+match-words-by-style:200> word2='est '
+match-words-by-style:201> ws3=''
+match-words-by-style:203> matched_words=( '' -t '' '' 'est ' '' bar )
+delete-whole-word-match:23> [[ -n '' ]]
+delete-whole-word-match:29> pos1=0
+delete-whole-word-match:32> [[ -n '' ]]
+delete-whole-word-match:39> ((  pos2 = CURSOR + 4 + 1  ))
+delete-whole-word-match:44> ((  CURSOR = pos1  ))
+delete-whole-word-match:48> [[ kill-whole-word-match == '*kill*' ]]
+delete-whole-word-match:49> local 'word=-test '
+delete-whole-word-match:50> [[ backward-char == '*kill*' ]]
+delete-whole-word-match:53> zle copy-region-as-kill '-test '

delete-whole-word-match:zle:53: unknown option: test
+delete-whole-word-match:56> BUFFER=bar

Adding the -- fixes it as it instead runs
+delete-whole-word-match:53> zle copy-region-as-kill -- -test

In both cases, the end result is the same, but the error message
messes up the display a bit in the first.

-- 
Mikael Magnusson

Re: delete-whole-word-match fails on words starting with -, patch

[Peter Stephenson]

On Fri, 9 Nov 2007 23:38:55 +0100
"Mikael Magnusson" <mikachu@gmail.com> wrote:
> By that, if you mean the problem with words starting with -, i
> realized it only happens if you call it as kill-whole-word-match.

I should have realised that, but didn't, because I was looking at the
wrong test for *kill*  (the $LASTWIDGET one, which is in the opposite
sense).

> +delete-whole-word-match:53> zle copy-region-as-kill '-test '
> 
> delete-whole-word-match:zle:53: unknown option: test

The penny has finally dropped:  I was also trying it out with words
beginning "--" in a GNUish fashion, which already look like an end of
option list marker, so I didn't see this.

Yes, your patch is entirely correct and the option handling is for
setting the numeric prefix and the widget name locally; these follow the
widget name to distinguish them from options to zle instead of to the
widget call.  Here's a more complete version.  I think the various
places where "$@" gets passed to widgets in ZLE functions are intended
to include options passed, so need to be left alone.

I haven't looked at the other problems yet.

Index: Doc/Zsh/zle.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/zle.yo,v
retrieving revision 1.56
diff -u -r1.56 zle.yo
--- Doc/Zsh/zle.yo	19 Apr 2007 14:16:23 -0000	1.56
+++ Doc/Zsh/zle.yo	10 Nov 2007 14:19:19 -0000
@@ -554,7 +554,9 @@
 active.  With the option tt(-w), tt(WIDGET) and related parameters are set
 to reflect the widget being executed by the tt(zle) call.
 
-Any further arguments will be passed to the widget.  If it is a shell
+Any further arguments will be passed to the widget; note that as
+standard argument handling is performed, any general argument list
+should be preceded by tt(-)tt(-).  If it is a shell
 function, these are passed down as positional parameters; for builtin
 widgets it is up to the widget in question what it does with them.
 Currently arguments are only handled by the incremental-search commands,
Index: Functions/Zle/backward-kill-word-match
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/backward-kill-word-match,v
retrieving revision 1.2
diff -u -r1.2 backward-kill-word-match
--- Functions/Zle/backward-kill-word-match	19 Apr 2007 14:16:23 -0000	1.2
+++ Functions/Zle/backward-kill-word-match	10 Nov 2007 14:19:19 -0000
@@ -23,7 +23,7 @@
     if [[ -n $done || $LASTWIDGET = *kill* ]]; then
       CUTBUFFER="$word$CUTBUFFER"
     else
-      zle copy-region-as-kill "$word"
+      zle copy-region-as-kill -- "$word"
     fi
     LBUFFER=$matched_words[1]
   else
Index: Functions/Zle/delete-whole-word-match
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/delete-whole-word-match,v
retrieving revision 1.3
diff -u -r1.3 delete-whole-word-match
--- Functions/Zle/delete-whole-word-match	19 Apr 2007 14:16:23 -0000	1.3
+++ Functions/Zle/delete-whole-word-match	10 Nov 2007 14:19:19 -0000
@@ -9,7 +9,7 @@
 # saved for future yanking in the normal way.
 
 emulate -L zsh
-setopt extendedglob
+setopt extendedglob xtrace
 
 local curcontext=:zle:$WIDGET
 local -a matched_words
@@ -49,7 +49,7 @@
   if [[ $LASTWIDGET = *kill* ]]; then
     CUTBUFFER="$CUTBUFFER$word"
   else
-    zle copy-region-as-kill "$word"
+    zle copy-region-as-kill -- "$word"
   fi
 fi
 BUFFER="${BUFFER[1,pos1]}${BUFFER[pos2,-1]}"
Index: Functions/Zle/kill-word-match
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/kill-word-match,v
retrieving revision 1.3
diff -u -r1.3 kill-word-match
--- Functions/Zle/kill-word-match	19 Apr 2007 14:16:23 -0000	1.3
+++ Functions/Zle/kill-word-match	10 Nov 2007 14:19:19 -0000
@@ -22,7 +22,7 @@
     if [[ -n $done || $LASTWIDGET = *kill* ]]; then
       CUTBUFFER="$CUTBUFFER$word"
     else
-      zle copy-region-as-kill $word
+      zle copy-region-as-kill -- $word
     fi
     RBUFFER=${(j..)matched_words[6,7]}
   else


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug#451382: i18n is NOT so easy!

[Clint Adams]

On Wed, Dec 05, 2007 at 09:08:25PM +0100, Dr. Markus Waldeck wrote:
> I am waiting for a useful answer which is necessary for further contributions!

I don't think I have one.  Nothing in zsh is particularly suited to
gettext or translations.

Re: Bug#451382: i18n is NOT so easy!

[Ismail Dönmez]

Thursday 06 December 2007 17:54:36 Clint Adams yazmıştı:
> On Wed, Dec 05, 2007 at 09:08:25PM +0100, Dr. Markus Waldeck wrote:
> > I am waiting for a useful answer which is necessary for further
> > contributions!
>
> I don't think I have one.  Nothing in zsh is particularly suited to
> gettext or translations.

nothing as in it wouldn't be useful? Imho it would be useful for warning 
messages like "do you want to delete all files" etc.

Regards,
ismail

-- 
Never learn by your mistakes, if you do you may never dare to try again.

Re: Bug#451382: i18n is NOT so easy!

[Clint Adams]

On Thu, Dec 06, 2007 at 06:08:55PM +0200, Ismail Dönmez wrote:
> nothing as in it wouldn't be useful? Imho it would be useful for warning 
> messages like "do you want to delete all files" etc.

Certainly it would be useful.

Re: Bug#451382: i18n is NOT so easy!

[Clint Adams]

On Thu, Dec 06, 2007 at 05:56:12PM +0100, Dr. Markus Waldeck wrote:
> Please correct _cut as mentioned in my mail from Fri, 16 Nov 2007 10:16:44 +0100.

Okay, but it is certainly ugly.

Index: Completion/Unix/Command/_cut
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Command/_cut,v
retrieving revision 1.2
diff -u -r1.2 _cut
--- Completion/Unix/Command/_cut	31 Oct 2007 00:35:37 -0000	1.2
+++ Completion/Unix/Command/_cut	6 Dec 2007 19:04:17 -0000
@@ -11,7 +11,7 @@
              delimiter       "Delimiter anstelle von Tabulator als Trenner benutzen"
              fields          "nur diese Felder und alle Zeilen OHNE Trennzeichen ausgeben"
              n               "(ignoriert)"
-             complement      "das Komplement der Menge der gewählten Bytes, Zeichen oder Felder bilden"
+             complement      "das Komplement der Menge der gew$(print \\u00e4)hlten Bytes, Zeichen oder Felder bilden"
              only-delimited  "keine Zeilen ausgeben, die keinen Trenner enthalten"
              output-delimiter "Zeichenkette als Ausgabetrennzeichen benutzen"
              help            "diese Hilfe anzeigen und beenden"

Re: Bug#451382: i18n is NOT so easy!

[Oliver Kiddle]

Clint Adams wrote:
> On Thu, Dec 06, 2007 at 05:56:12PM +0100, Dr. Markus Waldeck wrote:
> > Please correct _cut as mentioned in my mail from Fri, 16 Nov 2007 10:16:44 +0100.
> 
> Okay, but it is certainly ugly.

You can use $'\u00e4' which wouldn't be quite so ugly.

Oliver

Re: Bug#451382: i18n is NOT so easy!

[Peter Stephenson]

On Thu, 6 Dec 2007 11:10:22 -0500
Clint Adams <clint@zsh.org> wrote:
> On Thu, Dec 06, 2007 at 06:08:55PM +0200, Ismail Dönmez wrote:
> > nothing as in it wouldn't be useful? Imho it would be useful for warning 
> > messages like "do you want to delete all files" etc.
> 
> Certainly it would be useful.

We need a completion framework for translation which might or might not use
gettext in the back end.  There were some discussions about this a while
ago, I think about the time we first got the line editor to use multibyte
characters, but I can't find them now.  This is a big project somebody will
have to volunteer for.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Bug#451382: i18n is NOT so easy!

[Peter Stephenson]

Peter Stephenson wrote:
> On Thu, 6 Dec 2007 11:10:22 -0500
> Clint Adams <clint@zsh.org> wrote:
> We need a completion framework for translation which might or might not use
> gettext in the back end.  There were some discussions about this a while
> ago, I think about the time we first got the line editor to use multibyte
> characters, but I can't find them now.

Found it: see thread around

http://www.zsh.org/mla/workers/2006/msg00753.html

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Bug#451382: i18n is NOT so easy!

[Clint Adams]

On Fri, Dec 07, 2007 at 02:11:41PM +0000, Peter Stephenson wrote:
> Found it: see thread around
> 
> http://www.zsh.org/mla/workers/2006/msg00753.html

I think it would be easier to do something like bash's $"" interface to
gettext and co-opt that for completion translations.

Re: Bug#451382: i18n is NOT so easy!

[Peter Stephenson]

Clint Adams wrote:
> On Fri, Dec 07, 2007 at 02:11:41PM +0000, Peter Stephenson wrote:
> > Found it: see thread around
> > 
> > http://www.zsh.org/mla/workers/2006/msg00753.html
> 
> I think it would be easier to do something like bash's $"" interface to
> gettext and co-opt that for completion translations.

As far as I understood it (it doesn't seem to be well documented) that
only does translations which are pre-compiled into the shell (or rather
its libraries).  We need something which can be updated with completion
functions.  It's OK if the definitions are in another file (though we
could presumably have an interface which adds translations from the
completion function itself) but it needs to be added at run time.

Possibly we can still do this with $"...", but I don't like the idea
that if you change the original message you can no longer find the
translation, which seems to me to be asking for trouble.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Bug#451382: i18n is NOT so easy!

[Peter Stephenson]

On Fri, 07 Dec 2007 17:26:57 +0000
Peter Stephenson <pws@csr.com> wrote:
> Clint Adams wrote:
> > On Fri, Dec 07, 2007 at 02:11:41PM +0000, Peter Stephenson wrote:
> > > Found it: see thread around
> > > 
> > > http://www.zsh.org/mla/workers/2006/msg00753.html
> > 
> > I think it would be easier to do something like bash's $"" interface to
> > gettext and co-opt that for completion translations.
> 
> As far as I understood it (it doesn't seem to be well documented) that
> only does translations which are pre-compiled into the shell (or rather
> its libraries).  We need something which can be updated with completion
> functions.  It's OK if the definitions are in another file (though we
> could presumably have an interface which adds translations from the
> completion function itself) but it needs to be added at run time.
> 
> Possibly we can still do this with $"...", but I don't like the idea
> that if you change the original message you can no longer find the
> translation, which seems to me to be asking for trouble.

Further thoughts after groping through the gettext documentation for a
bit... this is not a definitive answer (though rather closer to one than
when I originally wrote that two hours and counting ago) but unless I post
it now I'll forget it.  A summary is that I believe we can use the
internationalization functions in the library behind gettext(), to avoid
reinventing the wheel and maintain some compatibility, but it'll take a
bit more care to get this right than simply $"<msgid>" plus
gettext("<msgid>").

I think we have two basic problems with the simplest $"..." / gettext()
interface.


1.

The problem in the last paragraph quoted.  I'm convinced this is a real
problem:  unlike with C programmes, the urge to tinker with strings in
shell functions is strong and if there's no visual cue that this has bad
side effects then the interface is, in my view, fundamentally broken.
To put it another way, only programmers tinker with C programmes while
users are actively encouraged to tinker with shell functions, so the
whole nature of the interface needs to be rethought to make it clear and
robust rather than minimal.

However, this isn't insuperable.  The "msgid" is only by convention the
original string and could be anything; it was designed to be simple in
the case of having many calls to gettext() throughout a programme.  As
we essentially have only one point of entry for translations in shell
functions (the shell's C code is a separate and much simpler problem
since this isn't fundamentally different from any other C programme), we
can do it how we like.  We can, for example, have translation strings
like:

$"_mount_nfs_access_acregmin:specify cached file attributes minimum hold time"

and have the following rule:

- If the string is in the form
    <identifer_character> * ":" . *
  (we might need to make this more complicated eventually), first
  attempt look-up with the identifier characters.  If the lookup doesn't
  return the original string, this is the text we want.
- Otherwise look up with the whole string.  This is for compatibility.  Use
  of this in zsh functions would be deprecated.
- If it still returns the original string but there is an identifier
  part, return the string after the ":".
- Maybe we want some rule about aliasing, it's not clear (we can leave
  it until a use becomes obvious). 

This scheme has various merits:  (i) it is robust about changes to the
English text (ii) the explicit msgid serves as a visual cue that
there's something here that shouldn't be monkeyed with without good
reason (and that even if you change the English text it should mean the
same thing) (iii) the msgid in the catalogues is compact.


2.

Unfortunately there's also the problem of finding message catalogues.
For the same reason that it's designed for simplicity with pre-compiled
programmes, gettext() itself appears to require them to be in a
particular hierarchy the top of which is determined at compile time.

This isn't good enough in our case.  We have functions that are
installed at different places in the function path.  The path can change
and the only clean way of finding message catalogues is using the same
path.  We *could* collect all translations at shell installation and
simply shrug our shoulders saying "that's your lot", but in my view this
is too botched to consider.  (As far as I can tell this is what happens
in bash.)  It's a key part of the way the completion system works that
people can customize it themselves just by writing functions, and even
if adding translations to your own functions is unusual I still don't
think being limited to a predefined set is acceptable.  I don't mind
users (which includes administrators) having to run some utility to add,
or add to, a message catalogue, but I do mind them having to modify the
shell configuration and reinstall; even updating the shell libraries
with something like one of Clint's out-of-tree modules seems a bit over
the top.

However, it seems like we can get something better by interfacing to the
library at a lower level, in particular to catopen() (strictly this is a
different family of interfaces).  That accepts an absolute path to a
catalogue and also uses the environment variable NLSPATH to search for
files.  It's currently unclear to me how to mix use of a shell-specified
directory (determined, in ways we'll need to discuss, from $fpath) with
a user-specified language (since I presume the library has an
intelligent system of fallbacks we don't want to have to imitate).
Unfortunately it looks like this absolute paths aren't portable, either.
If the worst comes to the worst, we may need to alter the environment
variable directly: for example, temporarily either appending or
prepending the zsh directories to it.  (I don't think requiring the user
to modify NLSPATH as well as $fpath is a good idea; I think the shell
should "just find" the right catalogues associated with functions, as
with .zwc files.)


Comments on this are obviously welcome.


To proceed I think we need the following.  The second and third parts
should wait until after 4.3.5 (which I'll make before Christmas, despite
the open bugs, since I haven't seen anything which is obviously worse
than in 4.3.4).  They should also wait until after the first part is
resonably clear.

I. Design:
- finalize the rule for $"..." (or equivalent)
- invent rules for finding the catalogue which should probably be
flexible, ideally allowing both per-fpath-directory and
per-autoloadable-function files while still allowing the user to have
all their own translations collected in one place.  For the last case it
would probably be OK to fall back on NLSPATH.  (I'm not implying
people will use all the mechanisms, just that at this stage we should
plan on flexibility.)
- decide if we want strings in the source to use a similar scheme
or (perhaps better) just normal gettext() rules.

II. Shell source:
- add parsing for $"..."
- add config support for locating libraries for language catalogues and
(where necessary) determining their abilities
- also (a separate job) we should prepare the C code for use of
gettext() --- as I said, this is conceptually simpler but still a lot
of work.  Someone needs to look at gettextize:  this is really part of
the previous point except that we won't want to rely just on the GNU
version; a quick look suggests it assumes a bit to much of a standard
GNU interface in some areas, but I haven't gone into any detail.
- add some trial mechanism behind $"..." using catopen() / catgets() /
catclose().  This is where we're going to need the most fiddling to get
the interface right.

III. Shell functions etc.:
- add a few trial translation files for the completion system and
possibly other files to test the water
- ditto translations for strings in the shell's source code
- write a whole set of utilities that
  - create bare catalogues
  - update catalogues with untranslated strings
  - check for uniqueness of the zsh msgid (needs some subtlety since
  obviously reuse is a good thing:  presumably we need to check that
  the English text after the colon is the same in both cases)
  - install catalogues
  - manipulate (e.g. agglomerate) catalogues
  - list or query what translations are available
  - check catalogues for redundant translations
This is probably the biggest chunk of work.  It would be OK at least
initially to rely on the gettext utilities where possible, but I suspect
that in many areas we're on our own:  it looks like this hasn't been
done before in a way that takes into account end user requirements
adequately (obviously I'd be interested in hearing otherwise).

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug#451382: i18n is NOT so easy!

[Bart Schaefer]

On Dec 9,  6:01pm, Peter Stephenson wrote:
} Subject: Re: Bug#451382: i18n is NOT so easy!
}
} This scheme has various merits: (i) it is robust about changes to
} the English text (ii) the explicit msgid serves as a visual cue that
} there's something here that shouldn't be monkeyed with without good
} reason (and that even if you change the English text it should mean
} the same thing) (iii) the msgid in the catalogues is compact.

This is close to the same scheme that I [*] adopted for localization
of zmail twelve years ago.  Except that we used a two-argument C
macro with the msgid and English text, rather than a delimited string.

We also had a number of tools that massaged the C source to add any new
msgid where a programmer had forgotten to use one, and to extract and
build the default English catalog file which could then be turned over
to translators.  It'd be pretty easy, I expect, to write a perl script
to find $"..." strings in shell scripts and extract them.

I'd be cautious about treating everything up to the first colon in a
$"..." string as a msgid key, though.  Error messages are going to
look like $"thing that failed: reason it failed" a lot of the time.  Or
would that have to be written "thing that failed: "$"reason it failed"
for this to work in the first place?  Anyway, it might be better to
adopt something like $"{msgid}original text" and treat both $"{message}"
and $"message" the same when only one of the two parts is found.

An additional issue that zsh may or may not have to address is that
you need entirely separate strings for things like plurals.  You can't
localize something like:

	There %s %d thing%s in the bucket

where the %s get replaced by "are" and "s" when the %d is not 1, and
"is" and "" otherwise.  You must instead have two strings (sometimes
three for the zero case):

	There are %d things in the bucket
	There is 1 thing in the bucket
	There is nothing in the bucket

There are gobs of other niggling details that I'm sure I've forgotten.

} However, it seems like we can get something better by interfacing to
} the library at a lower level, in particular to catopen() (strictly
} this is a different family of interfaces). That accepts an absolute
} path to a catalogue and also uses the environment variable NLSPATH to
} search for files.

This is also what I did back then in zmail -- gettext() didn't really
even exist yet at that point, at least not in a fully-developed form.
The POSIX cat*() interfaces work just fine, though NLSPATH searching
has some pretty nasty bugs on older operating systems.

[*] That's sort of the royal "I" as actually there was a whole team
of people working for me on it.

-- 

Re: Bug#451382: i18n is NOT so easy!

[Peter Stephenson]

Bart Schaefer wrote:
> I'd be cautious about treating everything up to the first colon in a
> $"..." string as a msgid key, though.  Error messages are going to
> look like $"thing that failed: reason it failed" a lot of the time.  Or
> would that have to be written "thing that failed: "$"reason it failed"
> for this to work in the first place?  Anyway, it might be better to
> adopt something like $"{msgid}original text" and treat both $"{message}"
> and $"message" the same when only one of the two parts is found.

Fine, that gives us an easier test for whether there's a special msgid
anyway.  (I would still propose that msgid has to consist only of shell
identifier characters for simplicity.)

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

PATCH: completion of glob qualifiers

[Peter Stephenson]

This is mostly useful as a memory jogger rather than actually to save
you typing a single character.  However, I've added a pretty useless
function to save you typing a delimiter character.  (The alternative
was simply to print a message saying "delimiter", which was
really low tech.)

It works OK so far but it does raise various issues about related things
that might be got to work better.  You might run across those.

The change to _alternative fixes a pretty obvious bug.  Presumably
no one has been using it with arguments that just print a message.

Index: Completion/Base/Utility/_alternative
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Base/Utility/_alternative,v
retrieving revision 1.3
diff -u -r1.3 _alternative
--- Completion/Base/Utility/_alternative	13 Mar 2003 10:00:21 -0000	1.3
+++ Completion/Base/Utility/_alternative	23 Feb 2008 00:04:31 -0000
@@ -75,7 +75,7 @@
 done
 
 for descr in "$mesgs[@]"; do
-  _message -e "${descr%%:*}" "${desc#*:}"
+  _message -e "${descr%%:*}" "${descr#*:}"
 done
 
 return 1
Index: Completion/Unix/Type/_path_files
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Type/_path_files,v
retrieving revision 1.26
diff -u -r1.26 _path_files
--- Completion/Unix/Type/_path_files	25 Oct 2007 09:00:04 -0000	1.26
+++ Completion/Unix/Type/_path_files	23 Feb 2008 00:04:32 -0000
@@ -6,8 +6,9 @@
 local linepath realpath donepath prepath testpath exppath skips skipped
 local tmp1 tmp2 tmp3 tmp4 i orig eorig pre suf tpre tsuf opre osuf cpre
 local pats haspats ignore pfx pfxsfx sopt gopt opt sdirs ignpar cfopt listsfx
-local nm=$compstate[nmatches] menu matcher mopts sort match mid accex fake
+local nm=$compstate[nmatches] menu matcher mopts sort mid accex fake
 local listfiles listopts tmpdisp
+local -a match mbegin mend
 
 typeset -U prepaths exppaths
 
@@ -349,7 +350,19 @@
 
     tmp2=( "$tmp1[@]" )
 
-    if [[ "$tpre$tsuf" = */* ]]; then
+    # Look for glob qualifiers.
+    # Extra nastiness to be careful about a quoted parenthesis.
+    # The initial tests look for parentheses with zero or an
+    # even number of backslashes in front.
+    # The later test looks for an outstanding quote.
+    if [[ ( -o bareglobqual && \
+	      "$tpre/$tsuf" = (#b)((*[^\\]|)(\\\\)#\()([^\)]#) || \
+            -o extendedglob && \
+		"$tpre/$tsuf" = (#b)((*[^\\]|)(\\\\)#"(#q")([^\)]#) \
+	  ) && -z $compstate[quote] ]]; then
+       compset -p ${#match[1]}
+       _globquals
+    elif [[ "$tpre$tsuf" = */* ]]; then
       compfiles -P$cfopt tmp1 accex "$skipped" "$_matcher $matcher[2]" "$sdirs" fake
     elif [[ "$sopt" = *[/f]* ]]; then
       compfiles -p$cfopt tmp1 accex "$skipped" "$_matcher $matcher[2]" "$sdirs" fake "$pats[@]"
Index: Completion/Zsh/Type/_delimiters
===================================================================
RCS file: Completion/Zsh/Type/_delimiters
diff -N Completion/Zsh/Type/_delimiters
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Completion/Zsh/Type/_delimiters	23 Feb 2008 00:04:32 -0000
@@ -0,0 +1,16 @@
+#autoload
+
+# Simple function to offer delimiters for modifiers and qualifers.
+# Single argument is tag to use.
+
+local expl
+local -a list
+
+zstyle -a ":completion:${curcontext}:$1" delimiters list ||
+  list=(: + / - %)
+
+if (( ${#list} )); then
+  _wanted delimiters expl delimiter compadd -S '' -a list
+else
+  _message delimiter
+fi
Index: Completion/Zsh/Type/_globqual_delims
===================================================================
RCS file: Completion/Zsh/Type/_globqual_delims
diff -N Completion/Zsh/Type/_globqual_delims
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Completion/Zsh/Type/_globqual_delims	23 Feb 2008 00:04:32 -0000
@@ -0,0 +1,24 @@
+#autoload
+
+# Helper for _globquals.  Sets delim to delimiter to match.
+
+# don't restore special parameters
+compstate[restore]=no
+
+delim=$PREFIX[1]
+compset -p 1
+
+# One of matching brackets?
+# These don't actually work: the parser gets very confused.
+local matchl="<({[" matchr=">)}]"
+integer ind=${matchl[(I)$delim]}
+
+(( ind )) && delim=$matchr[ind]
+
+if compset -P "[^$delim]#$delim"; then
+  # Completely matched.
+  return 0
+else
+  # Still in delimiter
+  return 1
+fi
Index: Completion/Zsh/Type/_globquals
===================================================================
RCS file: Completion/Zsh/Type/_globquals
diff -N Completion/Zsh/Type/_globquals
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Completion/Zsh/Type/_globquals	23 Feb 2008 00:04:32 -0000
@@ -0,0 +1,233 @@
+#autoload
+
+local state=qual expl char delim
+local -a alts
+
+while [[ -n $PREFIX ]]; do
+  char=$PREFIX[1]
+  compset -p 1
+  case $char in
+    ([-/F.@=p*rwxAIERWXsStUG^MTNDn,])
+    # no argument
+    ;;
+
+    (%)
+    # optional b, c
+    if [[ $PREFIX[1] = [bc] ]]; then
+      compset -p 1
+    fi
+    ;;
+
+    (f)
+    if ! compset -P "[-=+][0-7?]##"; then
+      if [[ -z $PREFIX ]]; then
+	_delimiters qualifier-f
+	return
+      elif ! _globqual_delims; then
+	# still completing mode spec
+	_message "mode spec"
+	return
+      fi
+    fi
+    ;;
+
+    (e)
+    # complete/skip delimited command line
+    if [[ -z $PREFIX ]]; then
+      _delimiters qualifer-e
+      return
+    elif ! _globqual_delims; then
+      # still completing command to eval
+      compset -q
+      _normal
+      return
+    fi
+    ;;
+
+    (+)
+    # complete/skip command name (no delimiters)
+    if [[ $PREFIX = [[:IDENT:]]# ]]; then
+      # either nothing there yet, or still on name
+      _command_names
+      return
+    fi
+    compset -P '[[:IDENT:]]##'
+    ;;
+
+    (d)
+    # complete/skip device
+    if [[ -z $PREFIX ]]; then
+      _message device ID
+      return
+    fi
+    # It's pointless trying to complete the device.
+    # Simply assume it's done.
+    compset -p '[[:digit:]]##'
+    ;;
+
+    (l)
+    # complete/skip link count
+    if [[ PREFIX = ([-+]|) ]]; then
+      _message link count
+      return
+    fi
+    # It's pointless trying to complete the link count.
+    # Simply assume it's done.
+    compset -P '([-+]|)[[:digit:]]##'
+    ;;
+
+    (u)
+    # complete/skip UID or delimited user
+    if ! compset -P '[[:digit:]]##'; then
+      if [[ -z $PREFIX ]]; then
+	_delimiters qualifier-u
+	return
+      elif ! _globqual_delims; then
+	# still completing user
+	_users -S $delim
+	return
+      fi
+    fi
+    ;;
+
+    (g)
+    # complete/skip GID or delimited group
+    if ! compset -P '[[:digit:]]##'; then
+      if [[ -z $PREFIX ]]; then
+	_delimiter qualifier-g
+	return
+      elif ! _globqual_delims; then
+	# still completing group
+	_groups -S $delim
+	return
+      fi
+    fi
+    ;;
+
+    ([amc])
+    if ! compset -P '([Mwhms]|)([-+]|)<->'; then
+      # complete/skip relative time spec
+      alts=()
+      if ! compset -P '[Mwhms]' && [[ -z $PREFIX ]]; then
+	alts+=(
+	  "time-specifiers:time specifier:\
+((M\:months w\:weeks h\:hours m:\minutes s\:seconds))")
+      fi
+      if ! compset -P '[-+]' && [[ -z $PREFIX ]]; then
+	alts+=("senses:sense:((-\:less\ than +\:more\ than))")
+      fi
+      alts+=('digits:digit: ')
+      _alternative $alts
+      return
+    fi
+    ;;
+
+    (L)
+    # complete/skip file size
+    if ! compset -P '([kKmMpP]|)([-+]|)<->'; then
+      # complete/skip size spec
+      alts=()
+      if ! compset -P '[kKmMpP]' && [[ -z $PREFIX ]]; then
+	alts+=(
+	  "size-specifiers:size specifier:\
+((k\:kb m\:mb p\:512-byte\ blocks))")
+      fi
+      if ! compset -P '[-+]' && [[ -z $PREFIX ]]; then
+	alts+=("senses:sense:((-\:less\ than +\:more\ than))")
+      fi
+      alts+=('digits:digit: ')
+      _alternative $alts
+      return
+    fi
+    ;;
+    
+    ([oO])
+    # complete/skip sort spec
+    if ! compset -P "?"; then
+      alts=(
+	"n:lexical order of name"
+	"L:size of file"
+	"l:number of hard links"
+	"a:last access time"
+	"m:last modification time"
+	"c:last inode change time"
+	"d:directory depth"
+	)
+      _describe -t sort-specifiers "sort specifier" alts -Q -S ''
+      return
+    fi
+    ;;
+
+    (\[)
+    # complete/skip range: check for closing bracket
+    if ! compset -P "(-|)[[:digit:]]##(,(-|)[[:digit:]]##|)]"; then
+      if compset -P "(-|)[[:digit:]]##,"; then
+	_message end of range
+      else
+	_message start of range
+      fi
+      return
+    fi
+    ;;
+
+    (:)
+    # complete modifiers and don't stop completing them
+    _history_modifiers
+    return
+    ;;
+  esac
+done
+
+case $state in
+  (qual)
+  local -a quals
+  quals=(
+    "/:directories"
+    "F:non-empty directories"
+    ".:plain files"
+    "@:symbolic links"
+    "=:sockets"
+    "p:name pipes (FIFOS)"
+    "*:executable plain files"
+    "%:device files"
+    "r:owner-readable"
+    "w:owner-writeable"
+    "x:owner-executable"
+    "A:group-readable"
+    "I:group-writeable"
+    "E:group-executable"
+    "R:world-readable"
+    "W:world-writeable"
+    "X:world-executable"
+    "s:setuid"
+    "S:setgid"
+    "t:sticky bit set"
+    "f:+ access rights"
+    "e:execute code"
+    "+:+ command name"
+    "d:+ device"
+    "l:+ link count"
+    "U:owned by EUID"
+    "G:owned by EGID"
+    "u:+ owning user"
+    "g:+ owning group"
+    "a:+ access time"
+    "m:+ modification time"
+    "c:+ inode change time"
+    "L:+ size"
+    "^:negate qualifiers"
+    "-:follow symlinks toggle"
+    "M:mark directories"
+    "T:mark types"
+    "N:use NULL_GLOB"
+    "D:glob dots"
+    "n:numeric glob sort"
+    "o:+ sort order, up"
+    "O:+ sort order, down"
+    "[:+ range of files"
+    "):end of qualifiers"
+    "\::modifier"
+    )
+  _describe -t globquals "glob qualifier" quals -Q -S ''
+  ;;
+esac
Index: Completion/Zsh/Type/_history_modifiers
===================================================================
RCS file: Completion/Zsh/Type/_history_modifiers
diff -N Completion/Zsh/Type/_history_modifiers
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Completion/Zsh/Type/_history_modifiers	23 Feb 2008 00:04:32 -0000
@@ -0,0 +1,84 @@
+#autoload
+
+# Complete history-style modifiers; the first : will have
+# been matched and compset -p 1'd.
+# The single argument is the type of context:
+#   h  history
+#   q  glob qualifier
+#   p  parameter
+
+local -a list
+
+local type=$1 delim expl
+integer global
+
+while true; do
+  if [[ -n $PREFIX ]]; then
+    local char=$PREFIX[1]
+
+    global=0
+    compset -p 1
+    case $char in
+      ([hretpqQxlu\&])
+      # single character modifiers
+      ;;
+
+      (s)
+      # match delimiter string delimiter string delimiter
+      if [[ -z $PREFIX ]]; then
+	_delimiters modifier-s
+	return
+      fi
+      delim=$PREFIX[1]
+      compset -p 1
+      if ! compset "[^$delim]#$delim[^$delim]#$delim"; then
+	if compset "[^$delim]#$delim"; then
+	  _message original string
+	else
+	  _message replacement string
+	fi
+	return
+      fi
+      ;;
+
+      (g)
+      global=1
+      continue
+      ;;
+    esac
+
+    # modifier completely matched, see what's next.
+    compset -P : && continue
+    # if there's something other than colon next, bummer
+    [[ -n $PREFIX ]] && return 1
+
+    list=("\::modifier")
+    [[ $type = g ]] && list+=("):end of qualifiers")
+    # strictly we want a normal suffix if end of qualifiers
+    _describe -t delimiters "delimiter" list -Q -S ''
+  else
+    list=(
+      "s:substitute string"
+      "&:repeat substitution"
+      )
+    if (( ! global )); then
+      list+=(
+	"g:globally apply s or &"
+	"h:head - strip trailing path element"
+	"t:tail - strip directories"
+	"r:root - strip suffix"
+	"e:leave only extension"
+	"Q:strip quotes"
+	"l:lower case all words"
+	"u:upper case all words"
+	)
+      [[ $type = h ]] && list+=(
+	"p:print without executing"
+	"x:quote words, breaking on whitespace"
+	)
+      [[ $type = [hp] ]] && list+=("q:quote to escape further substitutions")
+    fi
+    _describe -t modifiers "modifier" list -Q -S ''
+    return
+  fi
+done
Index: Doc/Zsh/compsys.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/compsys.yo,v
retrieving revision 1.203
diff -u -r1.203 compsys.yo
--- Doc/Zsh/compsys.yo	25 Oct 2007 09:00:05 -0000	1.203
+++ Doc/Zsh/compsys.yo	23 Feb 2008 00:04:35 -0000
@@ -1262,6 +1262,15 @@
 insertion of matches should be delayed unconditionally. The default is 
 `true'.
 )
+kindex(delimiters, completion style)
+item(tt(delimiters))(
+This style is used when adding a delimiter for use with history
+modifiers or glob qualifiers that have delimited arguments.  It is
+an array of preferred delimiters to add.  Non-special characters are
+preferred as the completion system may otherwise become confused.
+The default list is tt(:), tt(+), tt(/), tt(-), tt(%).  The list
+may be empty to force a delimiter to be typed.
+)
 kindex(disabled, completion style)
 item(tt(disabled))(
 If this is set to `true', the tt(_expand_alias) completer and bindable 


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: completion of glob qualifiers

[Bart Schaefer]

On Feb 23, 12:04am, Peter Stephenson wrote:
}
} It works OK so far but it does raise various issues about related things
} that might be got to work better.  You might run across those.

Yes, this is pretty cool.  I suppose the things that might work better
are e.g. completion of modifiers after !!: and $var: ?

schaefer<501> echo ${var:Oops. Bug in parse_subst_string: errflag
Oops. Bug in parse_subst_string: errflag
Oops. Bug in parse_subst_string: errflag
Oops. Bug in parse_subst_string: errflag
Oops. Bug in parse_subst_string: errflag
Oops. Bug in parse_subst_string: errflag
Oops. Bug in parse_subst_string: errflag
Oops. Bug in parse_subst_string: errflag
Oops. Bug in parse_subst_string: errflag
              echo ${var:
No matches for `file' or `corrections'

(That was probably there before ...)

It'd also be nice if, for example, it would notice that the qualifier
needs to be followed by a numeric value and wouldn't complete another
qualifier the number was present, but that's a nit.

Re: PATCH: completion of glob qualifiers

[Peter Stephenson]

Bart Schaefer wrote:
> On Feb 23, 12:04am, Peter Stephenson wrote:
> }
> } It works OK so far but it does raise various issues about related things
> } that might be got to work better.  You might run across those.
> 
> Yes, this is pretty cool.  I suppose the things that might work better
> are e.g. completion of modifiers after !!: and $var: ?

Those need adding, but there is actually all sorts of syntax nastiness
if you try anything with metacharacters, even in places where they are
parsed properly in normal operation.  Try "${(", for example.

> It'd also be nice if, for example, it would notice that the qualifier
> needs to be followed by a numeric value and wouldn't complete another
> qualifier the number was present, but that's a nit.

It's supposed to do that, and I'm not sure which one wasn't working,
although looking again I can see a few places I haven't handled
consistently.

Index: Completion/Zsh/Type/_globquals
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Zsh/Type/_globquals,v
retrieving revision 1.1
diff -u -r1.1 _globquals
--- Completion/Zsh/Type/_globquals	23 Feb 2008 00:10:25 -0000	1.1
+++ Completion/Zsh/Type/_globquals	23 Feb 2008 17:33:47 -0000
@@ -56,24 +56,18 @@
 
     (d)
     # complete/skip device
-    if [[ -z $PREFIX ]]; then
-      _message device ID
+    if ! compset -p '[[:digit:]]##'; then
+      _message "device ID"
       return
     fi
-    # It's pointless trying to complete the device.
-    # Simply assume it's done.
-    compset -p '[[:digit:]]##'
     ;;
 
     (l)
     # complete/skip link count
-    if [[ PREFIX = ([-+]|) ]]; then
-      _message link count
+    if ! compset -P '([-+]|)[[:digit:]]##'; then
+      _message "link count"
       return
     fi
-    # It's pointless trying to complete the link count.
-    # Simply assume it's done.
-    compset -P '([-+]|)[[:digit:]]##'
     ;;
 
     (u)
@@ -162,9 +156,9 @@
     # complete/skip range: check for closing bracket
     if ! compset -P "(-|)[[:digit:]]##(,(-|)[[:digit:]]##|)]"; then
       if compset -P "(-|)[[:digit:]]##,"; then
-	_message end of range
+	_message "end of range"
       else
-	_message start of range
+	_message "start of range"
       fi
       return
     fi
@@ -172,7 +166,7 @@
 
     (:)
     # complete modifiers and don't stop completing them
-    _history_modifiers
+    _history_modifiers q
     return
     ;;
   esac
Index: Completion/Zsh/Type/_history_modifiers
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Zsh/Type/_history_modifiers,v
retrieving revision 1.1
diff -u -r1.1 _history_modifiers
--- Completion/Zsh/Type/_history_modifiers	23 Feb 2008 00:10:25 -0000	1.1
+++ Completion/Zsh/Type/_history_modifiers	23 Feb 2008 17:33:47 -0000
@@ -53,9 +53,10 @@
     [[ -n $PREFIX ]] && return 1
 
     list=("\::modifier")
-    [[ $type = g ]] && list+=("):end of qualifiers")
+    [[ $type = q ]] && list+=("):end of qualifiers")
     # strictly we want a normal suffix if end of qualifiers
     _describe -t delimiters "delimiter" list -Q -S ''
+    return
   else
     list=(
       "s:substitute string"



-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: completion of glob qualifiers

[Peter Stephenson]

On Fri, 22 Feb 2008 22:06:55 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> schaefer<501> echo ${var:Oops. Bug in parse_subst_string: errflag

This certainly isn't new, and in fact I think it's overzealous
error checking.  Because we're in a context where we might not be
able to parse something (we're doing unquoting of a string which
could contain anything, not necessarily something that can be parsed),
the error should be suppressed, and indeed noerrs is on and errflag is
explicitly and unconditionally reset by the caller of this function.  So
I think we just need the debug code to respect noerrs.

I've put a comment by the definition of noerrs reminding us how it
works.  I feel such a spoilsport doing that.

(I note we end up in -default- rather than -brace-parameter- completion,
as in the case after "${(".  I suspect it might be better to handle this
in the shell code, say _dispatch, rather than trying to hack the already
horrific context parsing inside the shell.)

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.129
diff -u -r1.129 exec.c
--- Src/exec.c	21 Feb 2008 18:27:04 -0000	1.129
+++ Src/exec.c	23 Feb 2008 18:03:29 -0000
@@ -46,7 +46,10 @@
 /**/
 int noerrexit;
 
-/* suppress error messages */
+/*
+ * noerrs = 1: suppress error messages
+ * noerrs = 2: don't set errflag on parse error, either
+ */
  
 /**/
 mod_export int noerrs;
Index: Src/lex.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/lex.c,v
retrieving revision 1.43
diff -u -r1.43 lex.c
--- Src/lex.c	17 Dec 2007 14:20:25 -0000	1.43
+++ Src/lex.c	23 Feb 2008 18:03:29 -0000
@@ -1619,7 +1619,7 @@
      * Historical note: we used to check here for olen == l, but
      * that's not necessarily the case if we stripped an RCQUOTE.
      */
-    if (c != STRING || errflag) {
+    if (c != STRING || (errflag && !noerrs)) {
 	fprintf(stderr, "Oops. Bug in parse_subst_string: %s\n",
 		errflag ? "errflag" : "c != STRING");
 	fflush(stderr);


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

${a[(i)pattern]} if a=()

[Stephane Chazelas]

Hiya,

$ zsh -f -c 'a=(b); echo ${a[(i)a]}'
2
$ zsh -f -c 'a=(); echo ${a[(i)a]}'
0

Shouldn't the above return 1?

That at leasts causes compinstall to fail on

         elt=${amenu[(i)*select*]}
	 [...]
	 amenu[$elt]="select=$select"
__ci_do_selection:91: amenu: assignment to invalid subscript range

Best regards,
Stéphane

Re: ${a[(i)pattern]} if a=()

[Bart Schaefer]

On Mar 16,  2:40pm, Stephane Chazelas wrote:
} Subject: ${a[(i)pattern]} if a=()
}
} Hiya,
} 
} $ zsh -f -c 'a=(b); echo ${a[(i)a]}'
} 2
} $ zsh -f -c 'a=(); echo ${a[(i)a]}'
} 0
} 
} Shouldn't the above return 1?

Hmm ... it seems it has always returned zero when the array is empty.
It would seem to make more sense for it to return 1, but I'm worried
there are other unforseen consequences.
 
What changed is zsh-workers/23562, which made it an error to treat a
zero subscript the same as 1 (see the thread leading up to 23562 and
the related thread ending in 23440).

Naturally we had code relying on the old behavior and didn't find all
of those cases.

} That at leasts causes compinstall to fail

The following will work around the above issue to fix that:

--- compinstall.~1.11.~	2008-02-11 20:25:13.000000000 -0800
+++ compinstall	2008-03-16 10:17:55.000000000 -0700
@@ -1438,7 +1438,7 @@
 	 done
 	 amenu=(${=menu})
 	 elt=${amenu[(i)*select*]}
-	 [[ $elt -gt $#amenu ]] && elt=
+	 [[ $elt -eq 0 || $elt -gt $#amenu ]] && elt=
 	 case $select in
 	   <->) if [[ -n $elt ]]; then
 		  amenu[$elt]="select=$select"

Re: ${a[(i)pattern]} if a=()

[Bart Schaefer]

On Mar 16, 10:20am, Bart Schaefer wrote:
}
} } $ zsh -f -c 'a=(); echo ${a[(i)a]}'
} } 0
} 
} Hmm ... it seems it has always returned zero when the array is empty.
} It would seem to make more sense for it to return 1, but I'm worried
} there are other unforseen consequences.

The patch is trivial and all tests still pass.  Thoughts?

Index: Src/params.c
===================================================================
RCS file: /extra/cvsroot/zsh/zsh-4.0/Src/params.c,v
retrieving revision 1.40
diff -c -r1.40 params.c
--- ../zsh-forge/current/Src/params.c	11 Jan 2008 01:18:07 -0000
+++ Src/params.c	16 Mar 2008 17:52:10 -0000
@@ -1317,7 +1317,7 @@
 	    } else
 		ta = getarrvalue(v);
 	    if (!ta || !*ta)
-		return 0;
+		return !down;
 	    len = arrlen(ta);
 	    if (beg < 0)
 		beg += len;

Re: ${a[(i)pattern]} if a=()

[Bart Schaefer]

On Mar 16, 10:59am, Bart Schaefer wrote:
}
} The patch is trivial and all tests still pass.

Here's a test for that patch.  Also fixes the previous test which
declared an empty array but then referenced an undefined variable.
Or at least I think what I changed is what the test really meant.
I won't commit this (or 24718) without some confirmation.

Index: Tests/D06subscript.ztst
===================================================================
--- Tests/D06subscript.ztst	26 Nov 2007 17:38:14 -0000	1.8
+++ Tests/D06subscript.ztst	16 Mar 2008 18:44:20 -0000
@@ -178,11 +178,15 @@
 >lower
 >upper
 
-  typeset -a empty_array
-  echo X${${l##*}[-1]}X
+  typeset -ga empty
+  echo X${${empty##*}[-1]}X
 0:Negative index applied to substition result from empty array
 >XX
 
+  print $empty[(i)] $empty[(I)]
+0:(i) returns 1 for empty array, (I) returns 0.
+>1 0
+
   array=(one two three four)
   print X$array[0]X
 0:Element zero is empty if KSH_ZERO_SUBSCRIPT is off.

Re: ${a[(i)pattern]} if a=()

[Peter Stephenson]

Bart Schaefer wrote:
> On Mar 16, 10:20am, Bart Schaefer wrote:
> }
> } } $ zsh -f -c 'a=(); echo ${a[(i)a]}'
> } } 0
> } 
> } Hmm ... it seems it has always returned zero when the array is empty.
> } It would seem to make more sense for it to return 1, but I'm worried
> } there are other unforseen consequences.

I think it's supposed to return 0 because it didn't match.  1 indicates
it matched the first array element.

What has changed is that 0 is now always treated as being a failed
matched, instead of being kludged to 1.  This agrees exactly, unless I'm
missing something, with the original behaviour here, which is a failed
match and therefore shouldn't correspond to an index indicating a
successful match.  The fact we happen to need a genuine index later
is a separate issue; the point here is that we haven't found it by
reverse-matching the array.  I don't see the point in propagating the
inability to detect a failed match.

The original patch,

-        [[ $elt -gt $#amenu ]] && elt=
+        [[ $elt -eq 0 || $elt -gt $#amenu ]] && elt=

seems to me a much better solution.  The first test means we're off the
beginning, the second test off the end.  Being able to distinguish both
cases is a feature.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: ${a[(i)pattern]} if a=()

[Stephane Chazelas]

On Sun, Mar 16, 2008 at 09:57:04PM +0000, Peter Stephenson wrote:
> Bart Schaefer wrote:
> > On Mar 16, 10:20am, Bart Schaefer wrote:
> > }
> > } } $ zsh -f -c 'a=(); echo ${a[(i)a]}'
> > } } 0
> > } 
> > } Hmm ... it seems it has always returned zero when the array is empty.
> > } It would seem to make more sense for it to return 1, but I'm worried
> > } there are other unforseen consequences.
> 
> I think it's supposed to return 0 because it didn't match.  1 indicates
> it matched the first array element.
[...]

According to the manual:

i
     Like `r', but gives the index of the match instead; this may not be
     combined with a second argument.  On the left side of an
     assignment, behaves like `r'.  For associative arrays, the key
     part of each pair is compared to the pattern, and the first
     matching key found is the result.  On failure substitutes one more
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     than the last currently valid index, as discussed under the
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     description of `r'.
     ^^^^^^^^^^^^^^^^^^^

That's why you get:

$ zsh -f -c 'a=(b); echo ${a[(i)a]}'
2

And I thought you should get 1 (1 more than the first valid
index)

$ zsh -f -c 'a=(); echo ${a[(i)a]}'
1

-- 
Stéphane

Re: ${a[(i)pattern]} if a=()

[Bart Schaefer]

On Mar 16, 11:23pm, Stephane Chazelas wrote:
}
} > I think it's supposed to return 0 because it didn't match.  1 indicates
} > it matched the first array element.
} [...]
} 
} According to the manual:
} 
}      matching key found is the result.  On failure substitutes one more
}                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
}      than the last currently valid index, as discussed under the
}      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
}      description of `r'.
}      ^^^^^^^^^^^^^^^^^^^

The reason we're getting into trouble here is because there is *no* valid
index when the array is empty.  0 is no longer a "currently valid index",
so to return "one more than the last" is in some sense also invalid, and
zero is the only index that is always invalid (barring KSH_* behavior).

My personal opinion is that Stephane has the right of it here, and this
empty-array case is just one that we missed way back when we added the
"missing (i) falls of the top end" behavior.

However, another way out may be to return -1 for (i) on an empty array.
Then it's "valid" for appending to the array, but distinguishable from
matching the first element.  That would be kind of a strange edge case,
though, because the $a[(i)] < $#array test for presence of an element
would still break.

Re: ${a[(i)pattern]} if a=()

[Peter Stephenson]

Bart Schaefer wrote:
> On Mar 16, 11:23pm, Stephane Chazelas wrote:
> }
> } > I think it's supposed to return 0 because it didn't match.  1 indicates
> } > it matched the first array element.
> } [...]
> } 
> } According to the manual:
> } 
> }      matching key found is the result.  On failure substitutes one more
> }                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> }      than the last currently valid index, as discussed under the
> }      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> }      description of `r'.
> }      ^^^^^^^^^^^^^^^^^^^

> The reason we're getting into trouble here is because there is *no* valid
> index when the array is empty.  0 is no longer a "currently valid index",
> so to return "one more than the last" is in some sense also invalid, and
> zero is the only index that is always invalid (barring KSH_* behavior).
> 
> My personal opinion is that Stephane has the right of it here, and this
> empty-array case is just one that we missed way back when we added the
> "missing (i) falls of the top end" behavior.

I've read through the manual again, and you're right, the sense of it is
it should be one, because it's "off the end".  As Stephane pointed out,
zero looks screwy here (whatever the arguments about validity of
indices).  So you should commit the patch.

The one that should definitely return zero is (I), but it does.  This is
where the ambiguity was in the past, since there was no "off the beginning".

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: ${a[(i)pattern]} if a=()

[Bart Schaefer]

On Mar 18, 12:13pm, Peter Stephenson wrote:
}
} I've read through the manual again, and you're right, the sense of it is
} it should be one, because it's "off the end".  As Stephane pointed out,
} zero looks screwy here (whatever the arguments about validity of
} indices).  So you should commit the patch.

Done.

Looking at documenation for this, I was reminded about this recent bit:

     Note that in subscripts with both `r' and `R' pattern characters
     are active even if they were substituted for a parameter
     (regardless of the setting of GLOB_SUBST which controls this
     feature in normal pattern matching).  It is therefore necessary to
     quote pattern characters for an exact string match.

Maybe we could press the (e) flag into service here?  I haven't looked
at how hard that would be to do, but it's semantically similar to the
existing use:

e
     This flag has no effect and for ordinary arrays is retained for
     backward compatibility only.  For associative arrays, this flag
     can be used to force * or @ to be interpreted as a single key
     rather than as a reference to all values.  This flag may be used
     on the left side of an assignment.

Re: ${a[(i)pattern]} if a=()

[Peter Stephenson]

On Tue, 18 Mar 2008 08:47:28 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Looking at documenation for this, I was reminded about this recent bit:
> 
>      Note that in subscripts with both `r' and `R' pattern characters
>      are active even if they were substituted for a parameter
>      (regardless of the setting of GLOB_SUBST which controls this
>      feature in normal pattern matching).  It is therefore necessary to
>      quote pattern characters for an exact string match.
> 
> Maybe we could press the (e) flag into service here?  I haven't looked
> at how hard that would be to do, but it's semantically similar to the
> existing use

Yes, that seems perfectly reasonable, and it was easy to do (except I've
just got back from holiday so it's appeared a week late).  It might look
a little bizarre that in one case we untokenize() and in the other case
we tokenize():  you might think we'd need just one or the other.  The
difference occurs if the substitution is inside double quotes: if so, we
need to tokenize to do pattern matching, while if not we need to
untokenize to make sure we don't.

It's still necessary to use a parameter as the key to guarantee all
characters are interpreted literally.  The issue is that we don't do
full argument parsing on the subscript; it's handled a bit like a
special case of double quoting (but with a different terminator), so
single and double quotes don't have their quoting effect there.  I don't
think we want to change this in a hurry.

I noticed meanwhile that the optimization for pattern-character-free
strings was being confused by multibyte mode; the only difference is
speed, so it's unlikely anybody would have noticed.

Index: Doc/Zsh/params.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/params.yo,v
retrieving revision 1.41
diff -u -r1.41 params.yo
--- Doc/Zsh/params.yo	25 Oct 2007 09:33:01 -0000	1.41
+++ Doc/Zsh/params.yo	25 Mar 2008 17:08:38 -0000
@@ -227,16 +227,14 @@
 If tt(KSH_ARRAYS) is in effect, the tt(-le) should be replaced by tt(-lt).
 
 Note that in subscripts with both `tt(r)' and `tt(R)' pattern characters
-are active even if they were substituted for a parameter (regardless
-of the setting of tt(GLOB_SUBST) which controls this feature in normal
-pattern matching).  It is therefore necessary to quote pattern characters
-for an exact string match.  Given a string in tt($key), and assuming
-the tt(EXTENDED_GLOB) option is set, the following is sufficient to
-match an element of an array tt($array) containing exactly the value of
-tt($key):
+are active even if they were substituted for a parameter (regardless of the
+setting of tt(GLOB_SUBST) which controls this feature in normal pattern
+matching).  The flag `tt(e)' can be added to inhibit pattern matching.  As
+this flag does not inhibit other forms of substitution, care is still
+required; using a parameter to hold the key has the desired effect:
 
-example(key2=${key//(#m)[\][+LPAR()+RPAR()\\*?#<>~^]/\\$MATCH}
-print ${array[(R)$key2]})
+example(key2='original key'
+print ${array[(Re)$key2]})
 )
 item(tt(R))(
 Like `tt(r)', but gives the last match.  For associative arrays, gives
@@ -283,11 +281,15 @@
 The delimiter character tt(:) is arbitrary; see above.
 )
 item(tt(e))(
-This flag has no effect and for ordinary arrays is retained for backward
-compatibility only.  For associative arrays, this flag can be used to
-force tt(*) or tt(@) to be interpreted as a single key rather than as a
-reference to all values.  This flag may be used on the left side of an
-assignment.
+This flag causes any pattern matching that would be performed on the
+subscript to use plain string matching instead.  Hence
+`tt(${array[(re)*]})' matches only the array element whose value is tt(*).
+Note that other forms of substitution such as parameter substitution are
+not inhibited.
+
+This flag can also be used to force tt(*) or tt(@) to be interpreted as
+a single key rather than as a reference to all values.  It may be used
+for either purpose on the left side of an assignment.
 )
 enditem()
 
Index: Src/params.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/params.c,v
retrieving revision 1.141
diff -u -r1.141 params.c
--- Src/params.c	10 Jan 2008 10:25:31 -0000	1.141
+++ Src/params.c	25 Mar 2008 17:08:38 -0000
@@ -1007,7 +1007,7 @@
     int hasbeg = 0, word = 0, rev = 0, ind = 0, down = 0, l, i, ishash;
     int keymatch = 0, needtok = 0, arglen, len;
     char *s = *str, *sep = NULL, *t, sav, *d, **ta, **p, *tt, c;
-    zlong num = 1, beg = 0, r = 0;
+    zlong num = 1, beg = 0, r = 0, quote_arg = 0;
     Patprog pprog = NULL;
 
     ishash = (v->pm && PM_TYPE(v->pm->node.flags) == PM_HASHED);
@@ -1058,8 +1058,7 @@
 		sep = "\n";
 		break;
 	    case 'e':
-		/* Compatibility flag with no effect except to prevent *
-		 * special interpretation by getindex() of `*' or `@'. */
+		quote_arg = 1;
 		break;
 	    case 'n':
 		t = get_strarg(++s, &arglen);
@@ -1286,7 +1285,10 @@
 	    }
 	}
 	if (!keymatch) {
-	    tokenize(s);
+	    if (quote_arg)
+		untokenize(s);
+	    else
+		tokenize(s);
 	    remnulargs(s);
 	    pprog = patcompile(s, 0, NULL);
 	} else
Index: Src/pattern.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/pattern.c,v
retrieving revision 1.41
diff -u -r1.41 pattern.c
--- Src/pattern.c	23 Oct 2007 16:09:10 -0000	1.41
+++ Src/pattern.c	25 Mar 2008 17:08:38 -0000
@@ -511,7 +511,7 @@
 
     if (!(patflags & PAT_ANY)) {
 	/* Look for a really pure string, with no tokens at all. */
-	if (!patglobflags
+	if (!(patglobflags & ~GF_MULTIBYTE)
 #ifdef __CYGWIN__
 	    /*
 	     * If the OS treats files case-insensitively and we
Index: Test/D04parameter.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D04parameter.ztst,v
retrieving revision 1.32
diff -u -r1.32 D04parameter.ztst
--- Test/D04parameter.ztst	11 Mar 2008 10:00:39 -0000	1.32
+++ Test/D04parameter.ztst	25 Mar 2008 17:08:43 -0000
@@ -282,6 +282,7 @@
   print ${(P)bar}
 0:${(P)...}
 >I'm nearly out of my mind with tedium
+#' deconfuse emacs
 
   foo=(I could be watching that programme I recorded)
   print ${(o)foo}
@@ -375,6 +376,7 @@
   print ${(QX)foo}
 1:${(QX)...}
 ?(eval):2: unmatched "
+# " deconfuse emacs
 
   array=(characters in an array)
   print ${(c)#array}
@@ -411,6 +413,7 @@
   print ${(pl.10..\x22..X.)foo}
 0:${(pl...)...}
 >Xresulting """"Xwords roariously """Xpadded
+#" deconfuse emacs
 
   print ${(l.5..X.r.5..Y.)foo}
   print ${(l.6..X.r.4..Y.)foo}
@@ -870,6 +873,7 @@
 0:Parameters associated with backreferences
 >match 12 16 match
 >1 1 1
+#' deconfuse emacs
 
   string='and look for a MATCH in here'
   if [[ ${(S)string%%(#m)M*H} = "and look for a  in here" ]]; then
@@ -1010,3 +1014,36 @@
 >fields
 >in
 >it
+
+  array=('%' '$' 'j' '*' '$foo')
+  print ${array[(i)*]} "${array[(i)*]}"
+  print ${array[(ie)*]} "${array[(ie)*]}"
+  key='$foo'
+  print ${array[(ie)$key]} "${array[(ie)$key]}"
+  key='*'
+  print ${array[(ie)$key]} "${array[(ie)$key]}"
+0:Matching array indices with and without quoting
+>1 1
+>4 4
+>5 5
+>4 4
+
+# Ordering of associative arrays is arbitrary, so we need to use
+# patterns that only match one element.
+  typeset -A assoc_r
+  assoc_r=(star '*' of '*this*' and '!that!' or '(the|other)')
+  print ${(kv)assoc_r[(re)*]}
+  print ${(kv)assoc_r[(re)*this*]}
+  print ${(kv)assoc_r[(re)!that!]}
+  print ${(kv)assoc_r[(re)(the|other)]}
+  print ${(kv)assoc_r[(r)*at*]}
+  print ${(kv)assoc_r[(r)*(ywis|bliss|kiss|miss|this)*]}
+  print ${(kv)assoc_r[(r)(this|that|\(the\|other\))]}
+0:Reverse subscripting associative arrays with literal matching
+>star *
+>of *this*
+>and !that!
+>or (the|other)
+>and !that!
+>of *this*
+>or (the|other)


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

PATCH: rewrite of completion matching

[Peter Stephenson]

This is the long-hoped-for rewrite of the completion matching code so as
to handle arbitary characters instead of just a set of 256 single-byte
characters.  However, this is only half the story:  I have not modified
it to use wide characters yet, so it won't work with multibyte character
sets.  That second step ought to be much more mechanical than this one,
though it's still not trivial to identify anything as standard as a
well-defined interface to the matching code so it will take some
fiddling.

Astonishingly, after I'd removed all the trivial errors, it passed all
the compmatch tests straight away; so, although there are bound to be
glitches, it's going to need wider testing to find them out.

It's now possible (and recommended) to use m:{[:lower:]}={[:upper:]}
etc. for case matching, but until I change the code for wide characters
this won't actually gain you much.  I've updated compinstall to do
this.

By the way, you can now do partial word completion with
'r:|[[:punct:]]=**' instead of 'r:|[.,_-]=**', should you want to.

There are some limitations.  I'm aware of one place which is plain
wrong: if you use a negative character class assertion, [^...], in file
matching, then compfiles -p/-P is supposed to generate a match that
matches either that class, or the character on the command line.  Cases
like "[^[:upper:]]" with the character on the command line "A" stumped
me, so this doesn't work.  (It ought to be possible to do this as
"(A|[^[:upper:]])" but I've left such subtleties for now.)  What the
effect of this is likely to be is obscure, given that none of the
compfiles code is documented and nor is the point where it's called in
_path_files.  However, I don't think negative character class assertions
are very common in completion.

Other places marked "TODO" are mostly points where I noticed the
existing code was probably imperfect but haven't done anything about it,
rather than new problems.

I've handled the difficult case, the one where Bart and Andrey provided
commentary on what was supposed to be going on, as follows.  Instead of
attempting to assemble the string for the command line immediately and
then testing whether it was correct, I have assembled an appropriate set
of pattern match assertions (or restrictions, as I called them in the
code) for the string, and passed that down to the point where pattern
matching is done.  At this point the assertions are used to attempt to
recover the character on the line.  If it remained ambiguous even at
this point I have given up---clearly there's not a lot of point in
completion if it's simply guessing characters, so I don't think this is
likely to be a problem in practice.  Actually, I don't think this code
is used much for anything apart from correspondence classes, which is
the easy case.

I know that the matching test suite does tickle that code, but only in
small ways---individual characters being replaced by others.  Hence I
remain dubious that it's all perfect.  (In fact, I just discovered a
major error and the tests passed even before I'd fixed it.)  However,
by the same token I've no reason to believe the code it replaces was
perfect either.


Index: Completion/compinstall
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/compinstall,v
retrieving revision 1.12
diff -u -r1.12 compinstall
--- Completion/compinstall	18 Mar 2008 15:14:22 -0000	1.12
+++ Completion/compinstall	7 Jun 2008 20:19:34 -0000
@@ -985,8 +985,14 @@
   for (( eltcnt = 1; eltcnt <= $#mlist; eltcnt++ )); do
     [[ $mlist[eltcnt] == "+"* ]] && a_or_r[$eltcnt]='+'
     [[ -z $mlist[$eltcnt] ]] && n_list[$eltcnt]=$eltcnt
+    # Accept the old form of lower/upper correspondence, but we'll
+    # output the new one instead.
     [[ $mlist[$eltcnt] = *"m:{a-z}={A-Z}"* ]] && c_list[$eltcnt]=$eltcnt
+    [[ $mlist[$eltcnt] = *"m:{[:lower:]}={[:upper:]}"* ]] &&
+      c_list[$eltcnt]=$eltcnt
     [[ $mlist[$eltcnt] = *"m:{a-zA-Z}={A-Za-z}"* ]] && C_list[$eltcnt]=$eltcnt
+    [[ $mlist[$eltcnt] = *"m:{[:lower:][:upper:]}={[:upper:][:lower:]}"* ]] &&
+      C_list[$eltcnt]=$eltcnt
     # For partial word stuff, we use backreferences to find out what
     # the set of separators was.
     if [[ $mlist[$eltcnt] = (#b)*"r:|["([^\]]#)"]=*"#" r:|=*"* ]]; then
@@ -1105,8 +1111,10 @@
   # and reconstructing the elements of the matcher array.
   for (( eltcnt = 1; eltcnt <= 4; eltcnt++ )); do
     elt=
-    [[ $c_list[$eltcnt] != ' ' ]] && elt="${elt:+$elt }m:{a-z}={A-Z}"
-    [[ $C_list[$eltcnt] != ' ' ]] && elt="${elt:+$elt }m:{a-zA-Z}={A-Za-z}"
+    [[ $c_list[$eltcnt] != ' ' ]] &&
+      elt="${elt:+$elt }m:{[:lower:]}={[:upper:]}"
+    [[ $C_list[$eltcnt] != ' ' ]] &&
+      elt="${elt:+$elt }m:{[:lower:][:upper:]}={[:upper:][:lower:]}"
     [[ $p_list[$eltcnt] != ' ' ]] &&
       elt="${elt:+$elt }r:|[${pw_seps[$eltcnt]}]=*${pw_dstar[$eltcnt]}\
  r:|=*${pw_dstar[$eltcnt]}"
Index: Doc/Zsh/compwid.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/compwid.yo,v
retrieving revision 1.40
diff -u -r1.40 compwid.yo
--- Doc/Zsh/compwid.yo	7 Oct 2006 12:15:57 -0000	1.40
+++ Doc/Zsh/compwid.yo	7 Jun 2008 20:19:35 -0000
@@ -918,15 +918,35 @@
 meaning directly after the opening brace.  They indicate that a range of
 characters on the line match a range of characters in the trial
 completion, but (unlike ordinary character classes) paired according to
-the corresponding position in the sequence. For example, to make any
-lowercase letter on the line match the corresponding uppercase letter in
-the trial completion, you can use `tt(m:{a-z}={A-Z})'.  More than one
-pair of classes can occur, in which case the first class before the
-tt(=) corresponds to the first after it, and so on.  If one side has
+the corresponding position in the sequence.  For example, to make any
+ASCII lower case letter on the line match the corresponding upper case
+letter in the trial completion, you can use `tt(m:{a-z}={A-Z})'
+(however, see below for the recommended form for this).  More
+than one pair of classes can occur, in which case the first class before
+the tt(=) corresponds to the first after it, and so on.  If one side has
 more such classes than the other side, the superfluous classes behave
 like normal character classes.  In anchor patterns correspondence classes
 also behave like normal character classes.
 
+The standard `tt([:)var(name)tt(:])' forms described for standard shell
+patterns,
+ifnzman(noderef(Filename Generation))\
+ifzman(see the section FILENAME GENERATION in zmanref(zshexpn)),
+may appear in correspondence classes as well as normal character
+classes.  The only special behaviour in correspondence classes is if
+the form on the left and the form on the right are each one of
+tt([:upper:]), tt([:lower:]).  In these cases the
+character in the word and the character on the line must be the same up
+to a difference in case.  Hence to make any lower case character on the
+line match the corresponding upper case character in the trial
+completion you can use `tt(m:{[:lower:]}={[:upper:]})'.  Although the
+matching system does not yet handle multibyte characters, this is likely
+to be a future extension, at which point this syntax will handle
+arbitrary alphabets ; hence this form, rather than the use of explicit
+ranges, is thee recommended form.  In other cases
+`tt([:)var(name)tt(:])' forms are allowed, but imply no special
+constraint on the characters beyond that implied by the test itself.
+
 The pattern var(tpat) may also be one or two stars, `tt(*)' or
 `tt(**)'. This means that the pattern on the command line can match
 any number of characters in the trial completion. In this case the
@@ -948,7 +968,7 @@
 generated and uppercase letters on the line match the corresponding
 lowercase letters in the words:
 
-example(compadd -M 'L:|[nN][oO]= M:_= M:{A-Z}={a-z}' - \ 
+example(compadd -M 'L:|[nN][oO]= M:_= M:{[:up[er:]}={[:lower:]}' - \ 
   ${(k)options} )
 
 The first part says that the pattern `tt([nN][oO])' at the beginning
@@ -979,12 +999,12 @@
 the same as in the option example, except here we wish to retain the
 characters in the list of completions:
 
-example(compadd -M 'm:{a-z}={A-Z}' ... )
+example(compadd -M 'm:{[:lower:]}={[:upper:]}' ... )
 
 This makes lowercase letters match their uppercase counterparts.
 To make uppercase letters match the lowercase forms as well:
 
-example(compadd -M 'm:{a-zA-Z}={A-Za-z}' ... )
+example(compadd -M 'm:{[:lower:][:upper:]}={[:upper:][:lower:]}' ... )
 
 A nice example for the use of tt(*) patterns is partial word
 completion. Sometimes you would like to make strings like `tt(c.s.u)'
@@ -1046,23 +1066,23 @@
 complete strings with trailing numbers. Here one could use the simple
 form with only one anchor as in:
 
-example(compadd -M 'r:|[A-Z0-9]=* r:|=*' LikeTHIS FooHoo 5foo123 5bar234)
+example(compadd -M 'r:|[[:upper:]0-9]=* r:|=*' LikeTHIS FooHoo 5foo123 5bar234)
 
 But with this, the string `tt(H)' would neither complete to `tt(FooHoo)'
 nor to `tt(LikeTHIS)' because in each case there is an uppercase
 letter before the `tt(H)' and that is matched by the anchor. Likewise, 
 a `tt(2)' would not be completed. In both cases this could be changed
-by using `tt(r:|[A-Z0-9]=**)', but then `tt(H)' completes to both
+by using `tt(r:|[[:upper:]0-9]=**)', but then `tt(H)' completes to both
 `tt(LikeTHIS)' and `tt(FooHoo)' and a `tt(2)' matches the other
 strings because characters can be inserted before every uppercase
 letter and digit. To avoid this one would use:
 
-example(compadd -M 'r:[^A-Z0-9]||[A-Z0-9]=** r:|=*' \ 
+example(compadd -M 'r:[^[:upper:]0-9]||[[:upper:]0-9]=** r:|=*' \ 
     LikeTHIS FooHoo foo123 bar234)
 
 By using these two anchors, a `tt(H)' matches only uppercase `tt(H)'s that 
 are immediately preceded by something matching the left anchor
-`tt([^A-Z0-9])'. The effect is, of course, that `tt(H)' matches only
+`tt([^[:upper:]0-9])'. The effect is, of course, that `tt(H)' matches only
 the string `tt(FooHoo)', a `tt(2)' matches only `tt(bar234)' and so on.
 
 When using the completion system (see
Index: Src/pattern.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/pattern.c,v
retrieving revision 1.46
diff -u -r1.46 pattern.c
--- Src/pattern.c	17 May 2008 12:15:19 -0000	1.46
+++ Src/pattern.c	7 Jun 2008 20:19:37 -0000
@@ -193,25 +193,6 @@
  *	     	    v      v  	  	    ^
  *	            ------------------------
  */
-#define PP_ALPHA  1
-#define PP_ALNUM  2
-#define PP_ASCII  3
-#define PP_BLANK  4
-#define PP_CNTRL  5
-#define PP_DIGIT  6
-#define PP_GRAPH  7
-#define PP_LOWER  8
-#define PP_PRINT  9
-#define PP_PUNCT  10
-#define PP_SPACE  11
-#define PP_UPPER  12
-#define PP_XDIGIT 13
-#define PP_IDENT  14
-#define PP_IFS    15
-#define PP_IFSSPACE   16
-#define PP_WORD   17
-#define PP_UNKWN  18
-#define PP_RANGE  19
 
 #define	P_OP(p)		((p)->l & 0xff)
 #define	P_NEXT(p)	((p)->l >> 8)
@@ -1057,6 +1038,127 @@
     return 1;
 }
 
+
+static const char *colon_stuffs[]  = {
+    "alpha", "alnum", "ascii", "blank", "cntrl", "digit", "graph", 
+    "lower", "print", "punct", "space", "upper", "xdigit", "IDENT",
+    "IFS", "IFSSPACE", "WORD", NULL
+};
+
+/*
+ * Handle the guts of a [:stuff:] character class element.
+ * start is the beginning of "stuff" and len is its length.
+ * This code is exported for the benefit of completion matching.
+ */
+
+/**/
+mod_export int
+range_type(char *start, int len)
+{
+    const char **csp;
+
+    for (csp = colon_stuffs; *csp; csp++) {
+	if (!strncmp(start, *csp, len))
+	    return (csp - colon_stuffs) + PP_FIRST;
+    }
+
+    return PP_UNKWN;
+}
+
+
+/*
+ * Convert the contents of a [...] or [^...] expression (just the
+ * ... part) back into a string.  This is used by compfiles -p/-P
+ * for some reason.  The compiled form (a metafied string) is
+ * passed in rangestr.
+ *
+ * If outstr is non-NULL the compiled form is placed there.  It
+ * must be sufficiently long.  A terminating NULL is appended.
+ *
+ * Return the length required, not including the terminating NULL.
+ *
+ * TODO: this is non-multibyte for now.  It will need to be defined
+ * appropriately with MULTIBYTE_SUPPORT when the completion matching
+ * code catches up.
+ */
+
+/**/
+mod_export int
+pattern_range_to_string(char *rangestr, char *outstr)
+{
+    int len = 0;
+
+    while (*rangestr) {
+	if (imeta(STOUC(*rangestr))) {
+	    int swtype = STOUC(*rangestr) - STOUC(Meta);
+
+	    if (swtype == 0) {
+		/* Ordindary metafied character */
+		if (outstr)
+		{
+		    *outstr++ = Meta;
+		    *outstr++ = rangestr[1] ^ 32;
+		}
+		len += 2;
+		rangestr += 2;
+	    } else if (swtype == PP_RANGE) {
+		/* X-Y range */
+		int i;
+
+		for (i = 0; i < 2; i++) {
+		    if (*rangestr == Meta) {
+			if (outstr) {
+			    *outstr++ = Meta;
+			    *outstr++ = rangestr[1];
+			}
+			len += 2;
+			rangestr += 2;
+		    } else {
+			if (outstr)
+			    *outstr++ = *rangestr;
+			len++;
+			rangestr++;
+		    }
+
+		    if (i == 0) {
+			if (outstr)
+			    *outstr++ = '-';
+			len++;
+		    }
+		}
+	    } else if (swtype >= PP_FIRST && swtype <= PP_LAST) {
+		/* [:stuff:]; we need to output [: and :] */
+		const char *found = colon_stuffs[swtype - PP_FIRST];
+		int newlen = strlen(found);
+		if (outstr) {
+		    strcpy(outstr, "[:");
+		    outstr += 2;
+		    memcpy(outstr, found, newlen);
+		    outstr += newlen;
+		    strcpy(outstr, ":]");
+		    outstr += 2;
+		}
+		len += newlen + 4;
+		rangestr++;
+	    } else {
+		/* shouldn't happen */
+		DPUTS(1, "BUG: unknown PP_ code in pattern range");
+		rangestr++;
+	    }
+	} else {
+	    /* ordinary character, guaranteed no Meta handling needed */
+	    if (outstr)
+		*outstr++ = *rangestr;
+	    len++;
+	    rangestr++;
+	}
+    }
+
+    if (outstr)
+	*outstr = '\0';
+    return len;
+}
+
 /*
  * compile a chunk such as a literal string or a [...] followed
  * by a possible hash operator
@@ -1230,45 +1332,10 @@
 			/* Posix range. */
 			patparse += 2;
 			len = nptr - patparse;
-			if (!strncmp(patparse, "alpha", len))
-			    ch = PP_ALPHA;
-			else if (!strncmp(patparse, "alnum", len))
-			    ch = PP_ALNUM;
-			else if (!strncmp(patparse, "ascii", len))
-			    ch = PP_ASCII;
-			else if (!strncmp(patparse, "blank", len))
-			    ch = PP_BLANK;
-			else if (!strncmp(patparse, "cntrl", len))
-			    ch = PP_CNTRL;
-			else if (!strncmp(patparse, "digit", len))
-			    ch = PP_DIGIT;
-			else if (!strncmp(patparse, "graph", len))
-			    ch = PP_GRAPH;
-			else if (!strncmp(patparse, "lower", len))
-			    ch = PP_LOWER;
-			else if (!strncmp(patparse, "print", len))
-			    ch = PP_PRINT;
-			else if (!strncmp(patparse, "punct", len))
-			    ch = PP_PUNCT;
-			else if (!strncmp(patparse, "space", len))
-			    ch = PP_SPACE;
-			else if (!strncmp(patparse, "upper", len))
-			    ch = PP_UPPER;
-			else if (!strncmp(patparse, "xdigit", len))
-			    ch = PP_XDIGIT;
-			else if (!strncmp(patparse, "IDENT", len))
-			    ch = PP_IDENT;
-			else if (!strncmp(patparse, "IFS", len))
-			    ch = PP_IFS;
-			else if (!strncmp(patparse, "IFSSPACE", len))
-			    ch = PP_IFSSPACE;
-			else if (!strncmp(patparse, "WORD", len))
-			    ch = PP_WORD;
-			else
-			    ch = PP_UNKWN;
+			ch = range_type(patparse, len);
 			patparse = nptr + 2;
 			if (ch != PP_UNKWN)
-			    patadd(NULL, STOUC(Meta+ch), 1, PA_NOALIGN);
+			    patadd(NULL, STOUC(Meta) + ch, 1, PA_NOALIGN);
 			continue;
 		}
 		charstart = patparse;
@@ -1276,7 +1343,7 @@
 
 		if (*patparse == '-' && patparse[1] &&
 		    patparse[1] != Outbrack) {
-		    patadd(NULL, STOUC(Meta+PP_RANGE), 1, PA_NOALIGN);
+		    patadd(NULL, STOUC(Meta)+PP_RANGE, 1, PA_NOALIGN);
 		    if (itok(*charstart)) {
 			patadd(0, STOUC(ztokens[*charstart - Pound]), 1,
 			       PA_NOALIGN);
@@ -2369,19 +2436,19 @@
 		wchar_t cr = CHARREF(patinput, patinend);
 		char *scanop = (char *)P_OPERAND(scan);
 		if (patglobflags & GF_MULTIBYTE) {
-		    if (mb_patmatchrange(scanop, cr) ^
+		    if (mb_patmatchrange(scanop, cr, NULL, NULL) ^
 			(P_OP(scan) == P_ANYOF))
 			fail = 1;
 		    else
 			CHARINC(patinput, patinend);
-		} else if (patmatchrange(scanop, (int)cr) ^
+		} else if (patmatchrange(scanop, (int)cr, NULL, NULL) ^
 			   (P_OP(scan) == P_ANYOF))
 		    fail = 1;
 		else
 		    CHARINC(patinput, patinend);
 #else
 		if (patmatchrange((char *)P_OPERAND(scan),
-				   CHARREF(patinput, patinend)) ^
+				  CHARREF(patinput, patinend), NULL, NULL) ^
 		    (P_OP(scan) == P_ANYOF))
 		    fail = 1;
 		else
@@ -3122,12 +3189,33 @@
 /**/
 #ifdef MULTIBYTE_SUPPORT
 
+/*
+ * See if character ch matches a pattern range specification.
+ * The null-terminated specification is in range; the test
+ * character is in ch.
+ *
+ * indptr is used by completion matching, which is why this
+ * function is exported.  If indptr is not NULL we set *indptr
+ * to the index of the character in the range string, adjusted
+ * in the case of "A-B" ranges such that A would count as its
+ * normal index (say IA), B would count as IA + (B-A), and any
+ * character within the range as appropriate.  We're not strictly
+ * guaranteed this fits within a wint_t, but if this is Unicode
+ * in 32 bits we have a fair amount of distance left over.
+ *
+ * mtp is used in the same circumstances.  *mtp returns the match type:
+ * 0 for a standard character, else the PP_ index.  It's not
+ * useful if the match failed.
+ */
+
 /**/
-static int
-mb_patmatchrange(char *range, wchar_t ch)
+mod_export int
+mb_patmatchrange(char *range, wchar_t ch, wint_t *indptr, int *mtp)
 {
     wchar_t r1, r2;
 
+    if (indptr)
+	*indptr = 0;
     /*
      * Careful here: unlike other strings, range is a NULL-terminated,
      * metafied string, because we need to treat the Posix and hyphenated
@@ -3135,7 +3223,10 @@
      */
     while (*range) {
 	if (imeta(STOUC(*range))) {
-	    switch (STOUC(*range++) - STOUC(Meta)) {
+	    int swtype = STOUC(*range++) - STOUC(Meta);
+	    if (mtp)
+		*mtp = swtype;
+	    switch (swtype) {
 	    case 0:
 		/* ordinary metafied character */
 		range--;
@@ -3214,8 +3305,19 @@
 	    case PP_RANGE:
 		r1 = metacharinc(&range);
 		r2 = metacharinc(&range);
-		if (r1 <= ch && ch <= r2)
+		if (r1 <= ch && ch <= r2) {
+		    if (indptr)
+			*indptr += ch - r1;
 		    return 1;
+		}
+		/* Careful not to screw up counting with bogus range */
+		if (indptr && r1 < r2) {
+		    /*
+		     * This gets incremented again below to get
+		     * us past the range end.  This is correct.
+		     */
+		    *indptr += r2 - r1;
+		}
 		break;
 	    case PP_UNKWN:
 		DPUTS(1, "BUG: unknown posix range passed through.\n");
@@ -3224,21 +3326,130 @@
 		DPUTS(1, "BUG: unknown metacharacter in range.");
 		break;
 	    }
-	} else if (metacharinc(&range) == ch)
+	} else if (metacharinc(&range) == ch) {
+	    if (mtp)
+		*mtp = 0;
 	    return 1;
+	}
+	if (indptr)
+	    (*indptr)++;
     }
     return 0;
 }
 
+
+#if 0
+/*
+ * This is effectively the reverse of mb_patmatchrange().
+ * Given a range descriptor of the same form, and an index into it,
+ * try to determine the character that is matched.  If the index
+ * points to a [:...:] generic style match, set chr to WEOF and
+ * return the type in mtp instead.  Return 1 if successful, 0 if
+ * there was no corresponding index.  Note all pointer arguments
+ * must be non-null.
+ *
+ * TODO: for now the completion matching code does not handle
+ * multibyte.  When it does, we will need either this, or
+ * patmatchindex(), but not both---unlike user-initiated pattern
+ * matching, multibyte mode in the line editor is always on when available.
+ */
+
 /**/
+mod_export int
+mb_patmatchindex(char *range, wint_t ind, wint_t *chr, int *mtp)
+{
+    wchar_t r1, r2, rchr;
+    wint_t rdiff;
+
+    *chr = WEOF;
+    *mtp = 0;
+
+    while (*range) {
+	if (imeta(STOUC(*range))) {
+	    int swtype = STOUC(*range++) - STOUC(Meta);
+	    switch (swtype) {
+	    case 0:
+		range--;
+		rchr = metacharinc(&range);
+		if (!ind) {
+		    *chr = (wint_t) rchr;
+		    return 1;
+		}
+		break;
+
+	    case PP_ALPHA:
+	    case PP_ALNUM:
+	    case PP_ASCII:
+	    case PP_BLANK:
+	    case PP_CNTRL:
+	    case PP_DIGIT:
+	    case PP_GRAPH:
+	    case PP_LOWER:
+	    case PP_PRINT:
+	    case PP_PUNCT:
+	    case PP_SPACE:
+	    case PP_UPPER:
+	    case PP_XDIGIT:
+	    case PP_IDENT:
+	    case PP_IFS:
+	    case PP_IFSSPACE:
+	    case PP_WORD:
+		if (!ind) {
+		    *mtp = swtype;
+		    return 1;
+		}
+		break;
+
+	    case PP_RANGE:
+		r1 = metacharinc(&range);
+		r2 = metacharinc(&range);
+		rdiff = (wint_t)r2 - (wint_t)r1; 
+		if (rdiff >= ind) {
+		    *chr = (wint_t)r1 + ind;
+		    return 1;
+		}
+		/* note the extra decrement to ind below */
+		ind -= rdiff;
+		break;
+	    case PP_UNKWN:
+		DPUTS(1, "BUG: unknown posix range passed through.\n");
+		break;
+	    default:
+		DPUTS(1, "BUG: unknown metacharacter in range.");
+		break;
+	    }
+	} else {
+	    rchr = metacharinc(&range);
+	    if (!ind) {
+		*chr = (wint_t)rchr;
+		return 1;
+	    }
+	}
+	if (!ind--)
+	    break;
+    }
+
+    /* No corresponding index. */
+    return 0;
+}
 #endif
 
 /**/
-static int
-patmatchrange(char *range, int ch)
+#endif
+
+/*
+ * Identical function to mb_patmatchrange() above for single-byte
+ * characters.
+ */
+
+/**/
+mod_export int
+patmatchrange(char *range, int ch, int *indptr, int *mtp)
 {
     int r1, r2;
 
+    if (indptr)
+	*indptr = 0;
     /*
      * Careful here: unlike other strings, range is a NULL-terminated,
      * metafied string, because we need to treat the Posix and hyphenated
@@ -3246,7 +3457,10 @@
      */
     for (; *range; range++) {
 	if (imeta(STOUC(*range))) {
-	    switch (STOUC(*range)-STOUC(Meta)) {
+	    int swtype = STOUC(*range) - STOUC(Meta);
+	    if (mtp)
+		*mtp = swtype;
+	    switch (swtype) {
 	    case 0:
 		if (STOUC(*++range ^ 32) == ch)
 		    return 1;
@@ -3326,8 +3540,13 @@
 		r2 = STOUC(UNMETA(range));
 		if (*range == Meta)
 		    range++;
-		if (r1 <= ch && ch <= r2)
+		if (r1 <= ch && ch <= r2) {
+		    if (indptr)
+			*indptr += ch - r1;
 		    return 1;
+		}
+		if (indptr && r1 < r2)
+		    *indptr += r2 - r1;
 		break;
 	    case PP_UNKWN:
 		DPUTS(1, "BUG: unknown posix range passed through.\n");
@@ -3336,9 +3555,100 @@
 		DPUTS(1, "BUG: unknown metacharacter in range.");
 		break;
 	    }
-	} else if (STOUC(*range) == ch)
+	} else if (STOUC(*range) == ch) {
+	    if (mtp)
+		*mtp = 0;
 	    return 1;
+	}
+	if (indptr)
+	    (*indptr)++;
+    }
+    return 0;
+}
+
+/*
+ * Identical function to mb_patmatchindex() above for single-byte
+ * characters.  Here -1 represents a character that needs a special type.
+ */
+
+/**/
+mod_export int
+patmatchindex(char *range, int ind, int *chr, int *mtp)
+{
+    int r1, r2, rdiff, rchr;
+
+    *chr = -1;
+    *mtp = 0;
+
+    for (; *range; range++) {
+	if (imeta(STOUC(*range))) {
+	    int swtype = STOUC(*range) - STOUC(Meta);
+	    switch (swtype) {
+	    case 0:
+		/* ordinary metafied character */
+		rchr = STOUC(*++range) ^ 32;
+		if (!ind) {
+		    *chr = rchr;
+		    return 1;
+		}
+		break;
+
+	    case PP_ALPHA:
+	    case PP_ALNUM:
+	    case PP_ASCII:
+	    case PP_BLANK:
+	    case PP_CNTRL:
+	    case PP_DIGIT:
+	    case PP_GRAPH:
+	    case PP_LOWER:
+	    case PP_PRINT:
+	    case PP_PUNCT:
+	    case PP_SPACE:
+	    case PP_UPPER:
+	    case PP_XDIGIT:
+	    case PP_IDENT:
+	    case PP_IFS:
+	    case PP_IFSSPACE:
+	    case PP_WORD:
+		if (!ind) {
+		    *mtp = swtype;
+		    return 1;
+		}
+		break;
+
+	    case PP_RANGE:
+		range++;
+		r1 = STOUC(UNMETA(range));
+		METACHARINC(range);
+		r2 = STOUC(UNMETA(range));
+		if (*range == Meta)
+		    range++;
+		rdiff = r2 - r1; 
+		if (rdiff >= ind) {
+		    *chr = r1 + ind;
+		    return 1;
+		}
+		/* note the extra decrement to ind below */
+		ind -= rdiff;
+		break;
+	    case PP_UNKWN:
+		DPUTS(1, "BUG: unknown posix range passed through.\n");
+		break;
+	    default:
+		DPUTS(1, "BUG: unknown metacharacter in range.");
+		break;
+	    }
+	} else {
+	    if (!ind) {
+		*chr = STOUC(*range);
+		return 1;
+	    }
+	}
+	if (!ind--)
+	    break;
     }
+
+    /* No corresponding index. */
     return 0;
 }
 
@@ -3382,14 +3692,14 @@
 #ifdef MULTIBYTE_SUPPORT
 	    wchar_t cr = CHARREF(scan, patinend);
 	    if (patglobflags & GF_MULTIBYTE) {
-		if (mb_patmatchrange(opnd, cr) ^
+		if (mb_patmatchrange(opnd, cr, NULL, NULL) ^
 		    (P_OP(p) == P_ANYOF))
 		    break;
-	    } else if (patmatchrange(opnd, (int)cr) ^
+	    } else if (patmatchrange(opnd, (int)cr, NULL, NULL) ^
 		       (P_OP(p) == P_ANYOF))
 		break;
 #else
-	    if (patmatchrange(opnd, CHARREF(scan, patinend)) ^
+	    if (patmatchrange(opnd, CHARREF(scan, patinend), NULL, NULL) ^
 		(P_OP(p) == P_ANYOF))
 		break;
 #endif
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.136
diff -u -r1.136 zsh.h
--- Src/zsh.h	1 Jun 2008 17:58:42 -0000	1.136
+++ Src/zsh.h	7 Jun 2008 20:19:38 -0000
@@ -1307,6 +1307,48 @@
 #define PAT_HAS_EXCLUDP	0x0800	/* (internal): top-level path1~path2. */
 #define PAT_LCMATCHUC   0x1000  /* equivalent to setting (#l) */
 
+/*
+ * Special match types used in character classes.  These
+ * are represented as tokens, with Meta added.  The character
+ * class is represented as a metafied string, with only these
+ * tokens special.  Note that an active leading "!" or "^" for
+ * negation is not part of the string but is flagged in the
+ * surrounding context.
+ *
+ * These types are also used in character and equivalence classes
+ * in completion matching.
+ *
+ * This must be kept ordered by the array colon_stuffs in pattern.c.
+ */
+/* Special value for first definition */
+#define PP_FIRST  1
+/* POSIX-defined types:  [:alpha:] etc. */
+#define PP_ALPHA  1
+#define PP_ALNUM  2
+#define PP_ASCII  3
+#define PP_BLANK  4
+#define PP_CNTRL  5
+#define PP_DIGIT  6
+#define PP_GRAPH  7
+#define PP_LOWER  8
+#define PP_PRINT  9
+#define PP_PUNCT  10
+#define PP_SPACE  11
+#define PP_UPPER  12
+#define PP_XDIGIT 13
+/* Zsh additions:  [:IDENT:] etc. */
+#define PP_IDENT  14
+#define PP_IFS    15
+#define PP_IFSSPACE   16
+#define PP_WORD   17
+/* Special value for last definition */
+#define PP_LAST   17
+
+/* Unknown type.  Not used in a valid token. */
+#define PP_UNKWN  18
+/* Range: token followed by the (possibly multibyte) start and end */
+#define PP_RANGE  19
+
 /* Globbing flags: lower 8 bits gives approx count */
 #define GF_LCMATCHUC	0x0100
 #define GF_IGNCASE	0x0200
Index: Src/Zle/comp.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/comp.h,v
retrieving revision 1.17
diff -u -r1.17 comp.h
--- Src/Zle/comp.h	6 May 2008 16:01:19 -0000	1.17
+++ Src/Zle/comp.h	7 Jun 2008 20:19:38 -0000
@@ -162,12 +162,49 @@
 #define CMF_RIGHT 4
 #define CMF_INTER 8
 
+/*
+ * Types of cpattern structure.
+ * Note freecpattern() assumes any <= CPAT_EQUIV have string.
+ */
+enum {
+    CPAT_CCLASS,		/* [...]: ordinary character class */
+    CPAT_NCLASS,		/* [!...]: ordinary character class, negated */
+    CPAT_EQUIV,			/* {...}: equivalence class */
+    CPAT_ANY,			/* ?: any character */
+    CPAT_CHAR			/* Single character given explicitly */
+};
+
+/*
+ * A pattern element in a matcher specification.
+ * Unlike normal patterns this only presents one character in
+ * either the test completion or the word on the command line.
+ */
 struct cpattern {
     Cpattern next;		/* next sub-pattern */
-    unsigned char tab[256];	/* table of matched characters */
-    int equiv;			/* if this is a {...} class */
+    int tp;			/* type of object as above */
+    union {
+	char *str;		/* if a character class, the objects
+				 * in it in a similar form to normal
+				 * pattern matching (a metafied string
+				 * with tokens).
+				 * Note the allocated length may be longer
+				 * than the null-terminated string.
+				 */
+	int chr;		/* if a single character, it
+				 * TODO: eventually should be a
+				 * convchar_t.
+				 */
+    } u;
 };
 
+/*
+ * For now this just handles single-byte characters.
+ * TODO: this will change.
+ */
+#define PATMATCHRANGE(r, c, ip, mtp)	patmatchrange(r, c, ip, mtp)
+#define PATMATCHINDEX(r, i, cp, mtp)	patmatchindex(r, i, cp, mtp)
+#define CONVCAST(c)	(c)
+
 /* This is a special return value for parse_cmatcher(), *
  * signalling an error. */
 
Index: Src/Zle/complete.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/complete.c,v
retrieving revision 1.39
diff -u -r1.39 complete.c
--- Src/Zle/complete.c	6 Jul 2007 21:52:40 -0000	1.39
+++ Src/Zle/complete.c	7 Jun 2008 20:19:38 -0000
@@ -122,13 +122,15 @@
 
     while (p) {
 	n = p->next;
+	if (p->tp <= CPAT_EQUIV)
+	    free(p->u.str);
 	zfree(p, sizeof(struct cpattern));
 
 	p = n;
     }
 }
 
-/* Copy a completion matcher list. */
+/* Copy a completion matcher list into permanent storage. */
 
 /**/
 mod_export Cmatcher
@@ -157,22 +159,51 @@
     return r;
 }
 
+/*
+ * Copy a single entry in a matcher pattern.
+ * If useheap is 1, it comes from the heap.
+ */
+
+/**/
+mod_export Cpattern
+cp_cpattern_element(Cpattern o)
+{
+    Cpattern n = zalloc(sizeof(struct cpattern));
+
+    n->next = NULL;
+
+    n->tp = o->tp;
+    switch (o->tp)
+    {
+    case CPAT_CCLASS:
+    case CPAT_NCLASS:
+    case CPAT_EQUIV:
+	n->u.str = ztrdup(o->u.str);
+	break;
+
+    case CPAT_CHAR:
+	n->u.chr = o->u.chr;
+	break;
+
+    default:
+	/* just to keep compiler quiet */
+	break;
+    }
+
+    return n;
+}
+
 /* Copy a completion matcher pattern. */
 
 /**/
 static Cpattern
 cpcpattern(Cpattern o)
 {
-    Cpattern r = NULL, *p = &r, n;
+    Cpattern r = NULL, *p = &r;
 
     while (o) {
-	*p = n = (Cpattern) zalloc(sizeof(struct cpattern));
-
-	n->next = NULL;
-	memcpy(n->tab, o->tab, 256);
-	n->equiv = o->equiv;
-
-	p = &(n->next);
+	*p = cp_cpattern_element(o);
+	p = &((*p)->next);
 	o = o->next;
     }
     return r;
@@ -331,14 +362,26 @@
     return ret;
 }
 
-/* Parse a pattern for matcher control. */
+/*
+ * Parse a pattern for matcher control. 
+ * name is the name of the builtin from which this is called, for errors.
+ * *sp is the input string and will be updated to the end of the parsed
+ *   pattern.
+ * *lp will be set to the number of characters (possibly multibyte)
+ *   that the pattern will match.  This must be deterministic, given
+ *   the syntax allowed here.
+ * e, if non-zero, is the ASCII end character to match; if zero,
+ *   stop on a blank.
+ * *err is set to 1 to indicate an error, else to 0.
+ */
 
 /**/
 static Cpattern
 parse_pattern(char *name, char **sp, int *lp, char e, int *err)
 {
     Cpattern ret = NULL, r = NULL, n;
-    unsigned char *s = (unsigned char *) *sp;
+    char *s = *sp;
+    int inchar;
     int l = 0;
 
     *err = 0;
@@ -346,25 +389,18 @@
     while (*s && (e ? (*s != e) : !inblank(*s))) {
 	n = (Cpattern) hcalloc(sizeof(*n));
 	n->next = NULL;
-	n->equiv = 0;
 
-	if (*s == '[') {
-	    s = parse_class(n, s + 1, ']');
-	    if (!*s) {
-		*err = 1;
-		zwarnnam(name, "unterminated character class");
-		return NULL;
-	    }
-	} else if (*s == '{') {
-	    n->equiv = 1;
-	    s = parse_class(n, s + 1, '}');
+	if (*s == '[' || *s == '{') {
+	    s = parse_class(n, s);
 	    if (!*s) {
 		*err = 1;
 		zwarnnam(name, "unterminated character class");
 		return NULL;
 	    }
+	    s++;
 	} else if (*s == '?') {
-	    memset(n->tab, 1, 256);
+	    n->tp = CPAT_ANY;
+	    s++;
 	} else if (*s == '*' || *s == '(' || *s == ')' || *s == '=') {
 	    *err = 1;
 	    zwarnnam(name, "invalid pattern character `%c'", *s);
@@ -373,8 +409,13 @@
 	    if (*s == '\\' && s[1])
 		s++;
 
-	    memset(n->tab, 0, 256);
-	    n->tab[*s] = 1;
+	    if (*s == Meta)
+		inchar = STOUC(*++s) ^ 32;
+	    else
+		inchar = STOUC(*s);
+	    s++;
+	    n->tp = CPAT_CHAR;
+	    n->u.chr = inchar;
 	}
 	if (ret)
 	    r->next = n;
@@ -384,7 +425,6 @@
 	r = n;
 
 	l++;
-	s++;
     }
     *sp = (char *) s;
     *lp = l;
@@ -394,28 +434,86 @@
 /* Parse a character class for matcher control. */
 
 /**/
-static unsigned char *
-parse_class(Cpattern p, unsigned char *s, unsigned char e)
+static char *
+parse_class(Cpattern p, char *iptr)
 {
-    int n = 0, i = 1, j, eq = (e == '}'), k = 1;
-
-    if (!eq && (*s == '!' || *s == '^') && s[1] != e) { n = 1; s++; }
-
-    memset(p->tab, n, 256);
-
-    n = !n;
-    while (*s && (k || *s != e)) {
-	if (s[1] == '-' && s[2] && s[2] != e) {
-	    /* a run of characters */
-	    for (j = (int) *s; j <= (int) s[2]; j++)
-		p->tab[j] = (eq ? i++ : n);
+    int endchar, firsttime = 1;
+    char *optr, *nptr;
 
-	    s += 3;
+    if (*iptr++ == '[') {
+	endchar = ']';
+	/* TODO: surely [^]] is valid? */
+	if ((*iptr == '!' || *iptr == '^') && iptr[1] != ']') {
+	    p->tp = CPAT_NCLASS;
+	    iptr++;
 	} else
-	    p->tab[*s++] = (eq ? i++ : n);
-	k = 0;
+	    p->tp = CPAT_CCLASS;
+    } else {
+	endchar = '}';
+	p->tp = CPAT_EQUIV;
+    }
+
+    /* find end of class.  End character can appear literally first. */
+    for (optr = iptr; optr == iptr || *optr != endchar; optr++)
+	if (!*optr)
+	    return optr;
+    /*
+     * We can always fit the parsed class within the same length
+     * because of the tokenization (including a null byte).
+     *
+     * As the input string is metafied, but shouldn't contain shell
+     * tokens, we can just add our own tokens willy nilly.
+     */
+    optr = p->u.str = zalloc((optr-iptr) + 1);
+
+    while (firsttime || *iptr != endchar) {
+	int ch;
+
+	if (*iptr == '[' && iptr[1] == ':' &&
+	    (nptr = strchr((char *)iptr + 2, ':')) && nptr[1] == ']') {
+	    /* Range type */
+	    iptr += 2;
+	    ch = range_type((char *)iptr, nptr-iptr);
+	    iptr = nptr + 2;
+	    if (ch != PP_UNKWN)
+		*optr++ = STOUC(Meta) + ch;
+	} else {
+	    /* characters stay metafied */
+	    char *ptr1 = iptr;
+	    if (*iptr == Meta)
+		iptr++;
+	    iptr++;
+	    if (*iptr == '-' && iptr[1] && iptr[1] != endchar) {
+		/* a run of characters */
+		iptr++;
+		/* range token */
+		*optr++ = Meta + PP_RANGE;
+
+		/* start of range character */
+		if (*ptr1 == Meta) {
+		    *optr++ = Meta;
+		    *optr++ = ptr1[1] ^ 32;
+		} else
+		    *optr++ = *ptr1;
+
+		if (*iptr == Meta) {
+		    *optr++ = *iptr++;
+		    *optr++ = *iptr++;
+		} else
+		    *optr++ = *iptr++;
+	    } else {
+		if (*ptr1 == Meta) {
+		    *optr++ = Meta;
+		    *optr++ = ptr1[1] ^ 32;
+		} else
+		    *optr++ = *ptr1;
+	    }
+	}
+	firsttime = 0;
     }
-    return s;
+
+    *optr = '\0';
+    return iptr;
 }
 
 /**/
Index: Src/Zle/compmatch.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/compmatch.c,v
retrieving revision 1.53
diff -u -r1.53 compmatch.c
--- Src/Zle/compmatch.c	18 Jan 2008 16:41:36 -0000	1.53
+++ Src/Zle/compmatch.c	7 Jun 2008 20:19:39 -0000
@@ -30,37 +30,68 @@
 #include "complete.mdh"
 #include "compmatch.pro"
 
-/* This compares two cpattern lists and returns non-zero if they are
- * equal. */
+/*
+ * This compares two cpattern lists and returns non-zero if they are
+ * equal (N.B. opposite sense to usual *cmp()).
+ *
+ * The old version of this didn't worry about whether the lists
+ * were the same length.  This one does.  It's hard to see how
+ * that can be wrong even if it's unnecessary.
+ */
 
 /**/
 static int
-cmp_cpatterns(Cpattern a, Cpattern b)
+cpatterns_same(Cpattern a, Cpattern b)
 {
     while (a) {
-	if (a->equiv != b->equiv || memcmp(a->tab, b->tab, 256))
+	if (!b)
 	    return 0;
+	if (a->tp != b->tp)
+	    return 0;
+	switch (a->tp) {
+	case CPAT_CCLASS:
+	case CPAT_NCLASS:
+	case CPAT_EQUIV:
+	    /*
+	     * Patterns can actually match the same even if
+	     * the range strings don't compare differently, but
+	     * I don't think we need to handle that subtlety.
+	     */
+	    if (strcmp(a->u.str, b->u.str) != 0)
+		return 0;
+	    break;
+
+	case CPAT_CHAR:
+	    if (a->u.chr != b->u.chr)
+		return 0;
+	    break;
+
+	default:
+	    /* here to silence compiler */
+	    break;
+	}
+
 	a = a->next;
 	b = b->next;
     }
-    return 1;
+    return !b;
 }
 
 /* This compares two cmatchers and returns non-zero if they are equal. */
 
 /**/
 static int
-cmp_cmatchers(Cmatcher a, Cmatcher b)
+cmatchers_same(Cmatcher a, Cmatcher b)
 {
     return (a == b ||
 	    (a->flags == b->flags &&
 	     a->llen == b->llen && a->wlen == b->wlen &&
-	     (!a->llen || cmp_cpatterns(a->line, b->line)) &&
-	     (a->wlen <= 0 || cmp_cpatterns(a->word, b->word)) &&
+	     (!a->llen || cpatterns_same(a->line, b->line)) &&
+	     (a->wlen <= 0 || cpatterns_same(a->word, b->word)) &&
 	     (!(a->flags & (CMF_LEFT | CMF_RIGHT)) ||
 	      (a->lalen == b->lalen && a->ralen == b->ralen &&
-	       (!a->lalen || cmp_cpatterns(a->left, b->left)) &&
-	       (!a->ralen || cmp_cpatterns(a->right, b->right))))));
+	       (!a->lalen || cpatterns_same(a->left, b->left)) &&
+	       (!a->ralen || cpatterns_same(a->right, b->right))))));
 }
 
 /* Add the given matchers to the bmatcher list. */
@@ -97,7 +128,7 @@
 	t = 0;
 	for (ms = mstack; ms && !t; ms = ms->next)
 	    for (mp = ms->matcher; mp && !t; mp = mp->next)
-		t = cmp_cmatchers(mp, p->matcher);
+		t = cmatchers_same(mp, p->matcher);
 
 	p = p->next;
 	if (!t) {
@@ -449,7 +480,7 @@
     }
 }
 
-/* This tests if the string from the line l matches the word w. In bp
+/* This tests if the string from the line l matches the word w. In *bpp
  * the offset for the brace is returned, in rwlp the length of the
  * matched prefix or suffix, not including the stuff before or after
  * the last anchor is given. When sfx is non-zero matching is done from
@@ -1113,55 +1144,330 @@
     return r;
 }
 
-/* Check if the given pattern matches the given string.             *
+
+/*
+ * Guts of a single pattern for pattern_match().
+ * Return non-zero if match successful.
+ * If the class was an equivalence, return 1 + the index into
+ * the equivalence class (see pattern.c for how this is calculated).
+ */
+
+/**/
+mod_export int
+pattern_match1(Cpattern p, int c, int *mtp)
+{
+    /* TODO: should become convchar_t */
+    int ind;
+
+    *mtp = 0;
+    switch (p->tp) {
+    case CPAT_CCLASS:
+	return PATMATCHRANGE(p->u.str, CONVCAST(c), NULL, NULL);
+
+    case CPAT_NCLASS:
+	return !PATMATCHRANGE(p->u.str, CONVCAST(c), NULL, NULL);
+
+    case CPAT_EQUIV:
+	if (PATMATCHRANGE(p->u.str, CONVCAST(c), &ind, mtp))
+	    return ind + 1;
+	else
+	    return 0;
+
+    case CPAT_ANY:
+	return 1;
+
+    case CPAT_CHAR:
+	return (p->u.chr == c);
+
+    default:
+	DPUTS(1, "bad matcher pattern type");
+	return 0;
+    }
+}
+
+
+/*
+ * Use an equivalence to deduce the line character from the word, or
+ * vice versa.  (If vice versa, then "line" and "word" are reversed
+ * in what follows.  The logic is symmetric.)
+ * lp is the line pattern.
+ * wind is the index returned by a pattern match on the word pattern,
+ * with type wmtp.
+ * wchr is the word character.
+ * Return -1 if no matching character, else the character.
+ *
+ * Only makes sense if lp->tp == CPAT_EQUIV and the (unseen) word
+ * pattern also has that type.
+ */
+static int
+pattern_match_equivalence(Cpattern lp, int wind, int wmtp, int wchr)
+{
+    int lchr, lmtp;
+
+    if (!PATMATCHINDEX(lp->u.str, wind-1, &lchr, &lmtp)) {
+	/*
+	 * No equivalent.  No possible match; give up.
+	 */
+	return -1;
+    }
+    /*
+     * If we matched an exact character rather than a range
+     * type, return it.
+     */
+    if (lchr != -1)
+	return lchr;
+
+    /*
+     * Check the match types.  We may want a case-changed
+     * version of the word character.
+     */
+    if (wmtp == PP_UPPER && lmtp == PP_LOWER)
+	return tulower(wchr);
+    else if (wmtp == PP_LOWER && lmtp == PP_UPPER)
+	return tuupper(wchr);
+    else if (wmtp == lmtp) {
+	/*
+	 * Be lenient and allow identical replacements
+	 * for character classes, although in fact this
+	 * doesn't give special functionality for equivalence
+	 * classes.
+	 */
+	return wchr;
+    } else {
+	/*
+	 * Non-matching generic types; this can't work.
+	 */
+	return -1;
+    }
+}
+
+/*
+ * Check if the given pattern matches the given string.
  *  p and  s are either anchor or line pattern and string;
  * wp and ws are word (candidate) pattern and string
  *
- * If only one pattern is given, we just check if characters match
+ * If only one pattern is given, we just check if characters match.
  * If both line and word are given, we check that characters match
- * for {...} classes by comparing relative numbers in sequence.
+ * for {...} classes by comparing positions in the strings.
  *
  * Patterns and strings are always passed in pairs, so it is enough
  * to check for non-NULL wp. p should always be present.
+ *
+ * If prestrict is not NULL, it is a chain of patterns at least as long
+ * as the line string.  In this case we are still assembling the line at
+ * s (which has been allocated but doesn't yet contain anything useful)
+ * and must continue to do so as we go along; prestrict gives
+ * restrictions on the line character to be applied along side the other
+ * patterns.  In the simple case a restriction is a character to be put
+ * in place; otherwise it is a set of possible characters and we have to
+ * deduce an actual matching character.  Note prestrict is never an
+ * equivalence class.  In extreme cases we can't deduce a unique
+ * character; then the match fails.
  */
 
 /**/
 mod_export int
-pattern_match(Cpattern p, char *s, Cpattern wp, char *ws)
+pattern_match_restrict(Cpattern p, char *s, Cpattern wp, char *ws,
+		       Cpattern prestrict)
 {
-    unsigned char c;
-    unsigned char wc;
+    int c, ind;
+    int wc, wind;
+    int len, wlen, mt, wmt;
 
     while (p && wp && *s && *ws) {
-	c = p->tab[*((unsigned char *) s)];
-	wc = wp->tab[*((unsigned char *) ws)];
-
-	if (!c || !wc || c != wc)
+	/* First test the word character */
+	if (*ws == Meta) {
+	    wc = STOUC(ws[1]) ^ 32;
+	    wlen = 2;
+	} else {
+	    wc = STOUC(*ws);
+	    wlen = 1;
+	}
+	wind = pattern_match1(wp, wc, &wmt);
+	if (!wind)
 	    return 0;
 
-	s++;
-	ws++;
+	/*
+	 * Now the line character; deal with the case where
+	 * we don't yet have it, only a restriction on it.
+	 */
+	if (prestrict) {
+	    if (prestrict->tp == CPAT_CHAR) {
+		/*
+		 * Easy case: restricted to an exact character on
+		 * the line.  Procede as normal.
+		 */
+		c = prestrict->u.chr;
+	    } else {
+		if (p->tp == CPAT_CHAR) {
+		    /*
+		     * Normal line pattern is an exact character:  as
+		     * long as this matches prestrict, we can proceed
+		     * as usual.
+		     */
+		    c = p->u.chr;
+		} else if (p->tp == CPAT_EQUIV) {
+		    /*
+		     * An equivalence, so we can deduce the character
+		     * backwards from the word pattern and see if it
+		     * matches prestrict.
+		     */
+		    if ((c = pattern_match_equivalence(p, wind, wmt, wc)) == -1)
+			return 0;
+		} else {
+		    /*
+		     * Not an equivalence, so that means we must match
+		     * the word (not just the word pattern), so grab it
+		     * and make sure it fulfills our needs.  I think.
+		     * Not 100% sure about that, but what else can
+		     * we do?  We haven't actually been passed a string
+		     * from the command line.
+		     */
+		    c = wc;
+		}
+		/* Character so deduced must match the restriction. */
+		if (!pattern_match1(prestrict, c, &mt))
+		    return 0;
+	    }
+	    len = imeta(c) ? 2 : 1;
+	} else {
+	    /* We have the character itself. */
+	    if (*s == Meta) {
+		c = STOUC(s[1]) ^ 32;
+		len = 2;
+	    } else {
+		c = STOUC(*s);
+		len = 1;
+	    }
+	}
+	/*
+	 * If either is "?", they match each other; no further tests.
+	 * Apply this even if the character wasn't convertable;
+	 * there's no point trying to be clever in that case.
+	 */
+	if (p->tp != CPAT_ANY || wp->tp != CPAT_ANY)
+	{
+	    ind = pattern_match1(p, c, &mt);
+	    if (!ind)
+		return 0;
+	    if (ind != wind)
+		return 0;
+	    if (mt != wmt) {
+		/*
+		 * Special case if matching lower vs. upper or
+		 * vice versa.  The transformed characters must match.
+		 * We don't need to check the transformation is
+		 * the appropriate one for each character separately,
+		 * since that was done in pattern_match1(), so just
+		 * compare lower-cased versions of both.
+		 */
+		if ((mt == PP_LOWER || mt == PP_UPPER) &&
+		    (wmt == PP_LOWER || wmt == PP_UPPER)) {
+		    if (tulower(c) != tulower(wc))
+			return 0;
+		} else {
+		    /* Other different classes can't match. */
+		    return 0;
+		}
+	    }
+	}
+
+	if (prestrict) {
+	    /* We need to assemble the line */
+	    if (imeta(c)) {
+		*s++ = Meta;
+		*s++ = c ^ 32;
+	    } else {
+		*s++ = c;
+	    }
+	    prestrict = prestrict->next;
+	} else
+	    s += len;
+	ws += wlen;
 	p = p->next;
 	wp = wp->next;
     }
 
     while (p && *s) {
-	if (!p->tab[*((unsigned char *) s)])
+	if (prestrict) {
+	    /*
+	     * As above, but with even less info to go on.
+	     * (Can this happen?)  At least handle the cases where
+	     * one of our patterns has given us a specific character.
+	     */
+	    if (prestrict->tp == CPAT_CHAR) {
+		c = prestrict->u.chr;
+	    } else {
+		if (p->tp == CPAT_CHAR) {
+		    c = p->u.chr;
+		} else {
+		    /*
+		     * OK.  Here we are in a function with just a line
+		     * pattern and another pattern to restrict the
+		     * characters that can go on the line, and no actual
+		     * characters.  We're matching two patterns against
+		     * one another to generate a character to insert.
+		     * This is a bit too psychedelic, so I'm going to
+		     * bale out now.  See you on the ground.
+		     */
+		    return 0;
+		}
+		if (!pattern_match1(prestrict, c, &mt))
+		    return 0;
+	    }
+	} else {
+	    if (*s == Meta) {
+		c = STOUC(s[1]) ^ 32;
+		len = 2;
+	    } else {
+		c = STOUC(*s);
+		len = 1;
+	    }
+	}
+	if (!pattern_match1(p, c, &mt))
 	    return 0;
 	p = p->next;
-	s++;
+	if (prestrict) {
+	    if (imeta(c)) {
+		*s++ = Meta;
+		*s++ = c ^ 32;
+	    } else {
+		*s++ = c;
+	    }
+	    prestrict = prestrict->next;
+	} else
+	    s += len;
     }
 
     while (wp && *ws) {
-	if (!wp->tab[*((unsigned char *) ws)])
+	/* No funny business when we only have the word pattern. */
+	if (*ws == Meta) {
+	    wc = STOUC(ws[1]) ^ 32;
+	    wlen = 2;
+	} else {
+	    wc = STOUC(*ws);
+	    wlen = 1;
+	}
+	if (!pattern_match1(wp, wc, &wmt))
 	    return 0;
 	wp = wp->next;
-	ws++;
+	ws += wlen;
     }
 
     return 1;
 }
 
+/*
+ * The usual version of pattern matching, without the line string
+ * being handled by restriction.
+ */
+/**/
+mod_export int
+pattern_match(Cpattern p, char *s, Cpattern wp, char *ws)
+{
+    return pattern_match_restrict(p, s, wp, ws, NULL);
+}
+
 /* This splits the given string into a list of cline structs, separated
  * at those places where one of the anchors of an `*' pattern was found.
  * plen gives the number of characters on the line that matched this
@@ -1256,11 +1562,11 @@
     return ret;
 }
 
-/* This builds all the possible line patterns for the pattern pat in the
- * buffer line. Initially line is the same as lp, but during recursive
- * calls lp is incremented for storing successive characters. Whenever
- * a full possible string is build, we test if this line matches the
- * string given by wlen and word.
+
+/*
+ * This builds all the possible line patterns for the pattern pat in the
+ * buffer line.  Then we test if this line matches the string given by
+ * wlen and word.
  *
  * wpat contains pattern that matched previously
  * lpat contains the pattern for line we build
@@ -1269,91 +1575,297 @@
  *
  * The return value is the length of the string matched in the word, it
  * is zero if we couldn't build a line that matches the word.
+ *
+ * TODO: a lot of the nastiness associated with variable string
+ * lengths can go when we switch to wide characters.  (Why didn't
+ * I just keep line unmetafied and metafy into place at the end?  Er...)
  */
 
-
 /**/
 static int
-bld_line(Cpattern wpat, Cpattern lpat, char *line, char *lp,
-	 char *mword, char *word, int wlen, int sfx)
+bld_line(Cmatcher mp, char **linep, char *mword, char *word, int wlen, int sfx)
 {
-    if (lpat) {
-	/* Still working on the pattern. */
-
-	int i, l;
-	unsigned char c = 0;
+    Cpattern lpat = mp->line;
+    Cpattern wpat = mp->word;
+    Cpattern curgenpat;
+    VARARR(struct cpattern, genpatarr, mp->llen);
+    Cmlist ms;
+    int llen, rl;
+    char *oword = word, *line = *linep;
 
-	/* Get the number of the character for a correspondence class
-	 * if it has a corresponding class. */
-	if (lpat->equiv)
-	    if (wpat && *mword) {
-		c = wpat->tab[STOUC(*mword)];
-		wpat = wpat->next;
-		mword++;
+    /*
+     * Loop over all characters.  At this stage, line is an empty
+     * space of length llen (not counting the null byte) which we assemble as
+     * we go along.
+     *
+     * However, first we need to know what characters can appear at each
+     * point in the line.  For this we assemble an list genpatarr of the
+     * same length as the line.  (It's convenient to store this as an
+     * array but it's linked as a list, too.)  If there are equivalences
+     * we use mword to derive the equivalent character; when we've
+     * reached the end of mword, equivalences are treated just like
+     * ordinary character classes.  For character classes we just attach
+     * the class to the genpatarr list and apply it as a restriction
+     * when we finally match the line against the set of matchers.
+     */
+    curgenpat = genpatarr;
+    while (lpat) {
+	int wchr = (*mword == Meta) ? STOUC(mword[1]) ^ 32 : STOUC(*mword);
+	int wmtp, wind;
+	/*
+	 * If the line pattern is an equivalence, query wpat to find the
+	 * word part of the equivalence.  If we don't find one we don't try
+	 * equivalencing but use lpat as an ordinary match.  (It's not
+	 * entirely clear to me this is the correct behaviour on a
+	 * failed character match within the equivalence, but that was
+	 * the behaviour of the old logic that this replaces.)
+	 */
+	if (lpat->tp == CPAT_EQUIV && wpat && *mword) {
+	    wind = pattern_match1(wpat, wchr, &wmtp);
+	    wpat = wpat->next;
+	    mword += (*mword == Meta) ? 2 : 1;
+	} else
+	    wind = 0;
+	if (wind) {
+	    /*
+	     * Successful match for word side of equivalence.
+	     * Find the line equivalent.
+	     */
+	    int lchr;
+	    if ((lchr = pattern_match_equivalence(lpat, wind, wmtp, wchr))
+		== -1) {
+		/*
+		 * No equivalent.  No possible match; give up.
+		 */
+		return 0;
 	    }
+	    /*
+	     * We now have an exact character to match,
+	     * so make up a pattern element for it.
+	     */
+	    curgenpat->tp = CPAT_CHAR;
+	    curgenpat->u.chr = lchr;
+	} else {
+	    /*
+	     * Not an equivalence class, so we just keep the
+	     * test in the lpat as it is.
+	     */
+	    curgenpat->tp = lpat->tp;
+	    if (lpat->tp == CPAT_CHAR)
+		curgenpat->u.chr = lpat->u.chr;
+	    else if (lpat->tp != CPAT_ANY) {
+		/*
+		 * The string isn't modified and is only needed in calls from
+		 * this function, so we don't even need to copy it.
+		 */
+		curgenpat->u.str = lpat->u.str;
+	    }
+	}
+	lpat = lpat->next;
+	/*
+	 * This linked list is defined above as an array.
+	 * We could get away with just keeping it as an array
+	 * and passing it down as such, but that's a bit icky
+	 * since the generic linkage of Cpatterns is as a linked
+	 * list and we should keep our local memory management
+	 * problems to ourselvess.
+	 */
+	if (lpat)
+	    curgenpat->next = curgenpat+1;
+	else
+	    curgenpat->next = NULL;
+	curgenpat++;
+    }
 
+    /*
+     * We now know how to match the word with the line patterns; let's
+     * see if it does.  We will use the information in curgenpat if we
+     * are successful to work out what character goes on the line.  This
+     * is a bit hairy, as in "the Yeti is a creature that is a bit
+     * hairy".
+     */
+    llen = mp->llen;
+    rl = 0;
 
-	/* Walk through the table in the pattern and try the characters
-	 * that may appear in the current position. */
-	for (i = 0; i < 256; i++)
-	    if ((lpat->equiv && c) ? (c == lpat->tab[i]) : lpat->tab[i]) {
-		*lp = i;
-		/* We stored the character, now call ourselves to build
-		 * the rest. */
-		if ((l = bld_line(wpat, lpat->next, line, lp + 1,
-				  mword, word, wlen, sfx)))
-		    return l;
-	    }
-    } else {
-	/* We reached the end, i.e. the line string is fully build, now
-	 * see if it matches the given word. */
+    if (sfx)
+    {
+	/*
+	 * We need to work backwards from the end of both the
+	 * word and the line strings.
+	 *
+	 * Position at the end of the word by counting characters.
+	 */
+	int l = wlen;
+	while (l--)
+	    word += (*word == Meta) ? 2 : 1;
 
-	Cmlist ms;
-	Cmatcher mp;
-	int l = lp - line, t, rl = 0, ind, add;
+	/*
+	 * We construct the line from the end.  We've left
+	 * enough space for possible Meta's.
+	 */
+	line += 2 * llen;
+	*line = '\0';
+	curgenpat = genpatarr + llen;
+    } else
+	curgenpat = genpatarr;
 
-	/* Quick test if the strings are exactly the same. */
-	if (l == wlen && !strncmp(line, word, l))
-	    return l;
+    /* we now reuse mp, lpat, wpat for the global matchers */
+    while (llen && wlen) {
+	int wchr, wmtp;
+	char *wp;
+	Cpattern tmpgenpat;
 
 	if (sfx) {
-	    line = lp; word += wlen;
-	    ind = -1; add = -1;
-	} else {
-	    ind = 0; add = 1;
-	}
-	/* We loop through the whole line string built. */
-	while (l && wlen) {
-	    if (word[ind] == line[ind]) {
-		/* The same character in both strings, skip over. */
-		line += add; word += add;
-		l--; wlen--; rl++;
+	    if (word > oword + 1 && word[-2] == Meta)
+		wp = word - 2;
+	    else
+		wp = word - 1;
+	    curgenpat--;
+	} else
+	    wp = word;
+	if (*wp == Meta)
+	    wchr = STOUC(wp[1]) ^ 32;
+	else
+	    wchr = STOUC(*wp);
+	if (pattern_match1(curgenpat, wchr, &wmtp))
+	{
+	    int lchr;
+	    /*
+	     * We can match the line character directly with the word
+	     * character.  If the line character is a fixed one,
+	     * keep it, since we went to all that trouble above,
+	     * else if it's generic, keep the word character,
+	     * since we have no choice.
+	     */
+	    if (curgenpat->tp == CPAT_CHAR)
+		lchr = curgenpat->u.chr;
+	    else
+		lchr = wchr;
+	    if (imeta(lchr)) {
+		if (sfx)
+		    line -= 2;
+		line[0] = Meta;
+		line[1] = lchr ^ 32;
+		if (!sfx)
+		    line += 2;
 	    } else {
-		t = 0;
-		for (ms = bmatchers; ms && !t; ms = ms->next) {
-		    mp = ms->matcher;
-		    if (mp && !mp->flags && mp->wlen <= wlen && mp->llen <= l &&
-			pattern_match(mp->line, (sfx ? line - mp->llen : line),
-				      mp->word, (sfx ? word - mp->wlen : word))) {
-			/* Both the line and the word pattern matched,
-			 * now skip over the matched portions. */
+		if (sfx)
+		    line--;
+		line[0] = lchr;
+		if (!sfx)
+		    line++;
+	    }
+
+	    llen--;
+	    wlen--;
+	    rl++;
+
+	    if (sfx)
+		word = wp;
+	    else {
+		if (llen)
+		    curgenpat++;
+		word += (*word == Meta) ? 2 : 1;
+	    }
+	}
+	else
+	{
+	    char *lp;
+	    /*
+	     * Need to loop over pattern matchers.
+	     */
+	    for (ms = bmatchers; ms; ms = ms->next) {
+		mp = ms->matcher;
+		/*
+		 * This is the nightmare case: we have line and
+		 * and word matchers and some pattern which restricts
+		 * the value on the line without us knowing exactly
+		 * what it is.  Despatch to the special function
+		 * for that.
+		 */
+		if (mp && !mp->flags && mp->wlen <= wlen &&
+		    mp->llen <= llen)
+		{
+		    if (sfx) {
+			/*
+			 * We haven't assembled the line yet, and with
+			 * Meta characters we don't yet know the length.
+			 * We'll fix this up later.
+			 */
+			lp = line - 2 * mp->llen;
+		    } else
+			lp = line;
+		    wp = word;
+		    if (sfx) {
+			int l = mp->wlen;
+			while (l--) {
+			    if (wp > oword + 1 && wp[-2] == Meta)
+				wp -= 2;
+			    else
+				wp--;
+			}
+
+			tmpgenpat = curgenpat - mp->llen;
+		    } else
+			tmpgenpat = curgenpat;
+		    if (pattern_match_restrict(mp->line, lp,
+					       mp->word, wp, tmpgenpat)) {
+			/*
+			 * Matched: advance over as many characters
+			 * of the patterns and strings as
+			 * we've done matches.
+			 */
 			if (sfx) {
-			    line -= mp->llen; word -= mp->wlen;
+			    int imove = mp->llen, nchar;
+			    char *pmove = lp;
+			    word = wp;
+			    
+			    /* Close the gap we left in the line string */
+			    while (imove--)
+				pmove += (*pmove == Meta) ? 2 : 1;
+			    /* Number of bytes to move */
+			    nchar = (int)(pmove - lp);
+			    /* The size of the gap */
+			    imove = 2 * mp->llen - nchar;
+			    if (imove) {
+				lp = line - imove;
+				/* Moving up, so start at the top */
+				while (nchar--)
+				    *--line = *--lp;
+				/* line is at the start of the moved text */
+			    }
+
+			    curgenpat = tmpgenpat;
 			} else {
-			    line += mp->llen; word += mp->wlen;
+			    int cnt = mp->llen;
+			    while (cnt--) {
+				line += (*line == Meta) ? 2 : 1;
+			    }
+
+			    cnt = mp->wlen;
+			    while (cnt--)
+				word += (*word == Meta) ? 2 : 1;
+
+			    curgenpat += mp->llen;
 			}
-			l -= mp->llen; wlen -= mp->wlen; rl += mp->wlen;
-			t = 1;
+			llen -= mp->llen;
+			wlen -= mp->wlen;
+			rl += mp->wlen;
+			break;
 		    }
 		}
-		if (!t)
-		    /* Didn't match, give up. */
-		    return 0;
 	    }
+	    if (!ms)
+		return 0;	/* Didn't match, give up */
 	}
-	if (!l)
-	    /* Unmatched portion in the line built, return matched length. */
-	    return rl;
+    }
+    if (!llen) {
+	/* Unmatched portion in the line built, return matched length. */
+	if (sfx)
+	    *linep = line;
+	else
+	    *line = '\0';
+	return rl;
     }
     return 0;
 }
@@ -1386,8 +1898,10 @@
 		    if ((t = pattern_match(mp->word, sa, NULL, NULL)) ||
 			pattern_match(mp->word, sb, NULL, NULL)) {
 			/* It matched one of the strings, t says which one. */
-			VARARR(char, line, mp->llen + 1);
-			char **ap, **bp;
+			/* TODO: double to allow Meta, not necessary
+			   when properly unmetafied */
+			VARARR(char, linearr, 2*mp->llen + 1);
+			char **ap, **bp, *line = linearr;
 			int *alp, *blp;
 
 			if (t) {
@@ -1399,10 +1913,8 @@
 			}
 			/* Now try to build a string that matches the other
 			 * string. */
-			if ((bl = bld_line(mp->word, mp->line, line, line,
-					   *ap, *bp, *blp, 0))) {
+			if ((bl = bld_line(mp, &line, *ap, *bp, *blp, 0))) {
 			    /* Found one, put it into the return string. */
-			    line[mp->llen] = '\0';
 			    if (rr <= mp->llen) {
 				char *or = rs;
 
@@ -1444,7 +1956,11 @@
     return rs;
 }
 
-/* This compares the anchors stored in two top-level clines. */
+/*
+ * This compares the anchors stored in two top-level clines.
+ * It returns 1 if the anchors are the same, 2 if they are
+ * compatible (and have been combined in "o"), 0 otherwise.
+ */
 
 /**/
 static int
@@ -1591,9 +2107,11 @@
 				       NULL, NULL)) ||
 		     pattern_match(mp->word, nw - (sfx ? mp->wlen : 0),
 				   NULL, NULL))) {
-		    VARARR(char, line, mp->llen + 1);
+		    /* TODO: doubled to allow Meta, not necessary
+		     * when properly unmetafied */
+		    VARARR(char, linearr, 2*mp->llen + 1);
 		    int bl;
-		    char *mw;
+		    char *mw, *line = linearr;
 
 		    /* Then build all the possible lines and see
 		     * if one of them matches the other string. */
@@ -1602,11 +2120,10 @@
 		    else
 			mw = nw - (sfx ? mp->wlen : 0);
 
-		    if ((bl = bld_line(mp->word, mp->line, line, line,
-				       mw, (t ? nw : ow), (t ? nl : ol), sfx)))  {
+		    if ((bl = bld_line(mp, &line, mw, (t ? nw : ow),
+				       (t ? nl : ol), sfx)))  {
 			/* Yep, one of the lines matched the other
 			 * string. */
-			line[mp->llen] = '\0';
 
 			if (t) {
 			    ol = mp->wlen; nl = bl;
Index: Src/Zle/computil.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/computil.c,v
retrieving revision 1.106
diff -u -r1.106 computil.c
--- Src/Zle/computil.c	7 Mar 2008 23:06:42 -0000	1.106
+++ Src/Zle/computil.c	7 Jun 2008 20:19:40 -0000
@@ -3997,6 +3997,239 @@
     return (found ? ret : NULL);
 }
 
+
+/*
+ * This code constructs (from heap) and returns a string that
+ * corresponds to a series of matches; when compiled as a pattern, at
+ * each position it matches either the character from the string "add"
+ * or the corresponding single-character match from the set of matchers.
+ * To take a simple case, if add is "a" and the single matcher for the
+ * character position matches "[0-9]", the pattern returned is "[0-9a]".
+ * We take account of equivalences between the word and line, too.
+ *
+ * As there are virtually no comments in this file, I don't really
+ * know why we're doing this, but it's to do with a matcher which
+ * is passed as an argument to the utility compfiles -p/-P.
+ */
+static char *
+cfp_matcher_range(Cmatcher *ms, char *add)
+{
+    Cmatcher *mp, m;
+    int len = 0, mt;
+    char *ret = NULL, *p = NULL, *adds = add;
+
+    /*
+     * Do this twice:  once to work out the length of the
+     * string in len, the second time to build it in ret.
+     * This is probably worthwhile because otherwise memory
+     * management is difficult.
+     */
+    for (;;) {
+	for (mp = ms; *add; add++, mp++) {
+	    if (!(m = *mp)) {
+		/*
+		 * No matcher, so just match the character
+		 * itself.
+		 *
+		 * TODO: surely this needs quoting if it's a
+		 * metacharacter?
+		 */
+		if (ret) {
+		    if (imeta(*add)) {
+			*p++ = Meta;
+			*p++ = *add ^ 32;
+		    } else
+			*p++ = *add;
+		} else
+		    len += imeta(*add) ? 2 : 1;
+	    } else if (m->flags & CMF_RIGHT) {
+		/*
+		 * Right-anchored:  match anything followed
+		 * by the character itself.
+		 */
+		if (ret) {
+		    *p++ = '*';
+		    /* TODO: quote again? */
+		    if (imeta(*add)) {
+			*p++ = Meta;
+			*p++ = *add ^ 32;
+		    } else
+			*p++ = *add;
+		} else
+		    len += imeta(*add) ? 3 : 2;
+	    } else {
+		/* The usual set of matcher possibilities. */
+		int ind;
+		if (m->line->tp == CPAT_EQUIV &&
+		    m->word->tp == CPAT_EQUIV) {
+		    /*
+		     * Genuine equivalence.  Add the character to match
+		     * and the equivalent character from the word
+		     * pattern.
+		     *
+		     * TODO: we could be more careful here with special
+		     * cases as we are in the basic character class
+		     * code below.
+		     */
+		    if (ret) {
+			*p++ = '[';
+			if (imeta(*add)) {
+			    *p++ = Meta;
+			    *p++ = *add ^ 32;
+			} else
+			    *p++ = *add;
+		    } else
+			len += imeta(*add) ? 3 : 2;
+		    if (PATMATCHRANGE(m->line->u.str, CONVCAST(*add),
+				      &ind, &mt)) {
+			/*
+			 * Find the equivalent match for ind in the
+			 * word pattern.
+			 */
+			if ((ind = pattern_match_equivalence
+			     (m->word, ind, mt, CONVCAST(*add))) != -1) {
+			    if (ret) {
+				if (imeta(ind)) {
+				    *p++ = Meta;
+				    *p++ = ind ^ 32;
+				} else
+				    *p++ = ind;
+			    } else
+				len += imeta(ind) ? 2 : 1;
+			}
+		    }
+		    if (ret)
+			*p++ = ']';
+		    else
+			len++;
+		} else {
+		    int newlen, addadd;
+
+		    switch (m->word->tp) {
+		    case CPAT_NCLASS:
+			/*
+			 * TODO: the old logic implies that we need to
+			 * match *add, i.e. it should be deleted from
+			 * the set of character's we're not allowed to
+			 * match.  That's too much like hard work for
+			 * now.  Indeed, in general it's impossible
+			 * without trickery.  Consider *add == 'A',
+			 * range == "[^[:upper:]]": we would have to
+			 * resort to something like "(A|[^[:upper:]])";
+			 * and in an expression like that *add may or
+			 * may not need backslashing.  So we're deep
+			 * into see-if-we-can-get-away-without
+			 * territory.
+			 */
+			if (ret) {
+			    *p++ = '[';
+			    *p++ = '^';
+			} else
+			    len += 2;
+			/*
+			 * Convert the compiled range string back
+			 * to an ordinary string.
+			 */
+			newlen =
+			    pattern_range_to_string(m->word->u.str, p);
+			DPUTS(!newlen, "empty character range");
+			if (ret) {
+			    p += newlen;
+			    *p++ = ']';
+			} else
+			    len += newlen + 1;
+			break;
+			    
+		    case CPAT_CCLASS:
+			/*
+			 * If there is an equivalence only on one
+			 * side it's not equivalent to anything.
+			 * Treat it as an ordinary character class.
+			 */ 
+		    case CPAT_EQUIV:
+		    case CPAT_CHAR:
+			if (ret)
+			    *p++ = '[';
+			else
+			    len++;
+			/*
+			 * We needed to add *add specially only if
+			 * it is not covered by the range.  This
+			 * is necessary for correct syntax---consider
+			 * if *add is ] and ] is also the first
+			 * character in the range.
+			 */
+			addadd = !pattern_match1(m->word, CONVCAST(*add), &mt);
+			if (addadd && *add == ']') {
+			    if (ret)
+				*p++ = *add;
+			    else
+				len++;
+			}
+			if (m->word->tp == CPAT_CHAR) {
+			    /*
+			     * The matcher just matches a single
+			     * character, but we need to be able
+			     * to match *add, too, hence we do
+			     * this as a [...].
+			     */
+			    if (ret) {
+				if (imeta(m->word->u.chr)) {
+				    *p++ = Meta;
+				    *p++ = m->word->u.chr ^ 32;
+				} else
+				    *p++ = m->word->u.chr;
+			    } else
+				len += imeta(m->word->u.chr) ? 2 : 1;
+			} else {
+			    /*
+			     * Convert the compiled range string back
+			     * to an ordinary string.
+			     */
+			    newlen =
+				pattern_range_to_string(m->word->u.str, p);
+			    DPUTS(!newlen, "empty character range");
+			    if (ret)
+				p += newlen;
+			    else
+				len += newlen;
+			}
+			if (addadd && *add != ']') {
+			    if (ret) {
+				if (imeta(*add)) {
+				    *p++ = Meta;
+				    *p++ = *add ^ 32;
+				} else
+				    *p++ = *add;
+			    } else
+				len += imeta(*add) ? 2 : 1;
+			}
+			if (ret)
+			    *p++ = ']';
+			else
+			    len++;
+			break;
+
+		    case CPAT_ANY:
+			if (ret)
+			    *p++ = '?';
+			else
+			    len++;
+			break;
+		    }
+		}
+	    }
+	}
+	if (ret) {
+	    *p = '\0';
+	    return ret;
+	}
+	p = ret = zhalloc(len + 1);
+	add = adds;
+    }
+}
+
+
 static char *
 cfp_matcher_pats(char *matcher, char *add)
 {
@@ -4064,64 +4297,8 @@
 			break;
 		    }
 	}
-	if (*add) {
-	    char *ret = "", buf[259];
-
-	    for (mp = ms; *add; add++, mp++) {
-		if (!(m = *mp)) {
-		    buf[0] = *add;
-		    buf[1] = '\0';
-		} else if (m->flags & CMF_RIGHT) {
-		    buf[0] = '*';
-		    buf[1] = *add;
-		    buf[2] = '\0';
-		} else {
-		    unsigned char *t, c;
-		    char *p = buf;
-		    int i;
-
-		    for (i = 256, t = m->word->tab; i--; t++)
-			if (*t)
-			    break;
-		    if (i) {
-			t = m->word->tab;
-			*p++ = '[';
-			if (m->line->equiv && m->word->equiv) {
-			    *p++ = *add;
-			    c = m->line->tab[STOUC(*add)];
-			    for (i = 0; i < 256; i++)
-				if (m->word->tab[i] == c) {
-				    *p++ = (char) i;
-				    break;
-				}
-			} else {
-			    if (*add == ']' || t[STOUC(']')])
-				*p++ = ']';
-			    for (i = 0; i < 256; i++, t++)
-				if (*t && ((char) i) != *add &&
-				    i != ']' && i != '-' &&
-				    i != '^' && i != '!')
-				    *p++ = (char) i;
-			    *p++ = *add;
-			    t = m->word->tab;
-			    if (*add != '^' && t[STOUC('^')])
-				*p++ = '^';
-			    if (*add != '!' && t[STOUC('!')])
-				*p++ = '!';
-			    if (*add != '-' && t[STOUC('-')])
-				*p++ = '-';
-			}
-			*p++ = ']';
-			*p = '\0';
-		    } else {
-			*p = '?';
-			p[1] = '\0';
-		    }
-		}
-		ret = dyncat(ret, buf);
-	    }
-	    return ret;
-	}
+	if (*add)
+	    return cfp_matcher_range(ms, add);
     }
     return add;
 }


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: rewrite of completion matching

[Bart Schaefer]

Here it is a week later, and I'm finally getting a chance to really
pay attention to this patch.

All of the exposition sounds sensible so I'm diving right down into
the patch itself for comments/questions ... seems I don't have many.

On Jun 7,  9:34pm, Peter Stephenson wrote:
}
} +classes.  The only special behaviour in correspondence classes is if
} +the form on the left and the form on the right are each one of
} +tt([:upper:]), tt([:lower:]).  In these cases the
} +character in the word and the character on the line must be the same up
} +to a difference in case.  Hence to make any lower case character on the
} +line match the corresponding upper case character in the trial
} +completion you can use `tt(m:{[:lower:]}={[:upper:]})'.  Although the
} +matching system does not yet handle multibyte characters, this is likely
} +to be a future extension, at which point this syntax will handle
} +arbitrary alphabets ; hence this form, rather than the use of explicit
} +ranges, is thee recommended form.  In other cases
} +`tt([:)var(name)tt(:])' forms are allowed, but imply no special
} +constraint on the characters beyond that implied by the test itself.

I'm afraid I don't follow "imply no special constraint".  Suppose I
write m:{[:punct:]}={[:space:]} -- what have I just told zsh to do?
And why does that differ in other than the obvious way from what it
does when I tell it m:{[:lower:]}={[:upper:]} ?

I *think* you mean that m:{[:punct:]}={[:space:]} is "any single
punctuation character matches any single whitespace character,"
whereas m:{[:lower:]}={[:upper:]} is "each lower case character
matches the corresponding upper case character."  That is, the former
works like "sed 's/[[:punct:]]/[[:space:]]/g'" but the latter works
like "tr '[:lower:]' '[:upper:]'".

(As "tr" allows only lower/upper to be used like this, I guess that
makes sense.)

} +example(compadd -M 'L:|[nN][oO]= M:_= M:{[:up[er:]}={[:lower:]}' - \ 

Typo there s/p\[e/ppe/.

Re: PATCH: rewrite of completion matching

[Peter Stephenson]

On Mon, 16 Jun 2008 00:17:00 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I *think* you mean that m:{[:punct:]}={[:space:]} is "any single
> punctuation character matches any single whitespace character,"
> whereas m:{[:lower:]}={[:upper:]} is "each lower case character
> matches the corresponding upper case character."  That is, the former
> works like "sed 's/[[:punct:]]/[[:space:]]/g'" but the latter works
> like "tr '[:lower:]' '[:upper:]'".

Yes.  I missed a special case, in fact: if the classes are the same,
then the character is forced to be the same as well.  (If you don't want
that, obviously you should use ordinary character classes.)

Index: Doc/Zsh/compwid.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/compwid.yo,v
retrieving revision 1.41
diff -u -r1.41 compwid.yo
--- Doc/Zsh/compwid.yo	8 Jun 2008 17:53:55 -0000	1.41
+++ Doc/Zsh/compwid.yo	16 Jun 2008 12:52:25 -0000
@@ -944,8 +944,12 @@
 to be a future extension, at which point this syntax will handle
 arbitrary alphabets; hence this form, rather than the use of explicit
 ranges, is the recommended form.  In other cases
-`tt([:)var(name)tt(:])' forms are allowed, but imply no special
-constraint on the characters beyond that implied by the test itself.
+`tt([:)var(name)tt(:])' forms are allowed.  If the two forms on the left
+and right are the same, the characters must match exactly.  In remaining
+cases, the corresponding tests are applied to both characters, but they
+are not otherwise constrained; any matching character in one set goes
+with any matching character in the other set:  this is equivalent to the
+behaviour of ordinary character classes.
 
 The pattern var(tpat) may also be one or two stars, `tt(*)' or
 `tt(**)'. This means that the pattern on the command line can match

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: rewrite of completion matching

[Bart Schaefer]

On Jun 16,  1:54pm, Peter Stephenson wrote:
} Subject: Re: PATCH: rewrite of completion matching
}
} On Mon, 16 Jun 2008 00:17:00 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > I *think* you mean that m:{[:punct:]}={[:space:]} is "any single
} > punctuation character matches any single whitespace character,"
} > whereas m:{[:lower:]}={[:upper:]} is "each lower case character
} > matches the corresponding upper case character."  That is, the former
} > works like "sed 's/[[:punct:]]/[[:space:]]/g'" but the latter works
} > like "tr '[:lower:]' '[:upper:]'".
} 
} Yes.

OK, then what do each of the following mean, and how could I figure it
out from the documentation?

  m:{[:lower:][:digit:]}={[:upper:][:digit:]}
  m:{[:digit:][:lower:]}={[:upper:][:digit:]}
  m:{[:lower:]0-9}={[:upper:][:digit:]}
  m:{[:lower:][:digit:]}={[:alnum:]}
  m:{[:lower:]}={[:alpha:]}

} I missed a special case, in fact: if the classes are the same,
} then the character is forced to be the same as well.

So m:{[:alnum:]}={[:alnum:]} is effectively an expensive no-op?

Found another doc typo, by the way:

Index: Doc/Zsh/expn.yo
1426c1426
< item(tt([:lower:]))(l
---
> item(tt([:lower:]))(

Re: PATCH: rewrite of completion matching

[Peter Stephenson]

Bart Schaefer wrote:
> OK, then what do each of the following mean, and how could I figure it
> out from the documentation?

Well, by reading it, obviously.  Here's what I think it says, please tell
me if you think it says something different and what I should do about
it since there's no point in me simply playing guessing games about what
you think it should say.

- [:lower:] matches [:upper:] explicitly for one pair of characters (a/A)
  and vice versa.  This is the only really useful case for
  correspondence classes.  (Consequently, quite possibly this is the only one
  worth mentioning under the heading of correspondence classes to
  avoid complicating the issue.)
- [:ANY:] matches [:ANY:] explicitly character by character (X matches X for
  any X that matches [:ANY:])
- Anything else is no different from character classes on the
  appropriate side.

The only particular caes are:

>   m:{[:lower:]0-9}={[:upper:][:digit:]}

0-9 vs. [:digit:] isn't documented as working, so avoid.  I think I know
what it does but it doesn't appear useful and I don't want people using
it.

>   m:{[:lower:][:digit:]}={[:alnum:]}

The hanging [:digit:] is treated as an ordinary character class; that's
already documented for bits left over in correspondence classes.

> } I missed a special case, in fact: if the classes are the same,
> } then the character is forced to be the same as well.
> 
> So m:{[:alnum:]}={[:alnum:]} is effectively an expensive no-op?

In that it doesn't give you behaviour beyond what completion would do
with a match specifier, yes.  Although

m:{[:alnum:][:upper:]}={[:alnum:][:lower:]}

is of very minor interest (match lower against upper provided some
alphanumeric, not specified but forced to be the same on both sides,
occurred in the previous character), and it's hard to think of good use
in practice.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: rewrite of completion matching

[Bart Schaefer]

On Jun 16,  5:52pm, Peter Stephenson wrote:
} Subject: Re: PATCH: rewrite of completion matching
}
} Bart Schaefer wrote:
} > OK, then what do each of the following mean, and how could I figure it
} > out from the documentation?
} 
} Well, by reading it, obviously.

Touche.

Skipping ahead, this is one bit that I wouldn't have understood from the
documentation and that I'm not sure is correct:

} m:{[:alnum:][:upper:]}={[:alnum:][:lower:]}
} 
} is of very minor interest (match lower against upper provided some
} alphanumeric, not specified but forced to be the same on both sides,
} occurred in the previous character), and it's hard to think of good use
} in practice.

Really "occurred in the previous character"?  If one had written

  m:{a-zA-Z0-9A-Z}={A-Za-z0-9a-z}

That would not have implied two consecutive characters.  The entire class
on the left corresponds to the entire class on the right and is applied
to each character independent of the ones around it.  If what you said
is true, you've introduced something completely new.  So I'd have had
the expectation that m:{[:alnum:][:upper:]}={[:alnum:][:lower:]} is at
best a long way to write m:{[:upper:]}={[:lower:]} and at worst would
be the same as m:{[:alnum:]}={[:alnum:]}.

I'm going to have to reply to the rest of this later, out of time ...

Re: PATCH: rewrite of completion matching

[Peter Stephenson]

On Mon, 16 Jun 2008 10:49:06 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Skipping ahead, this is one bit that I wouldn't have understood from the
> documentation and that I'm not sure is correct:
> 
> } m:{[:alnum:][:upper:]}={[:alnum:][:lower:]}
> } 
> } is of very minor interest (match lower against upper provided some
> } alphanumeric, not specified but forced to be the same on both sides,
> } occurred in the previous character), and it's hard to think of good use
> } in practice.
> 
> Really "occurred in the previous character"?

No, I realised eventually that was nonsense.  That's not how equivalences
work.  This would say either an alphanumeric matches itself, or an upper
matches a lower.  I can't think of a good reason why you'd want to express
that in an equivalence class.

I think the doc may want changing to be clearer that [:stuff:] in
equivalence classes is of limited scope, but is much more widely useful
in ordinary character classes.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: PATCH: completion of glob qualifiers

[Oliver Kiddle]

On 23 Feb, Peter Stephenson wrote:
> This is mostly useful as a memory jogger rather than actually to save
> you typing a single character.  However, I've added a pretty useless
> function to save you typing a delimiter character.  (The alternative
> was simply to print a message saying "delimiter", which was
> really low tech.)
> 
> It works OK so far but it does raise various issues about related things
> that might be got to work better.  You might run across those.

I've only just noticed this change back in message 24585. Is there any
reason why globbing flags were not completed. This patch adds them.

Note that inside [[ ... ]] conditions, a bracket that is not preceded by
some other character is assumed to be used for grouping of conditions so
this won't work with glob flags at the beginning of a pattern.

One problem possibly due to your original patch is the following;
: */( <tab>
results in the /( being removed. I haven't worked out why but this
doesn't occur with the old zsh that comes with Debian stable.

I've also changed _globquals to use _message -e instead of plain
_message. This applies a tag and allows certain styles (such as fake) to
be used.

Oliver

Index: Completion/Unix/Type/_path_files
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Type/_path_files,v
retrieving revision 1.39
diff -u -r1.39 _path_files
--- Completion/Unix/Type/_path_files	29 Oct 2008 21:59:32 -0000	1.39
+++ Completion/Unix/Type/_path_files	7 Nov 2008 12:34:34 -0000
@@ -394,7 +394,11 @@
 		"$tpre/$tsuf" = (#b)((*[^\\]|)(\\\\)#"(#q")([^\)]#) \
 	  ) && -z $compstate[quote] ]]; then
        compset -p ${#match[1]}
-       _globquals
+       if [[ -o extendedglob ]] && compset -P '\#'; then
+         _globflags
+       else
+         _globquals
+       fi
     elif [[ "$tpre$tsuf" = */* ]]; then
       compfiles -P$cfopt tmp1 accex "$skipped" "$_matcher $matcher[2]" "$sdirs" fake
     elif [[ "$sopt" = *[/f]* ]]; then
Index: Completion/Zsh/Type/.distfiles
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Zsh/Type/.distfiles,v
retrieving revision 1.3
diff -u -r1.3 .distfiles
--- Completion/Zsh/Type/.distfiles	15 Mar 2008 23:21:02 -0000	1.3
+++ Completion/Zsh/Type/.distfiles	7 Nov 2008 12:34:34 -0000
@@ -7,6 +7,7 @@
 _directory_stack
 _file_descriptors
 _functions
+_globflags
 _globqual_delims
 _globquals
 _history_modifiers
Index: Completion/Zsh/Type/_globflags
===================================================================
RCS file: Completion/Zsh/Type/_globflags
diff -N Completion/Zsh/Type/_globflags
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Completion/Zsh/Type/_globflags	7 Nov 2008 12:34:34 -0000
@@ -0,0 +1,33 @@
+local ret=1
+local -a flags
+
+if compset -P a; then
+  _message -e numbers 'errors'
+  return
+elif compset -P q; then
+  _globquals
+  return
+fi
+
+flags=(
+  'i:case insensitive'
+  'l:lower case characters match uppercase'
+  'I:case sensitive matching'
+  's:match start of string'
+  'e:match end of string'
+)
+[[ $compstate[context] = condition ]] && flags+=(
+  'b:activate backreferences'
+  'B:deactivate backreferences'
+  'm:set reference to entire matched data'
+  'M:deactivate m flag'
+)
+
+_describe -t globflags "glob flag" flags -Q -S ')' && ret=0
+flags=(
+  'a:approximate matching'
+  'q:introduce glob qualifier'
+)
+_describe -t globflags "glob flag" flags -Q -S '' && ret=0
+
+return ret
Index: Completion/Zsh/Type/_globquals
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Zsh/Type/_globquals,v
retrieving revision 1.2
diff -u -r1.2 _globquals
--- Completion/Zsh/Type/_globquals	23 Feb 2008 18:34:02 -0000	1.2
+++ Completion/Zsh/Type/_globquals	7 Nov 2008 12:34:34 -0000
@@ -25,7 +25,7 @@
 	return
       elif ! _globqual_delims; then
 	# still completing mode spec
-	_message "mode spec"
+	_message -e modes "mode spec"
 	return
       fi
     fi
@@ -57,7 +57,7 @@
     (d)
     # complete/skip device
     if ! compset -p '[[:digit:]]##'; then
-      _message "device ID"
+      _message -e device-ids "device ID"
       return
     fi
     ;;
@@ -65,7 +65,7 @@
     (l)
     # complete/skip link count
     if ! compset -P '([-+]|)[[:digit:]]##'; then
-      _message "link count"
+      _message -e numbers "link count"
       return
     fi
     ;;

Re: PATCH: completion of glob qualifiers

[Peter Stephenson]

On Fri, 07 Nov 2008 13:38:56 +0100
Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> I've only just noticed this change back in message 24585. Is there any
> reason why globbing flags were not completed. This patch adds them.

Fine, the patch was simply incomplete.  The patch below will mess
yours up, however.

> Note that inside [[ ... ]] conditions, a bracket that is not preceded by
> some other character is assumed to be used for grouping of conditions so
> this won't work with glob flags at the beginning of a pattern.

We shouldn't complete glob flags right at the start of a word,
presumably---I don't think that's ever going to be useful.  I've
added a test that we're skipping over at least one character before the
parenthesis for a bare glob.

It seems the completion options always add BARE_GLOB_QUAL, in fact; it
would probably make more sense to rely on the use of (#q...) inside the
completion system, but I'm too lazy to fix this.

> One problem possibly due to your original patch is the following;
> : */( <tab>
> results in the /( being removed. I haven't worked out why but this
> doesn't occur with the old zsh that comes with Debian stable.

I'm not sure why I thought it was a good idea to match glob qualifiers
at a complicated point in the middle of path matching.  Even if the
qualifiers aren't after a glob expression, which they typically are, you
don't want to do completion on the earlier path if you're already
completing a qualifier.  I've therefore simply moved the chunk to the
start of _path_files.

Index: Completion/Unix/Type/_path_files
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Type/_path_files,v
retrieving revision 1.39
diff -u -r1.39 _path_files
--- Completion/Unix/Type/_path_files	29 Oct 2008 21:59:32 -0000	1.39
+++ Completion/Unix/Type/_path_files	8 Nov 2008 23:13:30 -0000
@@ -1,5 +1,30 @@
 #autoload
 
+local -a match mbegin mend
+
+# Look for glob qualifiers.  Do this first:  if we're really
+# in a glob qualifier, we don't actually want to expand
+# the earlier part of the path.  We can't expand inside
+# parentheses otherwise, so as we test that successfully
+# we should be able to commit to glob qualifiers here.
+#
+# Extra nastiness to be careful about a quoted parenthesis.
+# The initial tests look for parentheses with zero or an
+# even number of backslashes in front.  We also require that
+# there was at least one character before the parenthesis for
+# a bare glob qualifier.
+# The later test looks for an outstanding quote.
+if [[ ( -o bareglobqual && \
+           $PREFIX = (#b)((*[^\\]|)(\\\\)#\()([^\)]#) && \
+           ${#match[1]} -gt 1 || \
+        -o extendedglob && \
+	   $PREFIX = (#b)((*[^\\]|)(\\\\)#"(#q")([^\)]#) \
+      ) && -z $compstate[quote] ]]; then
+   compset -p ${#match[1]}
+   _globquals
+   return
+fi
+
 # Utility function for in-path completion. This allows `/u/l/b<TAB>'
 # to complete to `/usr/local/bin'.
 
@@ -9,7 +34,7 @@
 local nm=$compstate[nmatches] menu matcher mopts sort mid accex fake
 local listfiles listopts tmpdisp origtmp1 Uopt
 integer npathcheck
-local -a match mbegin mend Mopts
+local -a Mopts
 
 typeset -U prepaths exppaths
 
@@ -383,19 +408,7 @@
 
     tmp2=( "$tmp1[@]" )
 
-    # Look for glob qualifiers.
-    # Extra nastiness to be careful about a quoted parenthesis.
-    # The initial tests look for parentheses with zero or an
-    # even number of backslashes in front.
-    # The later test looks for an outstanding quote.
-    if [[ ( -o bareglobqual && \
-	      "$tpre/$tsuf" = (#b)((*[^\\]|)(\\\\)#\()([^\)]#) || \
-            -o extendedglob && \
-		"$tpre/$tsuf" = (#b)((*[^\\]|)(\\\\)#"(#q")([^\)]#) \
-	  ) && -z $compstate[quote] ]]; then
-       compset -p ${#match[1]}
-       _globquals
-    elif [[ "$tpre$tsuf" = */* ]]; then
+    if [[ "$tpre$tsuf" = */* ]]; then
       compfiles -P$cfopt tmp1 accex "$skipped" "$_matcher $matcher[2]" "$sdirs" fake
     elif [[ "$sopt" = *[/f]* ]]; then
       compfiles -p$cfopt tmp1 accex "$skipped" "$_matcher $matcher[2]" "$sdirs" fake "$pats[@]"

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: parse error in process substitution

[Peter Stephenson]

On Mon, 10 Nov 2008 14:28:50 +0000
Peter Stephenson <pws@csr.com> wrote:
> On Mon, 10 Nov 2008 09:21:52 +0100
> Louis-David Mitterrand <vindex+lists-zsh-users@apartia.org> wrote:
> > So how would you convert that working bash command to zsh?
> > 
> > 	root-tail <(ssh root@my.host.name tail -F /var/log/kern.log),red,
>...
> It would be good to fix the underlying problem; it will work a little
> differently from the way it is now, but that's so confusing if the <(...)
> isn't in a separate word that I don't think there's a good reason to keep
> it.

(Moved to zsh-workers.)

I'm working on that elsewhere.  Just to make quite sure I don't screw up
parsing of ordinary redirections in the process here's an extra test.
The fiddly bit is ensuring that normal redirection operators don't
require a space before them but are still parsed as a separate word.

Index: Test/A04redirect.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A04redirect.ztst,v
retrieving revision 1.13
diff -u -r1.13 A04redirect.ztst
--- Test/A04redirect.ztst	14 Mar 2008 11:40:59 -0000	1.13
+++ Test/A04redirect.ztst	12 Nov 2008 10:48:03 -0000
@@ -344,3 +344,21 @@
 0:Optimised here-string to filename
 >This string has been replaced
 >by a file containing it.
+
+  print This f$'\x69'le contains d$'\x61'ta. >redirfile
+  print redirection:
+  cat<redirfile>outfile
+  print output:
+  cat outfile
+  print append:
+  cat>>outfile<redirfile
+  print more output:
+  cat outfile
+0:Parsing of redirection operators (no space required before operators)
+>redirection:
+>output:
+>This file contains data.
+>append:
+>more output:
+>This file contains data.
+>This file contains data.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: parse error in process substitution

[Peter Stephenson]

On Mon, 10 Nov 2008 09:21:52 +0100
Louis-David Mitterrand <vindex+lists-zsh-users@apartia.org> wrote:
> So how would you convert that working bash command to zsh?
> 
> 	root-tail <(ssh root@my.host.name tail -F /var/log/kern.log),red,

This fixes the syntax.  Unfortunately, I had to impose some limitations
to prevent existing stuff stopping working.  Obviously I'd like to
know about anything else this messes up.  I'm not worried about
the undocumented feature that <(...) forced a new word.

A non-initial = usually isn't handled specially, and raw parentheses
(i.e. without a disambiguating character in front) are somewhat
overworked in zsh, so I couldn't make =(...) special anywhere except at
the start of an argument.  However, you can now put other things after
it.

">" and "<" never used to need quoting when used inside parentheses or
parameter substitutions, so to keep this I've had to forbid the use of
<(...) and >(...) in such places.  (The comptest function falls over if
this isn't done.)  This isn't likely to be a big problem
in practice.  Because of the previous rule this isn't relevant to
=(...).

Parsing of the contents of these expressions is done rather more simply
than for $(...) expressions.  I don't really know if there's any mileage
in making parsing of process substitutions more similar (inside the
parentheses only, the context dependence will have to remain
different).

Index: README
===================================================================
RCS file: /cvsroot/zsh/zsh/README,v
retrieving revision 1.58
diff -u -r1.58 README
--- README	30 Oct 2008 12:18:54 -0000	1.58
+++ README	13 Nov 2008 20:46:40 -0000
@@ -69,6 +69,16 @@
 consistent with recent versions of other shells.  The option
 DEBUG_BEFORE_CMD can be unset to revert to the previous behaviour.
 
+Previously, process substitutions of the form =(...), <(...) and
+>(...) were only handled if they appeared as separate command arguments,
+although the latter two forms caused a new command argument to be
+started at that point.  Now all three may be followed by other strings,
+and the latter two may also be preceeded by other string.  None may
+occur inside parameter substitutions, or inside parentheses used for
+grouping of patterns, in order to avoid clashes with cases where
+tt(<) or tt(>) where not treated specially in previous versions of the
+shell.
+
 In previous versions of the shell it was possible to use index 0 in an
 array or string subscript to refer to the same element as index 1 if the
 option KSH_ARRAYS was not in effect.  This was a limited approximation to
Index: Doc/Zsh/expn.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/expn.yo,v
retrieving revision 1.97
diff -u -r1.97 expn.yo
--- Doc/Zsh/expn.yo	26 Oct 2008 17:57:13 -0000	1.97
+++ Doc/Zsh/expn.yo	13 Nov 2008 20:46:41 -0000
@@ -353,11 +353,17 @@
 sect(Process Substitution)
 cindex(process substitution)
 cindex(substitution, process)
-Each command argument of the form
+Each part of a command argument that takes the form
 `tt(<LPAR())var(list)tt(RPAR())',
 `tt(>LPAR())var(list)tt(RPAR())' or
 `tt(=LPAR())var(list)tt(RPAR())'
-is subject to process substitution.
+is subject to process substitution.  The expression may be proceeded
+or followed by other strings except that, to prevent clashes with
+commonly occurring strings and patterns, the last
+form must occur at the start of a command argument, and none of
+the forms may occur inside parentheses used for grouping of patterns or
+inside parameter substitutions.
+
 In the case of the tt(<) or tt(>) forms, the shell runs process
 var(list) asynchronously.  If the system supports the tt(/dev/fd)
 mechanism, the command argument is the name of the device file
Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.160
diff -u -r1.160 exec.c
--- Src/exec.c	10 Oct 2008 09:55:34 -0000	1.160
+++ Src/exec.c	13 Nov 2008 20:46:43 -0000
@@ -3560,7 +3560,7 @@
 
 /**/
 static Eprog
-parsecmd(char *cmd)
+parsecmd(char *cmd, char **eptr)
 {
     char *str;
     Eprog prog;
@@ -3571,7 +3571,9 @@
 	return NULL;
     }
     *str = '\0';
-    if (str[1] || !(prog = parse_string(cmd + 2, 0))) {
+    if (eptr)
+	*eptr = str+1;
+    if (!(prog = parse_string(cmd + 2, 0))) {
 	zerr("parse error in process substitution");
 	return NULL;
     }
@@ -3582,7 +3584,7 @@
 
 /**/
 char *
-getoutputfile(char *cmd)
+getoutputfile(char *cmd, char **eptr)
 {
     pid_t pid;
     char *nam;
@@ -3592,7 +3594,7 @@
 
     if (thisjob == -1)
 	return NULL;
-    if (!(prog = parsecmd(cmd)))
+    if (!(prog = parsecmd(cmd, eptr)))
 	return NULL;
     if (!(nam = gettempname(NULL, 0)))
 	return NULL;
@@ -3677,7 +3679,7 @@
 
 /**/
 char *
-getproc(char *cmd)
+getproc(char *cmd, char **eptr)
 {
 #if !defined(HAVE_FIFOS) && !defined(PATH_DEV_FD)
     zerr("doesn't look like your system supports FIFOs.");
@@ -3696,7 +3698,7 @@
 	return NULL;
     if (!(pnam = namedpipe()))
 	return NULL;
-    if (!(prog = parsecmd(cmd)))
+    if (!(prog = parsecmd(cmd, eptr)))
 	return NULL;
     if (!jobtab[thisjob].filelist)
 	jobtab[thisjob].filelist = znewlinklist();
@@ -3723,7 +3725,7 @@
     if (thisjob == -1)
 	return NULL;
     pnam = hcalloc(strlen(PATH_DEV_FD) + 6);
-    if (!(prog = parsecmd(cmd)))
+    if (!(prog = parsecmd(cmd, eptr)))
 	return NULL;
     mpipe(pipes);
     if ((pid = zfork(&bgtime))) {
@@ -3772,7 +3774,7 @@
     pid_t pid;
     struct timeval bgtime;
 
-    if (!(prog = parsecmd(cmd)))
+    if (!(prog = parsecmd(cmd, NULL)))
 	return -1;
     mpipe(pipes);
     if ((pid = zfork(&bgtime))) {
Index: Src/lex.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/lex.c,v
retrieving revision 1.46
diff -u -r1.46 lex.c
--- Src/lex.c	31 Aug 2008 19:50:49 -0000	1.46
+++ Src/lex.c	13 Nov 2008 20:46:44 -0000
@@ -835,7 +835,7 @@
 	return OUTPAR;
     case LX1_INANG:
 	d = hgetc();
-	if (!incmdpos && d == '(') {
+	if (d == '(') {
 	    hungetc(d);
 	    lexstop = 0;
 	    unpeekfd:
@@ -1152,20 +1152,13 @@
 		c = Comma;
 	    break;
 	case LX2_OUTANG:
-	    if (!intpos) {
-		if (in_brace_param || sub)
-		    break;
-		else
-		    goto brk;
-	    }
+	    if (in_brace_param || sub)
+		break;
 	    e = hgetc();
 	    if (e != '(') {
 		hungetc(e);
 		lexstop = 0;
-		if (in_brace_param || sub)
-		    break;
-		else
-		    goto brk;
+		goto brk;
 	    }
 	    add(Outang);
 	    if (skipcomm()) {
@@ -1178,7 +1171,7 @@
 	    if (isset(SHGLOB) && sub)
 		break;
 	    e = hgetc();
-	    if(e == '(' && intpos) {
+	    if (!(in_brace_param || sub) && e == '(') {
 		add(Inang);
 		if (skipcomm()) {
 		    peek = LEXERR;
Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.90
diff -u -r1.90 subst.c
--- Src/subst.c	30 Oct 2008 15:34:18 -0000	1.90
+++ Src/subst.c	13 Nov 2008 20:46:45 -0000
@@ -56,43 +56,27 @@
 
     queue_signals();
     for (node = firstnode(list); node; incnode(node)) {
-	char *str, c;
-
-	str = (char *)getdata(node);
-	if (((c = *str) == Inang || c == Outang || c == Equals) &&
-	    str[1] == Inpar) {
-	    if (c == Inang || c == Outang)
-		setdata(node, (void *) getproc(str));	/* <(...) or >(...) */
-	    else
-		setdata(node, (void *) getoutputfile(str));	/* =(...) */
-	    if (!getdata(node)) {
-		setdata(node, dupstring(""));
-		unqueue_signals();
-		return;
-	    }
-	} else {
-	    if (isset(SHFILEEXPANSION)) {
-		/*
-		 * Here and below we avoid taking the address
-		 * of a void * and then pretending it's a char **
-		 * instead of a void ** by a little inefficiency.
-		 * This could be avoided with some extra linked list
-		 * machinery, but that would need quite a lot of work
-		 * to ensure consistency.  What we really need is
-		 * templates...
-		 */
-		char *cptr = (char *)getdata(node);
-		filesub(&cptr, flags & (PF_TYPESET|PF_ASSIGN));
-		/*
-		 * The assignment is so simple it's not worth
-		 * testing if cptr changed...
-		 */
-		setdata(node, cptr);
-	    }
-	    if (!(node = stringsubst(list, node, flags & PF_SINGLE, asssub))) {
-		unqueue_signals();
-		return;
-	    }
+	if (isset(SHFILEEXPANSION)) {
+	    /*
+	     * Here and below we avoid taking the address
+	     * of a void * and then pretending it's a char **
+	     * instead of a void ** by a little inefficiency.
+	     * This could be avoided with some extra linked list
+	     * machinery, but that would need quite a lot of work
+	     * to ensure consistency.  What we really need is
+	     * templates...
+	     */
+	    char *cptr = (char *)getdata(node);
+	    filesub(&cptr, flags & (PF_TYPESET|PF_ASSIGN));
+	    /*
+	     * The assignment is so simple it's not worth
+	     * testing if cptr changed...
+	     */
+	    setdata(node, cptr);
+	}
+	if (!(node = stringsubst(list, node, flags & PF_SINGLE, asssub))) {
+	    unqueue_signals();
+	    return;
 	}
     }
     for (node = firstnode(list); node; incnode(node)) {
@@ -168,7 +152,37 @@
     char *str  = str3, c;
 
     while (!errflag && (c = *str)) {
-	if ((qt = c == Qstring) || c == String) {
+	if ((c == Inang || c == Outang || (str == str3 && c == Equals)) &&
+	    str[1] == Inpar) {
+	    char *subst, *rest, *snew, *sptr;
+	    int str3len = str - str3, sublen, restlen;
+
+	    if (c == Inang || c == Outang)
+		subst = getproc(str, &rest);	/* <(...) or >(...) */
+	    else
+		subst = getoutputfile(str, &rest);	/* =(...) */
+	    if (!subst)
+		subst = "";
+
+	    sublen = strlen(subst);
+	    restlen = strlen(rest);
+	    sptr = snew = hcalloc(str3len + sublen + restlen + 1);
+	    if (str3len) {
+		memcpy(sptr, str3, str3len);
+		sptr += str3len;
+	    }
+	    if (sublen) {
+		memcpy(sptr, subst, sublen);
+		sptr += sublen;
+	    }
+	    if (restlen)
+		memcpy(sptr, rest, restlen);
+	    sptr[restlen] = '\0';
+	    str3 = snew;
+	    str = snew + str3len + sublen;
+	    setdata(node, str3);
+	    continue;
+	} else if ((qt = c == Qstring) || c == String) {
 	    if ((c = str[1]) == Inpar) {
 		if (!qt)
 		    list->list.flags |= LF_ARRAY;
Index: Test/D03procsubst.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D03procsubst.ztst,v
retrieving revision 1.4
diff -u -r1.4 D03procsubst.ztst
--- Test/D03procsubst.ztst	14 Apr 2005 04:41:28 -0000	1.4
+++ Test/D03procsubst.ztst	13 Nov 2008 20:46:45 -0000
@@ -36,3 +36,51 @@
 0:FDs remain open for external commands called from functions
 >First
 >Zweite
+
+  catfield2() {
+    local -a args
+    args=(${(s.,.)1})
+    print $args[1]
+    cat $args[2]
+    print $args[3]
+  }
+  catfield2 up,<(print $'\x64'own),sideways
+0:<(...) when embedded within an argument
+>up
+>down
+>sideways
+
+  outputfield2() {
+    local -a args
+    args=(${(s.,.)1})
+    print $args[1]
+    echo 'How sweet the moonlight sits upon the bank' >$args[2]
+    print $args[3]
+  }
+  outputfield2 muddy,>(sed -e s/s/th/g >outputfield2.txt),vesture
+  # yuk
+  while [[ ! -e outputfield2.txt || ! -s outputfield2.txt ]]; do :; done
+  cat outputfield2.txt
+0:>(...) when embedded within an argument
+>muddy
+>vesture
+>How thweet the moonlight thitth upon the bank
+
+  catfield1() {
+    local -a args
+    args=(${(s.,.)1})
+    cat $args[1]
+    print $args[2]
+  }
+  catfield1 =(echo s$'\x69't),jessica
+0:=(...) followed by something else without a break
+>sit
+>jessica
+
+  (
+  setopt nonomatch
+  # er... why is this treated as a glob?
+  print everything,=(here is left),alone
+  )
+0:=(...) preceded by other stuff has no special effect
+>everything,=(here is left),alone


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: parse error in process substitution

[Mikael Magnusson]

2008/11/13 Peter Stephenson <p.w.stephenson@ntlworld.com>:
> On Mon, 10 Nov 2008 09:21:52 +0100
> Louis-David Mitterrand <vindex+lists-zsh-users@apartia.org> wrote:
>> So how would you convert that working bash command to zsh?
>>
>>       root-tail <(ssh root@my.host.name tail -F /var/log/kern.log),red,
>
> This fixes the syntax.  Unfortunately, I had to impose some limitations
> to prevent existing stuff stopping working.  Obviously I'd like to
> know about anything else this messes up.  I'm not worried about
> the undocumented feature that <(...) forced a new word.
>
> A non-initial = usually isn't handled specially, and raw parentheses
> (i.e. without a disambiguating character in front) are somewhat
> overworked in zsh, so I couldn't make =(...) special anywhere except at
> the start of an argument.  However, you can now put other things after
> it.
>
> ">" and "<" never used to need quoting when used inside parentheses or
> parameter substitutions, so to keep this I've had to forbid the use of
> <(...) and >(...) in such places.  (The comptest function falls over if
> this isn't done.)  This isn't likely to be a big problem
> in practice.  Because of the previous rule this isn't relevant to
> =(...).
>
> Parsing of the contents of these expressions is done rather more simply
> than for $(...) expressions.  I don't really know if there's any mileage
> in making parsing of process substitutions more similar (inside the
> parentheses only, the context dependence will have to remain
> different).
>
> Index: README
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/README,v
> retrieving revision 1.58
> diff -u -r1.58 README
> --- README      30 Oct 2008 12:18:54 -0000      1.58
> +++ README      13 Nov 2008 20:46:40 -0000
> @@ -69,6 +69,16 @@
>  consistent with recent versions of other shells.  The option
>  DEBUG_BEFORE_CMD can be unset to revert to the previous behaviour.
>
> +Previously, process substitutions of the form =(...), <(...) and
> +>(...) were only handled if they appeared as separate command arguments,
> +although the latter two forms caused a new command argument to be
> +started at that point.

This sentence is somewhat confusing, but i guess it means that <() and
>() always started a new command argument even if they weren't
separate... :)

> +Now all three may be followed by other strings,
> +and the latter two may also be preceeded by other string.

strings

> +is subject to process substitution.  The expression may be proceeded

preceeded

> +or followed by other strings except that, to prevent clashes with
> +commonly occurring strings and patterns, the last
> +form must occur at the start of a command argument, and none of
> +the forms may occur inside parentheses used for grouping of patterns or
> +inside parameter substitutions.
> +
>  In the case of the tt(<) or tt(>) forms, the shell runs process
>  var(list) asynchronously.  If the system supports the tt(/dev/fd)
>  mechanism, the command argument is the name of the device file

Should this be "the process list"?

-- 
Mikael Magnusson

unusable fd in tcp_send

[Clint Adams]

Is there a reason not to set mystat or otherwise flag an error if
TCP_FD_CLOSED gets set in tcp_send?

Re: unusable fd in tcp_send

[Peter Stephenson]

Clint Adams wrote:
> Is there a reason not to set mystat or otherwise flag an error if
> TCP_FD_CLOSED gets set in tcp_send?

That sounds like it makes sense.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: unusable fd in tcp_send

[Clint Adams]

On Sat, Nov 15, 2008 at 04:32:28PM +0000, Peter Stephenson wrote:
> That sounds like it makes sense.

Index: Functions/TCP/tcp_send
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/TCP/tcp_send,v
retrieving revision 1.4
diff -u -r1.4 tcp_send
--- Functions/TCP/tcp_send	8 Aug 2006 11:27:40 -0000	1.4
+++ Functions/TCP/tcp_send	15 Nov 2008 16:45:37 -0000
@@ -74,6 +74,8 @@
     if [[ $? -ne 0 || -n $TCP_FD_CLOSED ]]; then
       print "Session ${TCP_SESS}: fd $fd unusable." >&2
       unset TCP_FD_CLOSED
+      mystat=1
+      continue
     fi
     if [[ -n $TCP_OUTPUT ]]; then
       tcp_output -P "$TCP_OUTPUT" -S $TCP_SESS -F $fd -q "${(j. .)*}"

Re: parse error in process substitution

[Bart Schaefer]

On Nov 13,  9:00pm, Peter Stephenson wrote:
} 
} A non-initial = usually isn't handled specially, and raw parentheses
} (i.e. without a disambiguating character in front) are somewhat
} overworked in zsh, so I couldn't make =(...) special anywhere except at
} the start of an argument.

I think this is OK, as it's already ambiguous with respect to array
assignment syntax when not at the start of a word.

I"m glad to see that var==(cmd) still assigns a file name to $var.

} ">" and "<" never used to need quoting when used inside parentheses or
} parameter substitutions, so to keep this I've had to forbid the use of
} <(...) and >(...) in such places.

Can you give examples of a couple of "such places"?  I thought I knew
what you meant, but when I try what I thought should fail, it doesn't.
For example:

schaefer[514] echo x<(echo foo)y
x/proc/self/fd/11y
schaefer[515] echo (<(echo foo)|>(echo bar))
bar
zsh: bad pattern: (/proc/self/fd/11|/proc/self/fd/13)

OK, so it's a bad pattern, but the substitutions still happened.

} Parsing of the contents of these expressions is done rather more simply
} than for $(...) expressions.  I don't really know if there's any mileage
} in making parsing of process substitutions more similar (inside the
} parentheses only, the context dependence will have to remain
} different).

Is the $(...) one perhaps more complex because it has to work inside
double-quotes and therefore with possible quote nesting?  Otherwise
I would think that the simplest parse that doesn't fail on convoluted
cases is the one that both of them should be using.  

One last item ... perhaps you can explain what's going on here:

schaefer[518] cat <<(echo foo)what?
foo
schaefer[519]

I'm not sure exactly what I expected, but that wasn't it.

Re: parse error in process substitution

[Peter Stephenson]

Bart Schaefer wrote:
> } ">" and "<" never used to need quoting when used inside parentheses or
> } parameter substitutions, so to keep this I've had to forbid the use of
> } <(...) and >(...) in such places.
> 
> Can you give examples of a couple of "such places"?  I thought I knew
> what you meant, but when I try what I thought should fail, it doesn't.

I think they're more specialised than I previously assumed.  The one I
was particularly aiming at was

% var="<foo><bar>"
% print ${var##<([a-z]##|)>}
<bar>

(which still works) which is similar to what's used in Test/comptest.
This is probably due to the comment quoted below.

> For example:
> 
> schaefer[514] echo x<(echo foo)y
> x/proc/self/fd/11y
> schaefer[515] echo (<(echo foo)|>(echo bar))
> bar
> zsh: bad pattern: (/proc/self/fd/11|/proc/self/fd/13)
> 
> OK, so it's a bad pattern, but the substitutions still happened.

Your're right, I've claimed more than what's actually the case.  I think
I've achieved what I was mainly aiming at, i.e. allowing anything that
was previously valid still to be so (cases like this previously reported
a parse error at the "<" or ">"), but it doesn't correspond to what I
said in the documentation.

The key point is this comment in the lexical analyser:

   When called from gettok(), with sub = 0, we have already identified
   any interesting initial character and want to get the rest of
   what we now know is a string.  However, the string may still include
   metacharacters and potentially substitutions.
 
   When called from parse_subst_string() with sub = 1, we are not
   fully parsing a command line, merely tokenizing a string.
   In this case we always add characters to the parsed string
   unless there is a parse error.

I've allowed <(...) and >(...) to be expanded when sub = 0, but not when
sub = 1, which is what happened before except only if the expansions
were complete arguments, so I need to find a better description.
"When parsing a string" is close but perhaps doesn't quite express it.

(It's not at all obvious why you would ever use the substitutions like
that but I'm aiming at consistency rather than imposing arbitrary
limitations on cases I don't feel like handling.)

> } Parsing of the contents of these expressions is done rather more simply
> } than for $(...) expressions.  I don't really know if there's any mileage
> } in making parsing of process substitutions more similar (inside the
> } parentheses only, the context dependence will have to remain
> } different).
>
> Is the $(...) one perhaps more complex because it has to work inside
> double-quotes and therefore with possible quote nesting?  Otherwise
> I would think that the simplest parse that doesn't fail on convoluted
> cases is the one that both of them should be using.  

Yes, that may well be the case, as occurred to me over the weekend, so I
don't need to do anything cleverer since I don't want < and > to become
special in double quotes.

> One last item ... perhaps you can explain what's going on here:
> 
> schaefer[518] cat <<(echo foo)what?
> foo
> schaefer[519]
> 
> I'm not sure exactly what I expected, but that wasn't it.

I haven't changed any of the parsing for redirections: they are handled
separately.  So it looks like this is now inconsistent and needs fixing.

Again, it's not clear why you would ever do that.  I suppose since the
file name produced by the substitution is essentially opaque to the user
you could argue it should report an error in this and similar
cases---otherwise, it looks like the case "when parsing a string", with
the complication that globs here trigger the multio syntax which isn't
relevant to the case of process substitution.  An error would probably
be simpler.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: parse error in process substitution

[Bart Schaefer]

On Nov 16,  9:18pm, Peter Stephenson wrote:
}
} I think they're more specialised than I previously assumed.  The one I
} was particularly aiming at was
} 
} % var="<foo><bar>"
} % print ${var##<([a-z]##|)>}
} <bar>

Ah, and hmm.  So:

schaefer[504] echo (<(echo|cat)|>(echo|cat))
zsh: bad pattern: (/proc/self/fd/11|/proc/self/fd/13)
schaefer[505] setopt nonomatch nocshnullglob
schaefer[506] echo ${:-(<(echo|cat)|>(echo|cat))}
(<(echo|cat)|>(echo|cat))
schaefer[507] echo ${~:-(<(echo|cat)|>(echo|cat))}
zsh: bad pattern: (<(echo|cat)|/proc/self/fd/12)
zsh: command not found: echocat

Something still seems a bit odd there.  Where did "echocat" come from?

Also there's this -- first, zsh-4.2.0:

schaefer<501> echo $(<(echo cat))                    
zsh: no such file or directory: (echo cat)

Now 4.3.9-dev-0 latest CVS:

schaefer[509] echo $(<(echo cat))                    
zsh: permission denied: /proc/self/fd/10

I wonder if the $(< ....) form should also be special-cased?

} I've allowed <(...) and >(...) to be expanded when sub = 0, but not when
} sub = 1, which is what happened before except only if the expansions
} were complete arguments, so I need to find a better description.
} "When parsing a string" is close but perhaps doesn't quite express it.

That also leaves unanswered the question of exactly what constitutes
"a string" ... is the shell grammar really completely documented enough
to make that discernible?

 
} > One last item ... perhaps you can explain what's going on here:
} > 
} > schaefer[518] cat <<(echo foo)what?
} > foo
} > schaefer[519]
} > 
} > I'm not sure exactly what I expected, but that wasn't it.
} 
} I haven't changed any of the parsing for redirections: they are handled
} separately.  So it looks like this is now inconsistent and needs fixing.

It used to produce a parse error.  I guess I was expecting that it
would now produce either a "no such file" or a "no match" error
because "/proc/self/fd/11what?" doesn't exist.
 
} Again, it's not clear why you would ever do that.

True, but the shell shouldn't silently discard typographical errors
unless somehow instructed to do so.

-- 

Re: parse error in process substitution

[Peter Stephenson]

On Sun, 16 Nov 2008 13:51:27 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> schaefer[504] echo (<(echo|cat)|>(echo|cat))
> zsh: bad pattern: (/proc/self/fd/11|/proc/self/fd/13)
> schaefer[505] setopt nonomatch nocshnullglob
> schaefer[506] echo ${:-(<(echo|cat)|>(echo|cat))}
> (<(echo|cat)|>(echo|cat))
> schaefer[507] echo ${~:-(<(echo|cat)|>(echo|cat))}
> zsh: bad pattern: (<(echo|cat)|/proc/self/fd/12)
> zsh: command not found: echocat
> 
> Something still seems a bit odd there.  Where did "echocat" come from?

That's from >(echo|cat) which is getting parsed in some strange way.  It
shouldn't be being substituted at all.  I'll have to track that down.

> Also there's this -- first, zsh-4.2.0:
> 
> schaefer<501> echo $(<(echo cat))                    
> zsh: no such file or directory: (echo cat)
> 
> Now 4.3.9-dev-0 latest CVS:
> 
> schaefer[509] echo $(<(echo cat))                    
> zsh: permission denied: /proc/self/fd/10
> 
> I wonder if the $(< ....) form should also be special-cased?

Possibly, although you could also argue that if the contents look like
anything other than < followed by something that expands to a string it
should do that.  It depends what you mean by "look like".  You need to
quote metacharacters to be sure of getting a file name in general, of
course.

> } I've allowed <(...) and >(...) to be expanded when sub = 0, but not when
> } sub = 1, which is what happened before except only if the expansions
> } were complete arguments, so I need to find a better description.
> } "When parsing a string" is close but perhaps doesn't quite express it.
> 
> That also leaves unanswered the question of exactly what constitutes
> "a string" ... is the shell grammar really completely documented enough
> to make that discernible?

Not really.  Here it means something on the lines of "anything other than a
positional argument to a command or to array assignment".

> } > One last item ... perhaps you can explain what's going on here:
> } > 
> } > schaefer[518] cat <<(echo foo)what?
> } > foo
> } > schaefer[519]
> } > 
> } > I'm not sure exactly what I expected, but that wasn't it.
> } 
> } I haven't changed any of the parsing for redirections: they are handled
> } separately.  So it looks like this is now inconsistent and needs fixing.
> 
> It used to produce a parse error.  I guess I was expecting that it
> would now produce either a "no such file" or a "no match" error
> because "/proc/self/fd/11what?" doesn't exist.

The reason I suggested it should perhaps still be a parse error is mostly
for the case of >>(...).  This won't work either but might give a more
confusing error message or use a different fd or something.  The latter
might happen with the input form, in fact.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: parse error in process substitution

[Bart Schaefer]

On Nov 17, 10:10am, Peter Stephenson wrote:
} Subject: Re: parse error in process substitution
}
} > schaefer[509] echo $(<(echo cat))                    
} > zsh: permission denied: /proc/self/fd/10
} > 
} > I wonder if the $(< ....) form should also be special-cased?
} 
} Possibly, although you could also argue that if the contents look like
} anything other than < followed by something that expands to a string
} it should do that.

The question is whether "$(<" binds more tightly than "<(".  I suppose
the answer is that really it's dollar oparen redirect word cparen that
has special meaning, and in lessthan oparen list cparen, lessthan is
not a redirect even though it "looks like" one (because "<(" is a
token unto itself, not the two tokens lessthan and oparen).

} It depends what you mean by "look like". You need to quote
} metacharacters to be sure of getting a file name in general, of
} course.

schaefer[502] bash                 
[schaefer@torch]$ echo < (echo cat)
bash: syntax error near unexpected token `('
[schaefer@torch]$ exit
schaefer[503] echo < (echo cat)
zsh: number expected
schaefer[504] echo $(< (echo cat) )
zsh: no such file or directory: (echo cat)
schaefer[505] echo $(<(echo cat) ) 
zsh: permission denied: /proc/self/fd/10

I'm not really making anything of the above yet, just pointing it out.

Re: parse error in process substitution

[Peter Stephenson]

On Sun, 16 Nov 2008 13:51:27 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> schaefer[507] echo ${~:-(<(echo|cat)|>(echo|cat))}
> zsh: bad pattern: (<(echo|cat)|/proc/self/fd/12)
> zsh: command not found: echocat
> 
> Something still seems a bit odd there.  Where did "echocat" come from?

OK, I now understand this one.  My change mixed up handling of process and
parameter substitution, so that after a GLOB_SUBST you could get back to
process substitution, and because it hadn't been parsed properly in the
first instance you were in a fairly confused state.

It's clearly documented and has been for a long time that process
substitution comes before parameter and command substitution, so the fix is
straightforward and doesn't need specially documenting.

I've changed the text about when process subsitutions happens.  Further
improvements with explicit text are welcome.

I think that's the end of actual bugs I know about, the rest are disputable
changes of behaviour which I'll think some more about.

Index: README
===================================================================
RCS file: /cvsroot/zsh/zsh/README,v
retrieving revision 1.60
diff -u -r1.60 README
--- README	15 Nov 2008 16:30:31 -0000	1.60
+++ README	17 Nov 2008 16:01:09 -0000
@@ -74,11 +74,9 @@
 (However, the latter two forms caused the current argument to be
 terminated and a new one started even if they occurred in the middle of
 a string.)  Now all three may be followed by other strings, and the
-latter two may also be preceeded by other strings.  None may occur inside
-parameter substitutions, or inside parentheses used for grouping of
-patterns, in order to avoid clashes with cases where
-tt(<) or tt(>) were not treated specially in previous versions of the
-shell.
+latter two may also be preceeded by other strings.  Remaining
+limitations on their use (to reduce incompatibilities to a minimum)
+are documented in the zshexpn.1 manual.
 
 In previous versions of the shell it was possible to use index 0 in an
 array or string subscript to refer to the same element as index 1 if the
Index: Doc/Zsh/expn.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/expn.yo,v
retrieving revision 1.98
diff -u -r1.98 expn.yo
--- Doc/Zsh/expn.yo	13 Nov 2008 21:18:14 -0000	1.98
+++ Doc/Zsh/expn.yo	17 Nov 2008 16:01:09 -0000
@@ -360,9 +360,8 @@
 is subject to process substitution.  The expression may be preceeded
 or followed by other strings except that, to prevent clashes with
 commonly occurring strings and patterns, the last
-form must occur at the start of a command argument, and none of
-the forms may occur inside parentheses used for grouping of patterns or
-inside parameter substitutions.
+form must occur at the start of a command argument, and the forms
+are only expanded when first parsing command or assignment arguments.
 
 In the case of the tt(<) or tt(>) forms, the shell runs the commands in
 var(list) asynchronously.  If the system supports the tt(/dev/fd)
Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.91
diff -u -r1.91 subst.c
--- Src/subst.c	13 Nov 2008 21:18:14 -0000	1.91
+++ Src/subst.c	17 Nov 2008 16:01:09 -0000
@@ -152,8 +152,8 @@
     char *str  = str3, c;
 
     while (!errflag && (c = *str)) {
-	if ((c == Inang || c == Outang || (str == str3 && c == Equals)) &&
-	    str[1] == Inpar) {
+	if (((c = *str) == Inang || c == Outang || (str == str3 && c == Equals))
+	    && str[1] == Inpar) {
 	    char *subst, *rest, *snew, *sptr;
 	    int str3len = str - str3, sublen, restlen;
 
@@ -181,8 +181,13 @@
 	    str3 = snew;
 	    str = snew + str3len + sublen;
 	    setdata(node, str3);
-	    continue;
-	} else if ((qt = c == Qstring) || c == String) {
+	} else
+	    str++;
+    }
+    str = str3;
+
+    while (!errflag && (c = *str)) {
+	if ((qt = c == Qstring) || c == String) {
 	    if ((c = str[1]) == Inpar) {
 		if (!qt)
 		    list->list.flags |= LF_ARRAY;

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: parse error in process substitution

[Peter Stephenson]

On Sat, 15 Nov 2008 20:10:55 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> One last item ... perhaps you can explain what's going on here:
> 
> schaefer[518] cat <<(echo foo)what?
> foo
> schaefer[519]
> 
> I'm not sure exactly what I expected, but that wasn't it.

This turns it back into an error with a specific message.

Index: Doc/Zsh/expn.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/expn.yo,v
retrieving revision 1.99
diff -u -r1.99 expn.yo
--- Doc/Zsh/expn.yo	17 Nov 2008 16:11:30 -0000	1.99
+++ Doc/Zsh/expn.yo	17 Nov 2008 16:47:07 -0000
@@ -362,6 +362,8 @@
 commonly occurring strings and patterns, the last
 form must occur at the start of a command argument, and the forms
 are only expanded when first parsing command or assignment arguments.
+Process substitutions may be used following redirection operators; in this
+case, the substitution must appear with no trailing string.
 
 In the case of the tt(<) or tt(>) forms, the shell runs the commands in
 var(list) asynchronously.  If the system supports the tt(/dev/fd)
Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.161
diff -u -r1.161 exec.c
--- Src/exec.c	13 Nov 2008 21:18:14 -0000	1.161
+++ Src/exec.c	17 Nov 2008 16:47:07 -0000
@@ -3773,9 +3773,14 @@
     int pipes[2], out = *cmd == Inang;
     pid_t pid;
     struct timeval bgtime;
+    char *ends;
 
-    if (!(prog = parsecmd(cmd, NULL)))
+    if (!(prog = parsecmd(cmd, &ends)))
 	return -1;
+    if (*ends) {
+	zerr("invalid syntax for process substitution in redirection");
+	return -1;
+    }
     mpipe(pipes);
     if ((pid = zfork(&bgtime))) {
 	zclose(pipes[out]);


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Bug in executable completion: unable to handle .. it $PATH

[Richard Hartmann]

Hi all,

don't ask me why anyone would use .. in $PATH, but here
goes:

roadwarrior% touch foobin
roadwarrior% chmod +x foobin
roadwarrior% export PATH=/home/richih/killme
roadwarrior% fo<tab>
foo        foobin     for        foreach
roadwarrior% export PATH=/home/richih/../richih/killme
roadwarrior% fo<tab>
roadwarrior% for
roadwarrior%


Richard

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=162291

Re: Bug in executable completion: unable to handle .. it $PATH

[Vincent Lefevre]

On 2009-01-01 17:24:17 +0100, Richard Hartmann wrote:
> roadwarrior% touch foobin
> roadwarrior% chmod +x foobin
> roadwarrior% export PATH=/home/richih/killme
> roadwarrior% fo<tab>
> foo        foobin     for        foreach
> roadwarrior% export PATH=/home/richih/../richih/killme
> roadwarrior% fo<tab>
> roadwarrior% for
> roadwarrior%

Same problem here: the "hash" utility outputs nothing in the second case.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: Bug in executable completion: unable to handle .. it $PATH

[Peter Stephenson]

On Thu, 1 Jan 2009 17:24:17 +0100
"Richard Hartmann" <richih.mailinglist@gmail.com> wrote:
> [the shell won't hash paths containing ".."]

This is done explicitly in the code, but I have no idea why; it precedes
the CVS archive.  The function isrelative() is only used by hashdir().

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.162
diff -u -r1.162 exec.c
--- Src/exec.c	17 Nov 2008 16:56:42 -0000	1.162
+++ Src/exec.c	7 Jan 2009 20:05:35 -0000
@@ -795,11 +795,16 @@
 {
     if (*s != '/')
 	return 1;
+#if 0
+    /*
+     * Eh?  .. or . in the path doesn't make the directory relative.
+     */
     for (; *s; s++)
 	if (*s == '.' && s[-1] == '/' &&
 	    (s[1] == '/' || s[1] == '\0' ||
 	     (s[1] == '.' && (s[2] == '/' || s[2] == '\0'))))
 	    return 1;
+#endif
     return 0;
 }
 
Index: Src/hashtable.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/hashtable.c,v
retrieving revision 1.30
diff -u -r1.30 hashtable.c
--- Src/hashtable.c	1 Nov 2008 01:19:35 -0000	1.30
+++ Src/hashtable.c	7 Jan 2009 20:05:35 -0000
@@ -588,7 +588,7 @@
  
 /**/
 mod_export char **pathchecked;
- 
+
 /* Create a new command hash table */
  
 /**/

Re: Bug in executable completion: unable to handle .. it $PATH

[Bart Schaefer]

On Jan 7,  8:09pm, Peter Stephenson wrote:
}
} This is done explicitly in the code, but I have no idea why; it precedes
} the CVS archive.  The function isrelative() is only used by hashdir().

I believe it's a security thing, so that an inherited $PATH can't fool
someone into executing code from an unexpected place.

Re: Bug in executable completion: unable to handle .. it $PATH

[Peter Stephenson]

Bart Schaefer wrote:
> On Jan 7,  8:09pm, Peter Stephenson wrote:
> }
> } This is done explicitly in the code, but I have no idea why; it precedes
> } the CVS archive.  The function isrelative() is only used by hashdir().
> 
> I believe it's a security thing, so that an inherited $PATH can't fool
> someone into executing code from an unexpected place.

I don't think that can be it, since this feature is only in the command
hashing.  If you type the command name in full it will still be
executed.  So this has virtually no effect on non-interactive use.

Since the path is still absolute I don't see how this could effect
security, either, except maybe at second hand... if you sanitized the
early part of the path but didn't look for "..", so the component could
end up pointing out of that area, for example.  But that doesn't seem to
me to be the shell's problem.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug in executable completion: unable to handle .. it $PATH

[Richard Hartmann]

On Wed, Jan 7, 2009 at 21:49, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:

> Since the path is still absolute I don't see how this could effect
> security, either, except maybe at second hand... if you sanitized the
> early part of the path but didn't look for "..", so the component could
> end up pointing out of that area, for example.  But that doesn't seem to
> me to be the shell's problem.

I am trying to construct a scenario with softlinks, but I don't think is
hurting here either.

In any case, thanks for fixing this :)


Richard

Re: sourcing a sh file in zsh

[Bart Schaefer]

[Moved to -workers]

On Jan 16,  7:39pm, Andrey Borzenkov wrote:
}
} > You'll probably need to do something like
} >
} >   emulate sh
} >   source bla.sh
} 
} Given that most distributions have equivalence of /etc/profile.d, having 
} simple zsh way to do it (without scoping issues) would be nice.

I'm reluctant to suggest adding options to "source", but introducing an
alternate name ("zsource"?) might allow something like

  zsource -e sh file.sh

with the obvious meaning of "turn on sh emulation while sourcing" and
would otherwise behave like "source".  Might need a variant that has
the path-searching semantics of "." as well.

Another possibility would be to extend the "emulate" command into a
precommand modifier sort of thing, where arguments following the name
of the emulation are treated as a command to execute.  Perhaps require
another option to make this active, to avoid any compatibily issue.

  emulate -LRE sh source file.sh

(I can't decide whether the new option should be S(ource), X(ecute),
or E(val), nor whether it should be upper case.)

In this case the -c and -a options of "exec" might be included as well.

Re: sourcing a sh file in zsh

[Phil Pennock]

On 2009-01-16 at 10:29 -0800, Bart Schaefer wrote:
> with the obvious meaning of "turn on sh emulation while sourcing" and
> would otherwise behave like "source".  Might need a variant that has
> the path-searching semantics of "." as well.

Isn't there a problem where a function is defined which assumes one set
of semantics and the reset afterwards changes the semantics?  Thus the
reason why the entry points into completion set the relevant emulation
options.

Really, we need to be able to have a function "remember" the options in
effect at the time it was defined and restore them, without having to
call emulate within them.

At $work there are some shell libraries which I could use if this were
doable, which is why my notes of Things I Want To Get Around To Making
Zsh Support includes this item:

----------------------------8< cut here >8------------------------------
zsh setopt inherit_options
When set, functions defined remember the options set at time of definition,
so it's similar to an automatic "emulate" in them.
Most usefully: setopt local_options inherit_options
function fred { setopt localoptions shwordsplit; alpha="foo bar baz bat"; print -l $alpha; function barney { print -l $alpha } }
(Multi-shell compatibility libraries)
----------------------------8< cut here >8------------------------------

Thoughts, comments?

I can see how to go about doing it but haven't gotten around to it yet.

-Phil

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 16,  7:59pm, Phil Pennock wrote:
}
} On 2009-01-16 at 10:29 -0800, Bart Schaefer wrote:
} > with the obvious meaning of "turn on sh emulation while sourcing" and
} > would otherwise behave like "source".  Might need a variant that has
} > the path-searching semantics of "." as well.
} 
} Isn't there a problem where a function is defined which assumes one set
} of semantics and the reset afterwards changes the semantics?

The reset after what?  This hypothetical feature?  Well, since it's
hypothetical, I'd have to say no, there isn't a problem, because if
the feature ever ceases to be hypothetical it should be implemented
so as to avoid that problem.

} Thus the reason why the entry points into completion set the relevant
} emulation options.

No, the reason the entry points into completion set the options is
because the functions in the completion system are written to expect
a specific set of options which are NOT the default for any of the
predefined emulations, so "emulate" isn't sufficient there.

If the constraints of (a) a function scope has been entered and (b)
"setopt localoptions" is in effect when the scope ends, then the
setopts after the scope is gone should be exactly the same as they
were before it began.  There is no bug I'm aware of where an option
is wrongly (not) restored after ending a localoptions scope.
 
} Really, we need to be able to have a function "remember" the options
} in effect at the time it was defined and restore them, without having
} to call emulate within them.

I think I begin to understand what you mean, but I think it's misguided.
There's no way for a function that's stored in script form as a file,
and then later sourced or autoloaded, to keep track of what options
were *intended* to be in effect at the time the file was stored to the
disk.  It can only know what options were in effect at the time that
the "source" command was executed, which doesn't solve the problem.

The closest you could get would be to add an option to zcompile to
have it prepend the compiled function definition with an appropriate 
setopts command.  This is already partly done by zcompile for the
special case of KSH_AUTOLOAD.

I see from your example that you suggest hacking this up semi-manually
by defining one function within another so that the outer function can
establish the option settings that the inner one should remember, but
that's only minimally helpful to the function writer and still doesn't
solve the original problem of variable scoping.

-- 

Re: sourcing a sh file in zsh

[Phil Pennock]

On 2009-01-16 at 21:00 -0800, Bart Schaefer wrote:
> On Jan 16,  7:59pm, Phil Pennock wrote:
> } On 2009-01-16 at 10:29 -0800, Bart Schaefer wrote:
> } > with the obvious meaning of "turn on sh emulation while sourcing" and
> } > would otherwise behave like "source".  Might need a variant that has
> } > the path-searching semantics of "." as well.
> } 
> } Isn't there a problem where a function is defined which assumes one set
> } of semantics and the reset afterwards changes the semantics?
> 
> The reset after what?  This hypothetical feature?  Well, since it's
> hypothetical, I'd have to say no, there isn't a problem, because if
> the feature ever ceases to be hypothetical it should be implemented
> so as to avoid that problem.

Okay, and that raises the issue of reset and options from outside a
function needing to be retained as part of the function definition.

> I think I begin to understand what you mean, but I think it's misguided.
> There's no way for a function that's stored in script form as a file,
> and then later sourced or autoloaded, to keep track of what options
> were *intended* to be in effect at the time the file was stored to the
> disk.  It can only know what options were in effect at the time that
> the "source" command was executed, which doesn't solve the problem.
> 
> The closest you could get would be to add an option to zcompile to
> have it prepend the compiled function definition with an appropriate 
> setopts command.  This is already partly done by zcompile for the
> special case of KSH_AUTOLOAD.
> 
> I see from your example that you suggest hacking this up semi-manually
> by defining one function within another so that the outer function can
> establish the option settings that the inner one should remember, but
> that's only minimally helpful to the function writer and still doesn't
> solve the original problem of variable scoping.

The example in my notes is probably not the best; it's based on an
assumption that loading external non-zsh files is best done using a
function which sets the options wanted and sources, to restore the state
needed afterwards, but a zsource would be equivalent.

The problem I have is some utility libraries which are written with bash
as an assumption and then justified to the zsh-users with "hey, they
work with zsh too" because they, oh, setopt sh_word_split.  I don't want
my normal shell contaminated with that garbage -- I like zsh's
behaviour.

The idea is basically just "set this option and functions have an
implicit emulate at the start which sets the options in effect at the
time the function was defined", to let me source utility libraries and
have sh_word_split turned on for them (and probably ksh_arrays too)
without having it effect everything else I do in the shell.  The
front-end interface file, which sources everything else and which has
shell-specific shims to "handle" zsh can then have a zsh version-check
to just instead catch the value of the options, setopt inherit_options
and the others, and then restore options after it's done pulling in the
other files.

I can easily get updates made to the interface, but there's no way that
shell-specific stuff will get into every function defined across all the
other files -- #ifdef littering is bad, in any language.

So, the problem is probably my lack of understanding of various areas of
internals I haven't previously touched; my rought plan is:

 * use copies of opts, which can be memcpy'd around (since it's a static
   variable and making it a variable pointer might be more intrusive;
   wrap it in an API and this can be changed if it's easier to make opts
   a pointer which can be moved)
 * minimally, when inherit_options is in effect, bar changes to
   restricted, interactive, privileged, etc
 * setopt inherit_options creates a copy of opts; either:
   (a) this is read-only, no options changed within inherit scope, so
       set things up carefully first;
   (b) all functions defined within the scope of one instance of setting
       inherit_options share a unified option space and changes in one
       affect all others
 * At time of function call, see if crossing a boundary between values
   of inherit_options; if so, put the pointer to the current opts into a
   new item in struct funcstack; when unwinding stack, if the
   restoreopts item is non-NULL, restore it.

The problem is how to decide if a function needs a new option; I haven't
looked too closely but I believe that functions are invoked from each
other directly via the Eprog code, rather than dispatch through the
shfunctab shfunc entries, so it's not viable to go via extending shfunc
(pity).

So I'm thinking an Eprog WC_OPTSPACE instruction which can be inserted
at some point arounf execfuncdef() time, which is stored with a value
which points to the opts value to be put into effect and which, when
encountered, adjusts the current funcstack entry to hold the current
opts before replacing opts with the one referenced in the Eprog.

Okay, how many fundamental misunderstandings on my part have I just
exposed?

-Phil

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 16,  9:55pm, Phil Pennock wrote:
} Subject: Re: sourcing a sh file in zsh
}
} On 2009-01-16 at 21:00 -0800, Bart Schaefer wrote:
} > On Jan 16,  7:59pm, Phil Pennock wrote:
} > } On 2009-01-16 at 10:29 -0800, Bart Schaefer wrote:
} > } > with the obvious meaning of "turn on sh emulation while sourcing" and
} > } > would otherwise behave like "source".  Might need a variant that has
} > } > the path-searching semantics of "." as well.
} > } 
} > } Isn't there a problem where a function is defined which assumes one
} > } set of semantics and the reset afterwards changes the semantics?
} > 
} > The reset after what?  This hypothetical feature?  Well, since it's
} > hypothetical, I'd have to say no, there isn't a problem, because if
} > the feature ever ceases to be hypothetical it should be implemented
} > so as to avoid that problem.
} 
} Okay, and that raises the issue of reset and options from outside a
} function needing to be retained as part of the function definition.

OK, I think I get it now, and sorry for being a bit dense before.  You
are talking about a situation where:

- There is a script file of some sort, meant to be read by "source".
- That script file doesn't just do something once (like assign to a
  bunch of environment variables) and go away; instead, it defines
  shell functions that are intended to be used interactively later.
- Those shell functions make the reasonable assumption that if the
  function defintion was parsed by, e.g., a POSIX-compatible shell,
  then the shell will still be POSIX-compatible when the function is
  actually run.

Leaving aside for the moment the fact that it's entirely possible to
parse a function definition with NO_SH_WORD_SPLIT that won't then work
without SH_WORD_SPLIT, so "the setopts in effect at parse time" might
not be a valid representation of "the setopts needed at run time", I
agree that it might be useful to be able to cause a specific set of
options to become set before entry to a given function.

This gets very tricky, however, when applied to functions that are
actually designed for zsh.  An implicit preset of options implies an
implicit restore later, but a zsh function which deliberately does
NOT "setopt localoptions" is expecting any setopt it executes to
persist after the function exits.  It'll be complicated to keep track
of which options should be restored because of implicit preset, and
which ones should not be restored because of explicit change.

Regardless I don't think we'd want to do this as yet another setopt.
It's potential can of worms along the lines of ALL_EXPORT, in that
one can end up accidentally applying a specific option context to a
function definition where one didn't (or shouldn't) mean to do so.
I'd lean toward something that works like the "emulation" global,
which is still localized to function scope but which is changable
only by the "emulate" command.

Combined with allowing "emulate" create a scope and run a command
in that scope, is there anything else you'd need?

PWS, any comment on how difficult it would be to extend "emulate" in
this way?  Would it require promoting emulate to a keyword?

} So I'm thinking an Eprog WC_OPTSPACE instruction which can be inserted
} at some point arounf execfuncdef() time, which is stored with a value
} which points to the opts value to be put into effect and which, when
} encountered, adjusts the current funcstack entry to hold the current
} opts before replacing opts with the one referenced in the Eprog.

This part sounds fine to me, but I've never been the expert on Eprogs.

Re: sourcing a sh file in zsh

[Phil Pennock]

On 2009-01-17 at 12:15 -0800, Bart Schaefer wrote:
> This gets very tricky, however, when applied to functions that are
> actually designed for zsh.  An implicit preset of options implies an
> implicit restore later, but a zsh function which deliberately does
> NOT "setopt localoptions" is expecting any setopt it executes to
> persist after the function exits.  It'll be complicated to keep track
> of which options should be restored because of implicit preset, and
> which ones should not be restored because of explicit change.

True.  My usage scenario was too narrow -- the functions being defined
would never use setopt since they'd be non-zsh functions, except at the
top-level where this new feature would be enabled.

In a sense, I'm creating something akin to a namespace, but being more
of a setopt-value-space.  I'm wondering if it's worth looking at this
from a different angle and whether or not there's something along the
lines of a namespace concept which it's worth implementing, with setopts
being local to a namespace and being able to source a script in another
namespace, created at source time.

Now, a separate pair of items of things I should explore when I have
time, as approaches, tie into this but I hadn't noticed the connection
before (down-side of what is mostly an unordered collection of crazy
ideas worked in with small nits):

----------------------------8< cut here >8------------------------------
Namespaced variables?
  Default is 'main'
  znamespace valid-var-name   to change current, reverts on return from scope?
  Case-insensitive, so that you can use consistent capitalisation for the entire qualified var-name
  ${namespace.varname} to get others (braces mandatory)
  All zsh builtin params also available in 'zsh' namespace.
  So ${ZSH.SECONDS}, ${zsh.fpath}, etc.
  All existing vars bound as before.
  Can we handle multiple names bound to exact same var, besides the tied FOO/foo case?
    -- only the namespace case-insensitive
  ZSHZLE.* for widget variables which persist, such as killring?
  How to handle assignment?  Make dot valid at parse-time separately?

Alternatively:
  Methods via var.method, so that scalarvar.charlen and sv.bytelen give results.
  Have special namespace variables which have the method return any variable
    within that namespace so that you can chain.
  Thus ${zsh.fpath.count} gives the number of elements in the zsh fpath array.
----------------------------8< cut here >8------------------------------
  (DISCLAIMER: my notes contain crazy ideas)

So, is it instead a case of making setopts namespace-bound, thinking
more carefully about syntax than my rough scratchings above, moving much
of completion into a separate namespace and then have functions called
from outside the namespace switch namespace upon invocation and restore
on departure, mechanics much as I described for setopt, but abstracted
out to a container for whatever semantics we want to make
switchable?

Please, bang this idea about a bit, shoot down or refine as appropriate.

-Phil

Re: sourcing a sh file in zsh

[Peter Stephenson]

On Sat, 17 Jan 2009 12:15:01 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Regardless I don't think we'd want to do this as yet another setopt.
> It's potential can of worms along the lines of ALL_EXPORT, in that
> one can end up accidentally applying a specific option context to a
> function definition where one didn't (or shouldn't) mean to do so.
> I'd lean toward something that works like the "emulation" global,
> which is still localized to function scope but which is changable
> only by the "emulate" command.
> 
> Combined with allowing "emulate" create a scope and run a command
> in that scope, is there anything else you'd need?
> 
> PWS, any comment on how difficult it would be to extend "emulate" in
> this way?  Would it require promoting emulate to a keyword?

I haven't heard anything that sounds particularly difficult, although it
obviously needs a firm grip on what's being set at what point.  I don't
see why emulate should have to behave specially as far as syntax is
concerned, as long as its behaviour is well defined.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 20,  7:48pm, Peter Stephenson wrote:
}
} > Combined with allowing "emulate" create a scope and run a command
} > in that scope, is there anything else you'd need?
} > 
} > PWS, any comment on how difficult it would be to extend "emulate" in
} > this way?  Would it require promoting emulate to a keyword?
} 
} I haven't heard anything that sounds particularly difficult, although it
} obviously needs a firm grip on what's being set at what point.  I don't
} see why emulate should have to behave specially as far as syntax is
} concerned, as long as its behaviour is well defined.

My thought was that emulate behave syntactically something like "exec":

    % var="a string with spaces"
    % emulate sh print -l $var
    a
    string
    with
    spaces

There'd also be an option to go with -L and -R to cause the emulation
mode to become "sticky" in the wordcode for any function defined "in
the scope of" the emulate ("-X" used here as placeholder):

    % emulate -X sh function foo { print -l $@ }
    % foo "more spaces here"
    more
    spaces
    here

It may be going too far to make emulate a syntactic element like that.
So the next best thing is if it works more like "eval":

    % emulate -X sh 'foo() { print -l $@ }'

The most frequent intended usage being

    % emulate -X sh source ~/some_bash_library.sh

which would apply the "sticky emulation" to all the functions that
might be defined by some_bash_library.sh.

Maybe the sticky-option isn't even needed, maybe it should just always
become sticky when called with a command as arguments.

There are all sorts of variants of this that I'd find acceptable.
One that might be esthetically pleasing would be using a -c option,
as if you were running the shell in question:

    % emulate ksh -c 'source ~/some_ksh_library.sh'

That's probably enough examples.

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 6578 bytes --]


On 16 января 2009 21:29:34 Bart Schaefer wrote:
> Another possibility would be to extend the "emulate" command into a
> precommand modifier sort of thing, where arguments following the name
> of the emulation are treated as a command to execute.  Perhaps
> require another option to make this active, to avoid any compatibily
> issue.
>
>   emulate -LRE sh source file.sh
>

What about the following patch. It simply eval's any code after setting 
requested emulation. As a bonus it prints current emulation if no argument 
is specified. Neither should be of compatibility issues as emulate always 
allowed exactly one argument.

Comments?

Index: Doc/Zsh/builtins.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/builtins.yo,v
retrieving revision 1.114
diff -u -p -r1.114 builtins.yo
--- Doc/Zsh/builtins.yo	18 Dec 2008 09:49:04 -0000	1.114
+++ Doc/Zsh/builtins.yo	24 Jan 2009 15:40:16 -0000
@@ -338,8 +338,11 @@ cindex(compatibility, csh)
 cindex(sh, compatibility)
 cindex(ksh, compatibility)
 cindex(csh, compatibility)
-item(tt(emulate) [ tt(-LR) ] {tt(zsh)|tt(sh)|tt(ksh)|tt(csh)})(
-Set up zsh options to emulate the specified shell as much as possible.
+item(tt(emulate) [ tt(-ELR) ] [ {tt(zsh)|tt(sh)|tt(ksh)|tt(csh)} [ tt(arg) 
... ] ])(
+Without any argument print current emulation mode.
+
+With single argument set up zsh options to emulate the specified shell
+as much as possible.
 bf(csh) will never be fully emulated.
 If the argument is not one of the shells listed above, tt(zsh)
 will be used as a default; more precisely, the tests performed on the
@@ -351,12 +354,19 @@ the section `Compatibility' in zmanref(z
 ifnzman(\
 noderef(Compatibility)
 )\
-.  If the tt(-R) option is given, all options
+.
+
+If the tt(-E) option is given, arguments that follow
+emulation mode are evaluated using tt(eval) after setting requested
+emulation. 
+If the tt(-R) option is given, all options
 are reset to their default value corresponding to the specified emulation
 mode, except for certain options describing the interactive
 environment; otherwise, only those options likely to cause portability
-problems in scripts and functions are altered.  If the tt(-L) option
-is given, the options tt(LOCAL_OPTIONS) and tt(LOCAL_TRAPS) will be set as
+problems in scripts and functions are altered.  The tt(-L) option together
+with tt(-E) causes emulation mode to be restored after evaluating the rest
+of arguments; without tt(-E), the options tt(LOCAL_OPTIONS) and 
tt(LOCAL_TRAPS)
+will be set as
 well, causing the effects of the tt(emulate) command and any tt(setopt) and
 tt(trap) commands to be local to the immediately surrounding shell
 function, if any; normally these options are turned off in all emulation
Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.218
diff -u -p -r1.218 builtin.c
--- Src/builtin.c	12 Nov 2008 12:57:26 -0000	1.218
+++ Src/builtin.c	24 Jan 2009 15:40:17 -0000
@@ -58,7 +58,7 @@ static struct builtin builtins[] =
     BUILTIN("disable", 0, bin_enable, 0, -1, BIN_DISABLE, "afmrs", NULL),
     BUILTIN("disown", 0, bin_fg, 0, -1, BIN_DISOWN, NULL, NULL),
     BUILTIN("echo", BINF_SKIPINVALID, bin_print, 0, -1, BIN_ECHO, "neE", 
"-"),
-    BUILTIN("emulate", 0, bin_emulate, 1, 1, 0, "LR", NULL),
+    BUILTIN("emulate", 0, bin_emulate, 0, -1, 0, "ELR", NULL),
     BUILTIN("enable", 0, bin_enable, 0, -1, BIN_ENABLE, "afmrs", NULL),
     BUILTIN("eval", BINF_PSPECIAL, bin_eval, 0, -1, BIN_EVAL, NULL, NULL),
     BUILTIN("exit", BINF_PSPECIAL, bin_break, 0, 1, BIN_EXIT, NULL, NULL),
@@ -4744,24 +4744,12 @@ bin_dot(char *name, char **argv, UNUSED(
     return ret ? ret : lastval;
 }
 
-/**/
-int
-bin_emulate(UNUSED(char *nam), char **argv, Options ops, UNUSED(int func))
-{
-    emulate(*argv, OPT_ISSET(ops,'R'));
-    if (OPT_ISSET(ops,'L'))
-	opts[LOCALOPTIONS] = opts[LOCALTRAPS] = 1;
-    return 0;
-}
-
-/* eval: simple evaluation */
-
-/**/
-mod_export int ineval;
+/*
+ * common for bin_emulate and bin_eval
+ */
 
-/**/
-int
-bin_eval(UNUSED(char *nam), char **argv, UNUSED(Options ops), UNUSED(int 
func))
+static int
+eval(char **argv)
 {
     Eprog prog;
     char *oscriptname = scriptname;
@@ -4838,6 +4826,83 @@ bin_eval(UNUSED(char *nam), char **argv,
     return lastval;
 }
 
+/* emulate: set emulation mode and optionally evaluate shell code */
+
+/**/
+int
+bin_emulate(UNUSED(char *nam), char **argv, Options ops, UNUSED(int func))
+{
+    int opt_E = OPT_ISSET(ops, 'E');
+    int opt_L = OPT_ISSET(ops, 'L');
+    int opt_R = OPT_ISSET(ops, 'R');
+    int saveemulation ;
+    int ret;
+    char saveopts[OPT_SIZE];
+
+    /* without arguments just print current emulation */
+    if (!*argv) {
+	if (opt_E || opt_L || opt_R) {
+	    zwarnnam("emulate", "not enough arguments");
+	    return 1;
+	}
+
+	printf("%s\n", emulation == EMULATE_CSH ? "csh" :
+		       emulation == EMULATE_KSH ? "ksh" :
+		       emulation == EMULATE_SH  ? "sh" :
+		       "zsh");
+	return 0;
+    }
+
+    /* with single argument set current emulation */
+    if (!*(argv+1)) {
+	if (opt_E) {
+	    zwarnnam("emulate", "not enough arguments");
+	    return 1;
+	}
+
+	emulate(*argv, OPT_ISSET(ops,'R'));
+	if (OPT_ISSET(ops,'L'))
+	    opts[LOCALOPTIONS] = opts[LOCALTRAPS] = 1;
+	return 0;
+    }
+
+    /* If more than one argument is given, first one is emulation
+     * and subsequent are eval'ed using specified emulation mode.
+     * If option -L is given, emulation restored after evaluation.
+     */
+    if (!opt_E) {
+	zwarnnam("emulate", "too many arguments");
+	return 1;
+    }
+
+    if (opt_L) {
+	memcpy(saveopts, opts, sizeof(opts));
+	saveemulation = emulation;
+    }
+    emulate(*argv, OPT_ISSET(ops,'R'));
+    ret = eval(argv+1);
+    if (opt_L) {
+	/* restore all shell options except PRIVILEGED and RESTRICTED */
+	saveopts[PRIVILEGED] = opts[PRIVILEGED];
+	saveopts[RESTRICTED] = opts[RESTRICTED];
+	memcpy(opts, saveopts, sizeof(opts));
+	emulation = saveemulation;
+    }
+    return ret;
+}
+
+/* eval: simple evaluation */
+
+/**/
+mod_export int ineval;
+
+/**/
+int
+bin_eval(UNUSED(char *nam), char **argv, UNUSED(Options ops), UNUSED(int 
func))
+{
+    return eval(argv);
+}
+
 static char *zbuf;
 static int readfd;
 


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: sourcing a sh file in zsh

[Peter Stephenson]

On Sat, 24 Jan 2009 18:59:27 +0300
Andrey Borzenkov <arvidjaar@gmail.com> wrote:
> On 16 января 2009 21:29:34 Bart Schaefer wrote:
> > Another possibility would be to extend the "emulate" command into a
> > precommand modifier sort of thing, where arguments following the name
> > of the emulation are treated as a command to execute.  Perhaps
> > require another option to make this active, to avoid any compatibily
> > issue.
> >
> >   emulate -LRE srh source file.sh
> >
> 
> What about the following patch. It simply eval's any code after setting 
> requested emulation. As a bonus it prints current emulation if no argument 
> is specified. Neither should be of compatibility issues as emulate always 
> allowed exactly one argument.
> 
> Comments?

Certainly looks like it's going the right way---as you say there
shouldn't be any compatibility issues, so we can probably roll it out
and see what happens.  You need quotes for defining functions in an
emulation, or to creating a separate file.  Hmm... I wonder if we should
allow code from stdin for things like:

emulate -LRE sh <<HERE
function foo {
  ...
}
HERE

?

It would be good to get some tests, even ideas for tests, to tease out
any subtle cases where things are/aren't being set incorrectly.


There's still the issue of retaining the emulation mode for functions
defined in that mode; I agree we really ought to tackle that to make the
new functionality properly useful.  I expect Bart has ideas on the next
step; how far will we need to go?  The simplest idea would be for shell
functions to get a flag giving the emulation mode (with the default
being don't change emulation as now); we could set a global flag while
the "emulate" eval is being run that the mode flag for a function needs
to be inherited from the emulation.  Would this need another flag to
"emulate", or would it be good enough always to inherit the emulation
mode when "emulate -E" is in effect?  I don't think functions should
*always* inherit the emulation because it changes the current behaviour
too much---we would be resetting a whole load of options before every
function call.

I would guess we don't want to apply it to autoloaded functions which
already have their own rules.  We can also add commands to tweak the
emulation mode for individual functions, but that can be done
separately.

We do need to be quite clear on the rules to avoid tying ourselves in
knots with things like "emaulate -LRE sh autoload foo".  "foo", in my
interpretation, would not inherit sh behaviour here.  I think nested
"emulate"s should be unproblematic, however.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: sourcing a sh file in zsh

[Richard Hartmann]

On Sat, Jan 24, 2009 at 18:38, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
> I don't think functions should
> *always* inherit the emulation because it changes the current behaviour
> too much---we would be resetting a whole load of options before every
> function call.

Me neither. If people think this is useful, it might make sense as an
option, but it's way too magical for my taste.


> I would guess we don't want to apply it to autoloaded functions which
> already have their own rules.  We can also add commands to tweak the
> emulation mode for individual functions, but that can be done
> separately.
>
> We do need to be quite clear on the rules to avoid tying ourselves in
> knots with things like "emaulate -LRE sh autoload foo".  "foo", in my
> interpretation, would not inherit sh behaviour here.  I think nested
> "emulate"s should be unproblematic, however.

Agreed.


Richard

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 24,  6:59pm, Andrey Borzenkov wrote:
}
} >   emulate -LRE sh source file.sh
} 
} What about the following patch. It simply eval's any code after setting 
} requested emulation. As a bonus it prints current emulation if no argument 
} is specified.

This looks a lot like what I had in mind, yes.

} Neither should be of compatibility issues as emulate always 
} allowed exactly one argument.

Agree.

} Comments?

I'm still undecided on (1) whether "-E" should either require or always
imply -L, and (2) whether I like "-c" better, as in

    emulate sh -c 'source file.sh'

As to (1), in what circumstances would you want to eval a command
under emulation and then leave the emulation in place?  If you were
going to leave emulation turned on anyway, why wouldn't you write
it as

    emulate sh
    source file.sh

??

(2) is a nit.  Mainly it affects whether emulate accepts only one argument
to be eval'd.

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 24,  5:38pm, Peter Stephenson wrote:
} Subject: Re: sourcing a sh file in zsh
}
} You need quotes for defining functions in an
} emulation, or to creating a separate file.

Yeah, that was one of my reasons for pondering making it a reserved
word or the like.

} Hmm... I wonder if we should allow code from stdin

That'd at least need another option, otherwise "emulate -E" would always
read standard input, which would be bad if it appeared in a script and
started evaluating whatever was written at it.

} for things like:
} 
} emulate -LRE sh <<HERE
} function foo {
}   ...
} }
} HERE

You can always do

emulate -LRE sh $(<<-\HERE
	function foo {
	 ...
	}
	HERE)

I'm not sure offhand if you could double-quote all of that, easily ...

} There's still the issue of retaining the emulation mode for functions
} defined in that mode; I agree we really ought to tackle that to make
} the new functionality properly useful. I expect Bart has ideas on the
} next step; how far will we need to go?

I refer you to workers/26336 where Phil makes some comments about how
he'd modify the wordcode to handle this.

} [...] we could set a global flag while the "emulate" eval is being
} run that the mode flag for a function needs to be inherited from the
} emulation. Would this need another flag to "emulate", or would it be
} good enough always to inherit the emulation mode when "emulate -E" is
} in effect?

I think the latter.

} I don't think functions should *always* inherit the emulation because
} it changes the current behaviour too much---we would be resetting a
} whole load of options before every function call.

I agree.

} I would guess we don't want to apply it to autoloaded functions which
} already have their own rules.

Also agreed, which would imply:

} We do need to be quite clear on the rules to avoid tying ourselves in
} knots with things like "emaulate -LRE sh autoload foo".  "foo", in my
} interpretation, would not inherit sh behaviour here.

Re: sourcing a sh file in zsh

[Richard Hartmann]

On Sun, Jan 25, 2009 at 00:11, Bart Schaefer <schaefer@brasslantern.com> wrote:

>    emulate sh -c 'source file.sh'

While a command and eval are not the same, -c might make more
sense as it's more natural to read.


> (2) is a nit.  Mainly it affects whether emulate accepts only one argument
> to be eval'd.

  emulate sh -c 'foo' -c 'bar'

might be an option, but I wonder if people would use that.


Richard

PS: Having `emulate` return the current emulation mode is a neat idea.

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 7086 bytes --]

On 25 января 2009 02:11:13 Bart Schaefer wrote:
>
> I'm still undecided on (1) whether "-E" should either require or
> always imply -L, and (2) whether I like "-c" better, as in
>
>     emulate sh -c 'source file.sh'
>
> As to (1), in what circumstances would you want to eval a command
> under emulation and then leave the emulation in place?  If you were
> going to leave emulation turned on anyway, why wouldn't you write
> it as
>
>     emulate sh
>     source file.sh
>
> ??
>

I did not want to shut the door. But you are right, implying -L makes both 
code and  documentation simpler. I do not think requiring it makes sense - -
LE implies that -E is possible, which is not.

I personally would flag -LE as error then for the same reason as well as 
do not confuse user who reads manuals. Let -L just always set 
LOCAL_OPTIONS.

> (2) is a nit.  Mainly it affects whether emulate accepts only one
> argument to be eval'd.

Note that

a) -c cannot be an option. It would be yet another argument. Not a 
problem (actually, it even simplifies parsing).

b) this makes impression that

	emulate sh -c "some command" arg1 arg2 ...

is valid (it is with shell, is not it?)  This could be added later if 
someone finds usage case.

Yes, I am more and more inclined to make it "-c ..."  ... OK to commit?

Index: Doc/Zsh/builtins.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/builtins.yo,v
retrieving revision 1.114
diff -u -p -r1.114 builtins.yo
--- Doc/Zsh/builtins.yo	18 Dec 2008 09:49:04 -0000	1.114
+++ Doc/Zsh/builtins.yo	25 Jan 2009 08:37:08 -0000
@@ -338,8 +338,11 @@ cindex(compatibility, csh)
 cindex(sh, compatibility)
 cindex(ksh, compatibility)
 cindex(csh, compatibility)
-item(tt(emulate) [ tt(-LR) ] {tt(zsh)|tt(sh)|tt(ksh)|tt(csh)})(
-Set up zsh options to emulate the specified shell as much as possible.
+item(tt(emulate) [ tt(-LR) ] [ {tt(zsh)|tt(sh)|tt(ksh)|tt(csh)} [ tt(-c) 
tt(arg) ] ])(
+Without any argument print current emulation mode.
+
+With single argument set up zsh options to emulate the specified shell
+as much as possible.
 bf(csh) will never be fully emulated.
 If the argument is not one of the shells listed above, tt(zsh)
 will be used as a default; more precisely, the tests performed on the
@@ -351,16 +354,21 @@ the section `Compatibility' in zmanref(z
 ifnzman(\
 noderef(Compatibility)
 )\
-.  If the tt(-R) option is given, all options
+.
+
+If tt(-c) tt(arg) is given, evaluate tt(arg) after temporary setting
+requested emulation. Emulation will be restored before tt(emulate) returns.
+
+If the tt(-R) option is given, all options
 are reset to their default value corresponding to the specified emulation
 mode, except for certain options describing the interactive
 environment; otherwise, only those options likely to cause portability
-problems in scripts and functions are altered.  If the tt(-L) option
-is given, the options tt(LOCAL_OPTIONS) and tt(LOCAL_TRAPS) will be set as
+problems in scripts and functions are altered.  If the tt(-L) option is 
given,
+the options tt(LOCAL_OPTIONS) and tt(LOCAL_TRAPS) will be set as
 well, causing the effects of the tt(emulate) command and any tt(setopt) and
 tt(trap) commands to be local to the immediately surrounding shell
 function, if any; normally these options are turned off in all emulation
-modes except tt(ksh).
+modes except tt(ksh). The tt(-L) and tt(-c) are mutually exclusive.
 )
 findex(enable)
 cindex(enabling commands)
Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.218
diff -u -p -r1.218 builtin.c
--- Src/builtin.c	12 Nov 2008 12:57:26 -0000	1.218
+++ Src/builtin.c	25 Jan 2009 08:37:08 -0000
@@ -58,7 +58,7 @@ static struct builtin builtins[] =
     BUILTIN("disable", 0, bin_enable, 0, -1, BIN_DISABLE, "afmrs", NULL),
     BUILTIN("disown", 0, bin_fg, 0, -1, BIN_DISOWN, NULL, NULL),
     BUILTIN("echo", BINF_SKIPINVALID, bin_print, 0, -1, BIN_ECHO, "neE", 
"-"),
-    BUILTIN("emulate", 0, bin_emulate, 1, 1, 0, "LR", NULL),
+    BUILTIN("emulate", 0, bin_emulate, 0, 3, 0, "LR", NULL),
     BUILTIN("enable", 0, bin_enable, 0, -1, BIN_ENABLE, "afmrs", NULL),
     BUILTIN("eval", BINF_PSPECIAL, bin_eval, 0, -1, BIN_EVAL, NULL, NULL),
     BUILTIN("exit", BINF_PSPECIAL, bin_break, 0, 1, BIN_EXIT, NULL, NULL),
@@ -4744,24 +4744,12 @@ bin_dot(char *name, char **argv, UNUSED(
     return ret ? ret : lastval;
 }
 
-/**/
-int
-bin_emulate(UNUSED(char *nam), char **argv, Options ops, UNUSED(int func))
-{
-    emulate(*argv, OPT_ISSET(ops,'R'));
-    if (OPT_ISSET(ops,'L'))
-	opts[LOCALOPTIONS] = opts[LOCALTRAPS] = 1;
-    return 0;
-}
-
-/* eval: simple evaluation */
-
-/**/
-mod_export int ineval;
+/*
+ * common for bin_emulate and bin_eval
+ */
 
-/**/
-int
-bin_eval(UNUSED(char *nam), char **argv, UNUSED(Options ops), UNUSED(int 
func))
+static int
+eval(char **argv)
 {
     Eprog prog;
     char *oscriptname = scriptname;
@@ -4838,6 +4826,79 @@ bin_eval(UNUSED(char *nam), char **argv,
     return lastval;
 }
 
+/* emulate: set emulation mode and optionally evaluate shell code */
+
+/**/
+int
+bin_emulate(UNUSED(char *nam), char **argv, Options ops, UNUSED(int func))
+{
+    int opt_L = OPT_ISSET(ops, 'L');
+    int opt_R = OPT_ISSET(ops, 'R');
+    int saveemulation ;
+    int ret;
+    char saveopts[OPT_SIZE];
+
+    /* without arguments just print current emulation */
+    if (!*argv) {
+	if (opt_L || opt_R) {
+	    zwarnnam("emulate", "not enough arguments");
+	    return 1;
+	}
+
+	printf("%s\n", emulation == EMULATE_CSH ? "csh" :
+		       emulation == EMULATE_KSH ? "ksh" :
+		       emulation == EMULATE_SH  ? "sh" :
+		       "zsh");
+	return 0;
+    }
+
+    /* with single argument set current emulation */
+    if (!argv[1]) {
+	emulate(*argv, OPT_ISSET(ops,'R'));
+	if (OPT_ISSET(ops,'L'))
+	    opts[LOCALOPTIONS] = opts[LOCALTRAPS] = 1;
+	return 0;
+    }
+
+    /* If "-c command" is given, evaluate command using specified
+     * emulation mode.
+     */
+    if (strcmp(argv[1], "-c")) {
+	zwarnnam("emulate", "unknown argument %s", argv[1]);
+	return 1;
+    }
+
+    if (!argv[2]) {
+	zwarnnam("emulate", "not enough arguments");
+	return 1;
+    }
+
+    if (opt_L) {
+	zwarnnam("emulate", "option -L incompatible with -c");
+	return 1;
+    }
+
+    memcpy(saveopts, opts, sizeof(opts));
+    saveemulation = emulation;
+    emulate(*argv, OPT_ISSET(ops,'R'));
+    ret = eval(argv+2);
+    memcpy(opts, saveopts, sizeof(opts));
+    emulation = saveemulation;
+    return ret;
+}
+
+/* eval: simple evaluation */
+
+/**/
+mod_export int ineval;
+
+/**/
+int
+bin_eval(UNUSED(char *nam), char **argv, UNUSED(Options ops), UNUSED(int 
func))
+{
+    return eval(argv);
+}
+
 static char *zbuf;
 static int readfd;
 



[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

On 25 января 2009 02:26:43 Bart Schaefer wrote:
> On Jan 24,  5:38pm, Peter Stephenson wrote:
> } Subject: Re: sourcing a sh file in zsh
> }
> } You need quotes for defining functions in an
> } emulation, or to creating a separate file.
>
> Yeah, that was one of my reasons for pondering making it a reserved
> word or the like.
>

If you are defining function in a *zsh* script, why not simply define it 
as *zsh* function?

While I see usage cases for sourcing file under specific emulation (and 
probably making it "sticky" for functions defined in it), I really do 
not see why would you want to define a csh function in a zsh script.

> } Hmm... I wonder if we should allow code from stdin
>
> That'd at least need another option, otherwise "emulate -E" would
> always read standard input, which would be bad if it appeared in a
> script and started evaluating whatever was written at it.
>
> } for things like:
> }
> } emulate -LRE sh <<HERE
> } function foo {
> }   ...
> } }
> } HERE
>
> You can always do
>
> emulate -LRE sh $(<<-\HERE
> 	function foo {
> 	 ...
> 	}
> 	HERE)
>

Actually you can always do

foo=$(<<-\HERE
 	function foo {
 	 ...
 	}
 	HERE)
emulate -c sh $foo


> I'm not sure offhand if you could double-quote all of that, easily
> ...
>
> } There's still the issue of retaining the emulation mode for
> functions } defined in that mode; I agree we really ought to tackle
> that to make } the new functionality properly useful. I expect Bart
> has ideas on the } next step; how far will we need to go?
>
> I refer you to workers/26336 where Phil makes some comments about how
> he'd modify the wordcode to handle this.
>

Assuming this is supported by zsh internals it is trivial to add option 
to emulate to set "sticky" bit during code evaluation.



[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: sourcing a sh file in zsh

[Richard Hartmann]

On Sun, Jan 25, 2009 at 09:56, Andrey Borzenkov <arvidjaar@gmail.com> wrote:

> Actually you can always do
>
> foo=$(<<-\HERE
>        function foo {
>         ...
>        }
>        HERE)
> emulate -c sh $foo

The problem with that is that it's not immediately apparent that this
function is intended to be executed in a different emulation.
Bart's suggestion is a lot easier to read for a third party and will
thus create less errors.


Richard

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

On 25 января 2009 13:25:34 Richard Hartmann wrote:
> On Sun, Jan 25, 2009 at 09:56, Andrey Borzenkov <arvidjaar@gmail.com> 
wrote:
> > Actually you can always do
> >
> > foo=$(<<-\HERE
> >        function foo {
> >         ...
> >        }
> >        HERE)
> > emulate -c sh $foo
>
> The problem with that is that it's not immediately apparent that this
> function is intended to be executed in a different emulation.
> Bart's suggestion is a lot easier to read for a third party and will
> thus create less errors.
>

I asked already - why would you want to create non-zsh function inline 
in zsh script?

Anyway, if you really need it

emulate -? sh -c "${$(<<-\HERE
function foo {
...
}
HERE
)}"



[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: sourcing a sh file in zsh

[Richard Hartmann]

On Sun, Jan 25, 2009 at 11:41, Andrey Borzenkov <arvidjaar@gmail.com> wrote:

> I asked already - why would you want to create non-zsh function inline
> in zsh script?

Personally, I don't. I am simply giving feedback on what I think would be
a clean implementation and try to think of problems which might arise
in the future, either from syntax choices or from user behaviour.


> emulate -? sh -c "${$(<<-\HERE

What is the -? for?


Richard

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 25, 11:40am, Andrey Borzenkov wrote:
}
} [...] you are right, implying -L makes
} both code and documentation simpler. [...]
} 
} I personally would flag -LE as error then for the same reason as well
} as do not confuse user who reads manuals. Let -L just always set
} LOCAL_OPTIONS.

Under the assumption that we carry through and cause the emulation
mode or possibly other options to be automatically saved/restored
for any functions defined in the emulate scope, the effect is going
to be quite similar to that of setting LOCAL_OPTIONS globally.

Consequently I think it makes little difference whether -L/-c are
mutually exclusive; I'm OK with it either way.
 
} 	emulate sh -c "some command" arg1 arg2 ...
} 
} is valid (it is with shell, is not it?)  This could be added later if 
} someone finds usage case.

That's an interesting one.  Yes, I could envision a case where one may
want to have a local $argv in the emulate scope.
 
} Yes, I am more and more inclined to make it "-c ..."  ... OK to commit?

I think so.

On Jan 25, 11:56am, Andrey Borzenkov wrote:
}
} On 25 Jan 2009 02:26:43 Bart Schaefer wrote:
} > On Jan 24,  5:38pm, Peter Stephenson wrote:
} > } Subject: Re: sourcing a sh file in zsh
} > }
} > } You need quotes for defining functions in an
} > } emulation, or to creating a separate file.
} >
} > Yeah, that was one of my reasons for pondering making it a reserved
} > word or the like.
} 
} If you are defining function in a *zsh* script, why not simply define it 
} as *zsh* function?

    emulate sh
    # ...
    emulate zsh -c "${$(<<\HERE
    function ...
    HERE)}"
    # ...

Yeah, that's a bit contrived.

-- 

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

On 25 января 2009 15:20:09 Bart Schaefer wrote:
>
> } Yes, I am more and more inclined to make it "-c ..."  ... OK to
> commit?
>
> I think so.
>

Committed. 

> } If you are defining function in a *zsh* script, why not simply
> define it } as *zsh* function?
>
>     emulate sh
>     # ...
>     emulate zsh -c "${$(<<\HERE
>     function ...
>     HERE)}"
>     # ...
>
> Yeah, that's a bit contrived.

Hmm ... not convinced by this example but I realize now that if sticky 
emulation is going to be implemented, it could be useful to define 
sticky *zsh* function in zsh script.

I still believe using option is the most simple and straightforward way 
to do it. For external script library (which is unaware of zsh in any 
case) it is simple

	emulate sh -c "setopt sticky_emulation; source /my/shell/library.sh"

and zsh script can obviously do

	setopt sticky_emulation
	function zsh_func {
		...
	}
	unsetopt sticky_emulation

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: sourcing a sh file in zsh

[Peter Stephenson]

On Sat, 24 Jan 2009 15:26:43 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> } There's still the issue of retaining the emulation mode for functions
> } defined in that mode; I agree we really ought to tackle that to make
> } the new functionality properly useful. I expect Bart has ideas on the
> } next step; how far will we need to go?
> 
> I refer you to workers/26336 where Phil makes some comments about how
> he'd modify the wordcode to handle this.

I can't offhand think of a case that needs to be that complicated.
Either you run a chunk of code using "emulate", or you have a shell
function, possibly defined in such a chunk, where sticking the flags in
the shfunc structure for future reference is good enough.  Where would
you need something more than that?

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: sourcing a sh file in zsh

[Phil Pennock]

On 2009-01-25 at 21:39 +0000, Peter Stephenson wrote:
> On Sat, 24 Jan 2009 15:26:43 -0800
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> > } There's still the issue of retaining the emulation mode for functions
> > } defined in that mode; I agree we really ought to tackle that to make
> > } the new functionality properly useful. I expect Bart has ideas on the
> > } next step; how far will we need to go?
> > 
> > I refer you to workers/26336 where Phil makes some comments about how
> > he'd modify the wordcode to handle this.
> 
> I can't offhand think of a case that needs to be that complicated.
> Either you run a chunk of code using "emulate", or you have a shell
> function, possibly defined in such a chunk, where sticking the flags in
> the shfunc structure for future reference is good enough.  Where would
> you need something more than that?

I need to sit down and trace out the mechanics of the existing code to
understand the ways that invocation of a function can occur.  What I
wrote was:

----------------------------8< cut here >8------------------------------
The problem is how to decide if a function needs a new option; I haven't
looked too closely but I believe that functions are invoked from each
other directly via the Eprog code, rather than dispatch through the
shfunctab shfunc entries, so it's not viable to go via extending shfunc
(pity).
----------------------------8< cut here >8------------------------------

If I was wrong, and all function invocation goes via the shfunc entries,
then (a) that's much easier and (b) I'm very happy.

-Phil

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 25,  9:26pm, Andrey Borzenkov wrote:
}
} Hmm ... not convinced by this example but I realize now that if sticky 
} emulation is going to be implemented, it could be useful to define 
} sticky *zsh* function in zsh script.
} 
} I still believe using option is the most simple and straightforward way 
} to do it.

I have a couple of quibbles with doing it with a setopt.

The first is one I already mentioned:
> It's potential can of worms along the lines of ALL_EXPORT, in that
> one can end up accidentally applying a specific option context to a
> function definition where one didn't (or shouldn't) mean to do so.

It's also unclear (to me) what to do with that setopt when one later
encounters "emulate -R".

I still think I'd rather that this be a property of "emulate ... -c"
rather than an independent setopt.

Some remars on your examples ...

} For external script library (which is unaware of zsh in any 
} case) it is simple
} 
}   emulate sh -c "setopt sticky_emulation; source /my/shell/library.sh"

This doesn't work, does it?  "emulate sh -c ..." does not imply a full
setopts reset at the end of the eval, only a reset of the emulation
mode.  Also, are you implying that "emulate sh" would implictly unset
that setopt, requiring that it be re-asserted inside the eval?

} and zsh script can obviously do
} 
} 	setopt sticky_emulation
} 	function zsh_func {
} 		...
} 	}
} 	unsetopt sticky_emulation

The trouble with this is that it's not obvious what emulation is being
made sticky.  In the same way that someone using zsh in "native mode"
might want to source a bash script, someone using zsh linked as sh may
want to source a zsh script.  The script author is either going to have
to explicitly call "emulate zsh -c ..." as in your first example, or
end up writing 

    local emu=$(emulation)
    [[ $emu != zsh ]] && emulate zsh 
    { setop sticky_emulation
      ...
    } always {
      unsetopt sticky_emulation
      emulate $emu
    }

It just seems so much cleaner to wrap it all up in "emulate" in the
first place.

Re: sourcing a sh file in zsh

[Richard Hartmann]

On Mon, Jan 26, 2009 at 08:51, Bart Schaefer <schaefer@brasslantern.com> wrote:

> I still think I'd rather that this be a property of "emulate ... -c"
> rather than an independent setopt.

Agreed.


Richard

Re: sourcing a sh file in zsh

[Peter Stephenson]

On Sun, 25 Jan 2009 22:18:19 -0800
Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> On 2009-01-25 at 21:39 +0000, Peter Stephenson wrote:
> > On Sat, 24 Jan 2009 15:26:43 -0800
> > Bart Schaefer <schaefer@brasslantern.com> wrote:
> > > } There's still the issue of retaining the emulation mode for functions
> > > } defined in that mode; I agree we really ought to tackle that to make
> > > } the new functionality properly useful. I expect Bart has ideas on the
> > > } next step; how far will we need to go?
> > > 
> > > I refer you to workers/26336 where Phil makes some comments about how
> > > he'd modify the wordcode to handle this.
> > 
> > I can't offhand think of a case that needs to be that complicated.
> > Either you run a chunk of code using "emulate", or you have a shell
> > function, possibly defined in such a chunk, where sticking the flags in
> > the shfunc structure for future reference is good enough.  Where would
> > you need something more than that?
> 
> I need to sit down and trace out the mechanics of the existing code to
> understand the ways that invocation of a function can occur.  What I
> wrote was:
> 
> ----------------------------8< cut here >8------------------------------
> The problem is how to decide if a function needs a new option; I haven't
> looked too closely but I believe that functions are invoked from each
> other directly via the Eprog code, rather than dispatch through the
> shfunctab shfunc entries, so it's not viable to go via extending shfunc
> (pity).
> ----------------------------8< cut here >8------------------------------
> 
> If I was wrong, and all function invocation goes via the shfunc entries,
> then (a) that's much easier and (b) I'm very happy.

For the case of shell functions, the Eprog is always used via the
placeholder in the Shfunc structure, so far as I know.  I don't see how it
could be otherwise---as all function execution by name goes through
doshfunc(), even if you copy the corresponding Eprog for some reason, it's then
no longer a shell function and the emulation guarantee no longer applies.
doshfunc() was changed a few months ago so we always pass in the Shfunc, so
I don't think there's a difficulty here.

One other point to clear up: what happens when a function defined to be in
sh emulation calls a function that isn't (and that would therefore usually
expect the user's usual options to be in effect when it starts).  I think
the answer is "nothing special"---we are not attempting to guarantee option
settings for all functions, simply to allow a complete set of code to
operate in emulation, and if you're mixing code in this way it's "caveat
executor".  So if you're calling out to a non-sh function, you're thrown
back on the standard zsh answer---if you want to sanitize the options, you
do it explicitly within the function (typically with "emulate -L").  I
don't think that's controversial, I just wanted to be clear about it.

(You *could* mark that other function as "always run in zsh emulation" with
suitable use of "emulate" during its definition or (TBD) by manipulating
flags, I suppose--that ought to be supported, I don't see why we should
exclude native emulation from the system.)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

On 26 января 2009 10:51:21 Bart Schaefer wrote:
>
> } For external script library (which is unaware of zsh in any
> } case) it is simple
> }
> }   emulate sh -c "setopt sticky_emulation; source
> /my/shell/library.sh"
>
> This doesn't work, does it? 

Right, because this option does not exist. Or do you mean something 
else?

> "emulate sh -c ..." does not imply a
> full setopts reset at the end of the eval, only a reset of the
> emulation mode. 

It does. This is exactly what LOCAL_OPTIONS does as well. Except 
LOCAL_OPTIONS also preserves PRIVILEGED and RESTRICTED (which does not 
seem to be documented btw). Initially I made this optional but changed 
on your request :)

> Also, are you implying that "emulate sh" would
> implictly unset that setopt, requiring that it be re-asserted inside
> the eval?
>

Oh, it was just saving extra unsetopt and localizing code :)

> It just seems so much cleaner to wrap it all up in "emulate" in the
> first place.

OK; as I already said, adding extra option to emulate is really the 
minor problem.

Re: sourcing a sh file in zsh

[Phil Pennock]

On 2009-01-25 at 13:41 +0300, Andrey Borzenkov wrote:
> I asked already - why would you want to create non-zsh function inline 
> in zsh script?

Unit tests.

If I, or anybody else, is going to make changes for which bugs could
crop up far and wide if I/we make mistakes, it will have unit tests
written.

Writing those tests is much easier if you can include the tests inline.

Now, those tests could be arranged to end up in external files before
being run, the most obvious mechanism being (assuming -S for Sticky):
  fn==(cat <<-\HERE
	# shell bits
	HERE
	) && emulate -SR sh -c "source $fn"

but being purely inline would be less grotesque to read.

I see the setopt/emulate argument and I can see benefits either way.  I
went with setopt just because it was what I thought of first and it
seems easiest, but that doesn't mean it's best.  What I do like with the
setopt approach is that there's a way for zsh to test if this is
currently the case.

Eg, for bash/zsh portability, you could do:
  if [[ -n $ZSH_VERSION && -o sticky_options ]]
since both support [[ conditional ]] and both support -o as a unary
prefix test (both using it for testing shell options).

-Phil

Re: sourcing a sh file in zsh

[Phil Pennock]

On 2009-01-26 at 12:21 +0000, Peter Stephenson wrote:
> For the case of shell functions, the Eprog is always used via the
> placeholder in the Shfunc structure, so far as I know.  I don't see how it
> could be otherwise---as all function execution by name goes through
> doshfunc(), even if you copy the corresponding Eprog for some reason, it's then
> no longer a shell function and the emulation guarantee no longer applies.
> doshfunc() was changed a few months ago so we always pass in the Shfunc, so
> I don't think there's a difficulty here.

You have made me very happy.  Thank you.

(the compressed Eprog stuff has always scared me away)

> One other point to clear up: what happens when a function defined to be in
> sh emulation calls a function that isn't (and that would therefore usually
> expect the user's usual options to be in effect when it starts).  I think
> the answer is "nothing special"---we are not attempting to guarantee option
> settings for all functions, simply to allow a complete set of code to
> operate in emulation, and if you're mixing code in this way it's "caveat
> executor".  So if you're calling out to a non-sh function, you're thrown
> back on the standard zsh answer---if you want to sanitize the options, you
> do it explicitly within the function (typically with "emulate -L").  I
> don't think that's controversial, I just wanted to be clear about it.

Given the possible behaviours I presented in zsh-workers/26336, this
seems problematic.  Either the new option value set is read-only, so
that fails, or the setopt in the zsh callback function change the values
back inside the sh area.

I think that the best thing to do would be to restore the option values
from before crossing that emulate boundary, unless the zsh function
itself has an emulate boundary.

I think that I shall have to write tests covering these cases and post
them before I write code to implement it -- anything else is too fraught
with peril.

-Phil

Re: sourcing a sh file in zsh

[Richard Hartmann]

On Tue, Jan 27, 2009 at 00:07, Phil Pennock
<zsh-workers+phil.pennock@spodhuis.org> wrote:

> What I do like with the
> setopt approach is that there's a way for zsh to test if this is
> currently the case.
>
> Eg, for bash/zsh portability, you could do:
>  if [[ -n $ZSH_VERSION && -o sticky_options ]]
> since both support [[ conditional ]] and both support -o as a unary
> prefix test (both using it for testing shell options).

emulate could simply set $ZSH_STICKY to 'sh' or something
similar could happen. It would give more information than a simple
[[ -o sticky_options ]] and tie into existing mechanisms just as
nicely.


Richard

Re: sourcing a sh file in zsh

[Peter Stephenson]

Phil Pennock wrote:
> > One other point to clear up: what happens when a function defined to be in
> > sh emulation calls a function that isn't (and that would therefore usually
> > expect the user's usual options to be in effect when it starts).  I think
> > the answer is "nothing special"---we are not attempting to guarantee option
> > settings for all functions, simply to allow a complete set of code to
> > operate in emulation, and if you're mixing code in this way it's "caveat
> > executor".  So if you're calling out to a non-sh function, you're thrown
> > back on the standard zsh answer---if you want to sanitize the options, you
> > do it explicitly within the function (typically with "emulate -L").  I
> > don't think that's controversial, I just wanted to be clear about it.
> 
> Given the possible behaviours I presented in zsh-workers/26336, this
> seems problematic.  Either the new option value set is read-only, so
> that fails, or the setopt in the zsh callback function change the values
> back inside the sh area.
> 
> I think that the best thing to do would be to restore the option values
> from before crossing that emulate boundary, unless the zsh function
> itself has an emulate boundary.

This is the road I really don't want to go down.  The system is to
provide a delimited execution environment where you can run code in an
emulation.  It is not to track every change of option at every stage in
the process.  Having the behaviour change bodily at explicit "emulate"s
and at functions marked for emulation should be good enough, and is much
more obvious behaviour than trying to insert magic conversion points.
If you call to a function that doesn't fit in that environment you
should make your own arrangements.

I don't see why options should ever be read-only, and I don't see why
executing a function that changes option values in the parent should be
forbidden from doing so, given that this is explicitly supported zsh
behaviour.  You always have enough rope to hang yourself with a
shell---if you don't want native zsh behaviour seeping out, don't use a
native zsh function, or be explicit with the "emulate -L"s.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: sourcing a sh file in zsh

[Peter Stephenson]

Peter Stephenson wrote:
> I don't see why options should ever be read-only, and I don't see why
> executing a function that changes option values in the parent should be
> forbidden from doing so, given that this is explicitly supported zsh
> behaviour.  You always have enough rope to hang yourself with a
> shell---if you don't want native zsh behaviour seeping out, don't use a
> native zsh function, or be explicit with the "emulate -L"s.

I wasn't very clear about what I *do* think it should do:  that is
simply set and restore options for the new use of emulate, and do
exactly the same for functions marked for emulate behaviour.  (Detecting
when to set and when to use the emulate marking for functions is a bit
more subtle.)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 27, 10:16am, Peter Stephenson wrote:
}
} I wasn't very clear about what I *do* think it should do: that is
} simply set and restore options for the new use of emulate, and
} do exactly the same for functions marked for emulate behaviour.
} (Detecting when to set and when to use the emulate marking for
} functions is a bit more subtle.)

I agree with this, for what that's worth.

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 27,  1:51am, Richard Hartmann wrote:
} Subject: Re: sourcing a sh file in zsh
}
} On Tue, Jan 27, 2009 at 00:07, Phil Pennock
} <zsh-workers+phil.pennock@spodhuis.org> wrote:
} 
} > What I do like with the
} > setopt approach is that there's a way for zsh to test if this is
} > currently the case.

This does raise an interesting edge case.  What happens if one invokes
"eumulate" (without the "-c") inside a sticky emulation context?  Does
it turn off the sticky, or does the new emulation become sticky?

This question needs to be answered whether or not stickiness is
implemented as a setopt.  My inclination is to say that it turns off
the sticky, but in that case "emulate -L" ought to restore stickyness
again.

Maybe the right way to do this is with a stack of emulation contexts.
Using -L or -c creates one, otherwise emulate modifies the current one.
That would parallel the LOCAL_OPTIONS semantics ... and I suppose may
be an argument in favor of adding a setopt, though I'm still leery of
that for other reasons.

} > Eg, for bash/zsh portability, you could do:
} >  if [[ -n $ZSH_VERSION && -o sticky_options ]]
} > since both support [[ conditional ]] and both support -o as a unary
} > prefix test (both using it for testing shell options).
} 
} emulate could simply set $ZSH_STICKY to 'sh' or something

We've already made "emulate" with no arguments return the current
emulation mode.  Perhaps it could append the -c when sticky emulation
is in effect.  That would make [[ $(emulation) = zsh ]] a bit less
useful ... you'd have to do [[ $(emulation) = zsh* ]] ... hmm.

Re: sourcing a sh file in zsh

[Bart Schaefer]

Getting back to this a bit later ...

On Jan 26,  7:49pm, Andrey Borzenkov wrote:
} Subject: Re: sourcing a sh file in zsh
}
} On 26 0x044F0x043D0x04320x04300x04400x044F 2009 10:51:21 Bart Schaefer wrote:
} >
} > }   emulate sh -c "setopt sticky_emulation; source
} > /my/shell/library.sh"
} >
} > This doesn't work, does it? 
} 
} Right, because this option does not exist. Or do you mean something 
} else?

I meant I didn't think it would do what you intended, but:
 
} > "emulate sh -c ..." does not imply a
} > full setopts reset at the end of the eval, only a reset of the
} > emulation mode. 
} 
} It does. This is exactly what LOCAL_OPTIONS does as well.

Obviously I understand that's what LOCAL_OPTIONS does, but I never meant
for that to be what "emulate" did.  Apparently I misread your patch.

I expected an approximate equivalence between

    emulate sh -c "echo foo"

and

    {
     emu=$(emulate)
     emulate sh
     eval "echo foo"
    } always {
     emulate $emu
    }

I *didn't* expect an equivalence to function scopes with respect to any
setopts that aren't normally affected by changing the emulation.

Having said that, however, I'm left undecided as to whether it really
makes any difference.  I admit there are cases where I wished I had
a LOCAL_OPTIONS scope and *not* a "local" parameter scope, a result
for which this form of emulate can be abused.

However, I do think that (if the implementation remains as-is) the
documentation needs to be clearer that the *entire* setopt context is
restored.  Currently it says only that "Emulation will be restored".

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 2831 bytes --]

On 30 of January 2009 07:25:01 Bart Schaefer wrote:
> Getting back to this a bit later ...
>
> On Jan 26,  7:49pm, Andrey Borzenkov wrote:
> } Subject: Re: sourcing a sh file in zsh
> }
> } On 26 0x044F0x043D0x04320x04300x04400x044F 2009 10:51:21 Bart
> Schaefer wrote: } >
> } > }   emulate sh -c "setopt sticky_emulation; source
> } > /my/shell/library.sh"
> } >
> } > This doesn't work, does it?
> }
> } Right, because this option does not exist. Or do you mean something
> } else?
>
> I meant I didn't think it would do what you intended, but:
>
> } > "emulate sh -c ..." does not imply a
> } > full setopts reset at the end of the eval, only a reset of the
> } > emulation mode.
> }
> } It does. This is exactly what LOCAL_OPTIONS does as well.
>
> Obviously I understand that's what LOCAL_OPTIONS does, but I never
> meant for that to be what "emulate" did.  Apparently I misread your
> patch.
>

I apologize for not being clear enough as well.

Discussion actually started with the this code (not exactly, but):

function run_on_sh_mode {
	emulate -L sh
	$*
}

So I intended emulate -c to be replacement for above without possible 
scoping issues (because e.g. startup scripts here do set quite a lot of 
variables).

> I expected an approximate equivalence between
>
>     emulate sh -c "echo foo"
>
> and
>
>     {
>      emu=$(emulate)
>      emulate sh
>      eval "echo foo"
>     } always {
>      emulate $emu
>     }
>
> I *didn't* expect an equivalence to function scopes with respect to
> any setopts that aren't normally affected by changing the emulation.
>

This is doable. I did not quite like it because my primary intention was 
to run foreign code. In which case I definitely would not like this code 
by accident change shell behaviour. So it is sort of sandbox.

Another observation that my intended usage was almost exclusively 
"emulate -R"; in which case to revert this you have to reset *all* 
options anyway.

> Having said that, however, I'm left undecided as to whether it really
> makes any difference.  I admit there are cases where I wished I had
> a LOCAL_OPTIONS scope and *not* a "local" parameter scope, a result
> for which this form of emulate can be abused.
>

I think we need to sort this out before usage sticks.

So let's consider what we get

emulate -L -c "command"
	implicitly set LOCAL_OPTIONS before executing command

emulate [-R] -c "command"
	implicitly do "emulate [-R] oldemulation" after executing command

Do I miss something? Is it what you intend?

> However, I do think that (if the implementation remains as-is) the
> documentation needs to be clearer that the *entire* setopt context is
> restored.  Currently it says only that "Emulation will be restored".

I will update it.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 31, 11:32am, Andrey Borzenkov wrote:
}
} > I *didn't* expect an equivalence to function scopes with respect to
} > any setopts that aren't normally affected by changing the emulation.
} 
} This is doable. I did not quite like it because my primary intention
} was to run foreign code.  In which case I definitely would not like
} this code by accident change shell behaviour.  So it is sort of
} sandbox.

This is a tricky problem.  What, for example, should happen when one
does
    emulate sh -c 'set -x'
??

I'd initally have expected XTRACE to become and remain set.  However,
I can see reasons for doing it either way, so as long as whatever is
chosen is documented I'm OK with it.

We face a similar problem with the "sticky" emulation idea.  Let's
suppose for purposes of illustration that "emulate -S" activates it.

    emulate -S sh -c 'debug() { set -vx }'

(Imagine the debug function is buried in, and used by, some lengthy
piece of /bin/sh code that's being loaded.)  What happens when the
debug function is executed?  An implicit LOCAL_OPTIONS breaks the
intended behavior.

} I think we need to sort this out before usage sticks.

Absolutely.

} So let's consider what we get
} 
} emulate -L -c "command"
} 	implicitly set LOCAL_OPTIONS before executing command

Well, no.  Literally setting LOCAL_OPTIONS has side-effects within other
functions that might be defined by the command.  Everything here has to
be described in terms of "behaves as if".  I suppose we could do it this
way (rope to hang yourself and all that, no different than putting an
"emulate -L" in your .zshrc) but I'm not opposed to -L / -c as mutually
exclusive.

} emulate [-R] -c "command"
} 	implicitly do "emulate [-R] oldemulation" after executing command
} 
} Do I miss something? Is it what you intend?

That's pretty much what my original thought was, yes, but again I can
see the argument both ways.

Re: sourcing a sh file in zsh

[Peter Stephenson]

Bart Schaefer wrote:
> We face a similar problem with the "sticky" emulation idea.  Let's
> suppose for purposes of illustration that "emulate -S" activates it.
> 
>     emulate -S sh -c 'debug() { set -vx }'
> 
> (Imagine the debug function is buried in, and used by, some lengthy
> piece of /bin/sh code that's being loaded.)  What happens when the
> debug function is executed?  An implicit LOCAL_OPTIONS breaks the
> intended behavior.

Although I see you can argue different ways, my view of this was that
this was "just tough".  An "emulate" of this kind specifically sets and
restores options.  If you want option propagation, you shouldn't mix
emulations.

What would be the intention of the code above, anyway?  I think the
answer is it's for debugging in sh emulation.  But then it belongs with
the code you're debugging, doesn't it?

emulate -S sh -c 'debug() { set -vx }
... sh code which calls debug ...
'

Now in *this* case, I think it's clear that the option should
propagate.  But I think that's OK... there's no "emulate" firewall
between the execution of debug and the code around.

To get this right (if that is right), we do need to be careful about how
we handle sticky emulation in functions:  if we're already in the
sticky emulation environment, as here, we don't want to set and restore
options around execution of functions defined in the same emulation
mode.  That would be regardless of which chunks of code they were
defined in, I think, otherwise we have a new type of firewall (between
code chunks rather than emulations) that we don't want.

In other words, your orignal code chunk would be fine if combined with
other code defined in another "emulate sh -c ..." chunk.  I think that's
all we need to get the job done.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Jan 31,  9:10pm, Peter Stephenson wrote:
} Subject: Re: sourcing a sh file in zsh
}
} Bart Schaefer wrote:
} > We face a similar problem with the "sticky" emulation idea.  Let's
} > suppose for purposes of illustration that "emulate -S" activates it.
} > 
} >     emulate -S sh -c 'debug() { set -vx }'
} 
} What would be the intention of the code above, anyway?

Well, suppose I've got this sort of thing in a file somewhere:

debug() { set -vx }
big_hairy_sh_app() {
  if [[ x$1 = x-d ]]; then debug; shift; fi
  do_some_sh_stuff
  do_more_sh_stuff
}
do_some_sh_stuff() { echo whatever }
do_more_sh_stuff() { echo whocares }

The intent would be to run
    emulate -S sh -c 'source big_hairy_sh_app_as_a_function_library.sh'
and them later invoke it with
    big_hairy_sh_app -d
and have all the right emulation settings magically take effect.

If *every* function defined in the sourced file gets an implicit
LOCAL_OPTIONS by virtue of emulation stickyness, then even within the
call to big_hairy_sh_app the embedded call to debug is going to clear
XTRACE and VERBOSE upon returning, and the -d option to the function
library has become useless.

I'm with you that we may just have to say "tough cookies" but we at
least need to understand what we're not allowing.

} To get this right (if that is right), we do need to be careful about
} how } we handle sticky emulation in functions:  if we're already in the
} sticky emulation environment, as here, we don't want to set and restore
} options around execution of functions defined in the same emulation
} mode.

I agree, but that means that "already in the sticky" may have to include
"have entered a function that was originally defined with the sticky."

} That would be regardless of which chunks of code they were
} defined in, I think, otherwise we have a new type of firewall (between
} code chunks rather than emulations) that we don't want.
} 
} In other words, your orignal code chunk would be fine if combined with
} other code defined in another "emulate sh -c ..." chunk.  I think that's
} all we need to get the job done.

Yep, devil in details.

Re: sourcing a sh file in zsh

[Peter Stephenson]

Bart Schaefer wrote:
> If *every* function defined in the sourced file gets an implicit
> LOCAL_OPTIONS by virtue of emulation stickyness

I suppose this is the bone of contention.  I just argued that you only
get the effect of LOCAL_OPTIONS for a function if you weren't already in
that sticky emulation when the function started---which is probably more
useful, I think, giving you a more consistent emulation environment, but
also needs a little more care.  (Nobody is arguing we don't need care, I
realise...)

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 2234 bytes --]

On 31 of January 2009 23:40:43 Bart Schaefer wrote:
> } I think we need to sort this out before usage sticks.
>
> Absolutely.
>
> } So let's consider what we get
> }
> } emulate -L -c "command"
> } 	implicitly set LOCAL_OPTIONS before executing command
>
> Well, no.  Literally setting LOCAL_OPTIONS has side-effects within
> other functions that might be defined by the command.  Everything
> here has to be described in terms of "behaves as if".  I suppose we
> could do it this way (rope to hang yourself and all that, no
> different than putting an "emulate -L" in your .zshrc) but I'm not
> opposed to -L / -c as mutually exclusive.
>

Yes, mea culpa. I did mean "as if LOCAL_OPTIONS were set". IOW restore 
options to the values they had before emulate command.

> } emulate [-R] -c "command"
> } 	implicitly do "emulate [-R] oldemulation" after executing command
> }
> } Do I miss something? Is it what you intend?
>
> That's pretty much what my original thought was, yes, but again I can
> see the argument both ways.

Unfortunately this one probably would be the most confusing 
implementation. This effectively results in options values after emulate 
that corresponds neither to options values before, nor to options set by 
commands inside of emulate -c.

If user has changed any of the option that is touched by emulate (and 
emulate -R would touch every option) before executing emulate -c, return 
from emulate -c would wipe out all changes.

We could try to interpret this as "set of options that were in effect 
before emulate -c as adjusted by options explicitly changed by command". 
This is doable, but this immediately poses another question - what to do 
with emulation mode set inside emulate -c?

Extreme example: emulate sh -c "emulate -R csh"

if we retain options set by "emulate -R csh" on exit from top level 
emulate, we effectively have options for csh emulation, but emulation 
itself is reset to whatever was in effect before (just in case - 
emulation is *not* limited to options unfortunately).

So I still tend to leave things as is (making documentation more clear) 
and if we ever need different behaviour, better add explicit option for 
it.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Feb 1,  9:03pm, Andrey Borzenkov wrote:
}
} So I still tend to leave things as is (making documentation more clear) 
} and if we ever need different behaviour, better add explicit option for 
} it.

A solution to this might be to create a way to force an option to remain
set even across the end of a LOCAL_OPTIONS scope.  E.g., take any option
name and prefix it with "FORCE_" and that puts it in the "surrounding"
scope, ala "typeset -g" for variables.  That would solve some problems
for ordinary function scopes as well.

Pick some other/better word than "force".  And I'm not sure that being
able to climb up exactly one scope is enough, but it's a start.

POSIX and the "&>" operator

[Bart Schaefer]

The following is part of an exchange on the austin-group mailing list.
The assertion is that "foo&>bar" has a well-defined semantics in POSIX
sh and therefore the bash extension (which zsh adopted) to make "&>" a
synonym for "2>&1 >" is in violation of POSIX compliance.

(I tried to send a follow-up to the austin-group about it, but I can't
figure out with what email address they think I'm subscribed, and the
list won't accept mail from non-subscribers.)

This is interesting for two reasons:
(1) I suspect that when zsh is emulating a POSIX shell, it should avoid
treating "&>" as a token and instead parse it as "&" ">"
(2) I'm puzzled by Korn's assertion that some syntax like "&|" is an
"invalid extension", as opposed for example to ">;"


---------- Forwarded message ----------
Date: Thu, 5 Feb 2009 12:19:52 -0500
From: Glenn Fowler <gsf@research.att.com>
To: austin-group-l@opengroup.org
Subject: shell redirection question
Resent-Date: 5 Feb 2009 17:20:33 -0000
Resent-From: austin-group-l@opengroup.org
Resent-To: austin-group-l@opengroup.org


in the posix shell grammar ';' and '&' are syntactically equivalent

so these two commands, modulo one being in the background,
should produce the same output ("stdout" to stdout, "stderr" to stderr):

sh -c "( echo stdout; echo stderr >&2 ) ;>/dev/null; wait"
sh -c "( echo stdout; echo stderr >&2 ) &>/dev/null; wait"

ksh produces the same output for both cases
i.e., it treats ';' and '&' as command separators

bash, even with --posix, and zsh, treat the '&>/dev/null'
as '>/dev/null 2>&1'

is a shell implementation permitted to treat
'&>/dev/null' as '>/dev/null 2>&1' ?

-- Glenn Fowler -- AT&T Research, Florham Park NJ --


--- Forwarded mail from David Korn <dgk@research.att.com>

Date: Thu, 5 Feb 2009 16:52:21 -0500
From: David Korn <dgk@research.att.com>
To: austin-group-l@opengroup.org, chet.ramey@case.edu
Subject:  Re: Re: shell redirection question

cc: chet.ramey@case.edu
Subject: Re: Re: shell redirection question
--------

> Chet Ramey <chet.ramey@case.edu> writes:
> 
> >> is a shell implementation permitted to treat
> >> '&>/dev/null' as '>/dev/null 2>&1' ?
> >
> > This is a documented bash extension.  It's not posix.  As such, it's
> > probably not appropriate for the group to decide.
> 
> Certainly it's not appropriate for the group to decide what extensions
> Bash should make, but here it seems clear that Bash and zsh do not
> conform to the current POSIX spec here.  Either the POSIX spec should
> change, or bash (with --posix) should change.  I prefer the former,
> that is, I suggest modifying POSIX so that "&>" leads to
> implementation-defined behavior.
> 
> 

Of course the group cann't say what extensions any shell should make,
but they can indicate whether an extension is compatible with posix.

When you run ksh93 with the -n option, it produces a warning whenever
it finds &< without a space in between.  The only time I have ever
seen this warning is when a script intended for bash was run so
I would say that &< in any other context is exceedingly rare.

Excluding the ( and ) operators, I have a list below of all the other
two character operator characters and how they are used.

The wording in the standard under token recognition, says, "If the
previous character was uses as part of an operator and the current
character is not quoted and can be used with the current characters
to form an operator, it should be used as part of that (operator)
token."

The grammar section also list which operators are defined double character
operators.  However, the standard doesn't state how extensions
should be made.  ksh93 has used the rule that if the second character
would generate a syntax error, then it is a valid extension.

Maybe, ;<, ;>, &< and &> should produce unspecified behavior without
an interviening space so that these can be used as an extension.
Minimally &> is commonly used in bash so this might be considered
a legal extension.

<<	used by posix
<>	used by posix
<;	valid extension
<|	valid extension
<&	used by posix
><	valid extension
>>	used by posix
>;	valid extension used by ksh93
>|	used by posix
>&	used by posix
;<	invalid extension
;>	invalid extension
;;	used by posix
;|	valid extension
;&	valid extension used by ksh93
|<	invalid extension
|>	invalid extension
|;	valid extension
||	used by posix
|&	valid extension used by ksh93
&<	invalid extension
&>	invalid extension used by bash
&;	valid extension
&|	invalid extension
&&	used by posix

ksh93 also uses <# and ># which are valid extensions.
	

David Korn
dgk@research.att.com


---End of forwarded mail from David Korn <dgk@research.att.com>

-- 

Re: POSIX and the "&>" operator

[Peter Stephenson]

Bart Schaefer wrote:
> The following is part of an exchange on the austin-group mailing list.
> The assertion is that "foo&>bar" has a well-defined semantics in POSIX
> sh and therefore the bash extension (which zsh adopted) to make "&>" a
> synonym for "2>&1 >" is in violation of POSIX compliance.

I think both got it from csh, which is why nobody thought about POSIX
compatibility---apart from the fact that the combination of (i) putting
a new command on the same line as backgrounded expression (ii) not
putting a space after the "&" (iii) starting the following command with
a redirection, while perfectly valid traditional sh syntax, is a very
strange way to write.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: POSIX and the "&>" operator

[Bart Schaefer]

On Feb 7, 12:02am, Peter Stephenson wrote:
}
} I think both got it from csh, which is why nobody thought about POSIX

No, csh uses ">&" with no descriptor numbers around it, which zsh also
supports.  "&>" wasn't added to zsh until around version 2.5 (I forget
the exact timeline) and was added for bash compatibility.

Re: sourcing a sh file in zsh

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On 30 of January 2009 07:25:01 Bart Schaefer wrote:
>
> However, I do think that (if the implementation remains as-is) the
> documentation needs to be clearer that the *entire* setopt context is
> restored.  Currently it says only that "Emulation will be restored".

Like this?

Index: Doc/Zsh/builtins.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/builtins.yo,v
retrieving revision 1.117
diff -u -p -r1.117 builtins.yo
--- Doc/Zsh/builtins.yo 6 Feb 2009 11:03:29 -0000       1.117
+++ Doc/Zsh/builtins.yo 7 Feb 2009 15:41:40 -0000
@@ -357,7 +357,8 @@ noderef(Compatibility)
 .

 If tt(-c) tt(arg) is given, evaluate tt(arg) after temporary setting
-requested emulation. Emulation will be restored before tt(emulate) returns.
+requested emulation. Emulation and all options will be restored to their
+original values before tt(emulate) returns.

 If the tt(-R) option is given, all options
 are reset to their default value corresponding to the specified emulation


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: sourcing a sh file in zsh

[Bart Schaefer]

On Feb 7,  6:45pm, Andrey Borzenkov wrote:
}
} Like this?

Works for me.


} Index: Doc/Zsh/builtins.yo
} +requested emulation. Emulation and all options will be restored to their
} +original values before tt(emulate) returns.

Re: POSIX and the "&>" operator

[Peter Stephenson]

On Fri, 06 Feb 2009 19:51:09 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Feb 7, 12:02am, Peter Stephenson wrote:
> }
> } I think both got it from csh, which is why nobody thought about POSIX
>
> No, csh uses ">&" with no descriptor numbers around it, which zsh also
> supports.  "&>" wasn't added to zsh until around version 2.5 (I forget
> the exact timeline) and was added for bash compatibility.

You're right.

We really ought to have a POSIX emulation separate from sh.  I have a
feeling we've discussed this before.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: POSIX and the "&>" operator

[Vincent Lefevre]

On 2009-02-06 14:50:29 -0800, Bart Schaefer wrote:
> The following is part of an exchange on the austin-group mailing list.
> The assertion is that "foo&>bar" has a well-defined semantics in POSIX
> sh and therefore the bash extension (which zsh adopted) to make "&>" a
> synonym for "2>&1 >" is in violation of POSIX compliance.

IMHO, this is not clear. I agree with David Korn when he says:

----------------------------------------------------------------------
The way the standard is written, the grammar lists operators
that are required.  It doesn't say whether implementations
can support operators that aren't listed.

In section 2.3, token recognition, the standard says:

2.      If the previous character was used as part of an operator
        and the current character is not quoted and can be used
        with the current characters to form an operator, it shall
        be used as part of that (operator) token.


If implementations are allowed to provide additional operators,
then this requires that conforming script leave a space after
every operator character.   This would mean that
        command &> file
is not conforming.
----------------------------------------------------------------------

On 2009-02-07 22:48:39 +0000, Peter Stephenson wrote:
> We really ought to have a POSIX emulation separate from sh.  I have a
> feeling we've discussed this before.

Yes, at least here:
  http://www.zsh.org/mla/workers/2007/msg00852.html
  http://www.zsh.org/mla/workers/2008/msg00342.html

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Modules/attr.c compile error on Mac OS X

[Taro M]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

The new Modules/attr.c does not compile on Mac OS X (and probably others):
gcc -c -I.  -DHAVE_CONFIG_H -DMODULE -Wall -Wmissing-prototypes -O2
-fno-common -o attr..o attr.c
attr.c: In function 'bin_getattr':
attr.c:45: error: too few arguments to function 'listxattr'
attr.c:46: error: too few arguments to function 'getxattr'
attr.c: In function 'bin_setattr':
attr.c:70: error: too few arguments to function 'setxattr'
attr.c: In function 'bin_delattr':
attr.c:84: error: too few arguments to function 'removexattr'
attr.c: In function 'bin_listattr':
attr.c:99: error: too few arguments to function 'listxattr'

[-- Attachment #2: Type: text/html, Size: 834 bytes --]

Re: Modules/attr.c compile error on Mac OS X

[François Revol]

> The new Modules/attr.c does not compile on Mac OS X (and probably
> others):
> gcc -c -I.  -DHAVE_CONFIG_H -DMODULE -Wall -Wmissing-prototypes -O2
> -fno-common -o attr..o attr.c
> attr.c: In function 'bin_getattr':
> attr.c:45: error: too few arguments to function 'listxattr'
> attr.c:46: error: too few arguments to function 'getxattr'
> attr.c: In function 'bin_setattr':
> attr.c:70: error: too few arguments to function 'setxattr'
> attr.c: In function 'bin_delattr':
> attr.c:84: error: too few arguments to function 'removexattr'
> attr.c: In function 'bin_listattr':
> attr.c:99: error: too few arguments to function 'listxattr'

Likely the xattr API OSX supports isn't compatible with the withdrawn-
POSIX-draft implemented in Linux...

Generally speaking xattrs are very different from one OS to another,
and aren't compatible on all API, storage and semantics level.

For example, BeOS has made pervasive use of xattrs for over a decade,
though its are typed (uint32 4CC code) and indexable on BFS, and the
API also allows for a 64bit offset inside it, though it wasn't
implemented in BFS, cf.
http://www.beos.tarman.pl/teksty/bebookpl/The%20Storage%20Kit/AttrFuncs.html

Even on Linux I recall finding 3 different API, some taking void * for
the value field, others using char *...

cf. Linux manpage:
http://linux.die.net/man/2/getxattr

     ssize_t getxattr (const char *path, const char *name,
     void *value, size_t size);
(though the kernel API is different, and there are/were other APIs)

cf. BSD manpage:
http://www.manpagez.com/man/2/getxattr/
     ssize_t
     getxattr(const char *path, const char *name, void *value, size_t
size,
         u_int32_t position, int options);

The OSX ones likely use the BSD API, and should map quite well 1:1 at
the API level to the Linux ones, but I'm not sure about the storage
level though, on Linux the user-accessible namespace is restricted to
"user.*".
Not even mentionning the different semantics of walking interaction
(NOFOLLOW...), or value interpretation (void * vs char * that can be
binary vs char * that means string, vs typed binary value..., or even a
full directory tree as it's possible on Solaris)

Ideally a mangling scheme onto a common name/value space would be
designed to allow mapping every OS xattrs in such a way that the scheme
is idempotent (can be reused on itself multiple times through different
protocols (NFS,extended FTP,...), OS and file system (NTFS, ext2,
BFS...) and still keeping the initial information usable to the
originator).

But that will need a cross-OS coordination, and a lot of work to make
people agree. Been thinking about this for quite some time...

As for zsh, at least it should probably have a separate backend for
each platform for clarity, until such a mangling scheme is devised.

François.

Re: Modules/attr.c compile error on Mac OS X

[Peter Stephenson]

On Sun, 8 Mar 2009 17:21:47 +0900
Taro M <taromeister@gmail.com> wrote:
> The new Modules/attr.c does not compile on Mac OS X (and probably others):

I don't know about the others, and you don't say, but a bit of research
suggests Mac OS X takes some extra arguments.  I don't have Mac OS X so
can't test that; if it still doesn't compile, please could one of the
numerous Mac users provide a patch.  If it compiles but doesn't work,
I will need someone else to research the problem.

General reminder not aimed at anyone particular: as I keep saying, I
really do need more help.  Anyone with knowledge about specific systems
that may need special help is encouraged to provide it; patches are
useful even if incomplete (and preferred to text statements along the
lines of "wouldn't it be better to change ...").  This is supposed to be
a developer's mailing list.  Thanks.

Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.119
diff -u -r1.119 configure.ac
--- configure.ac	3 Mar 2009 15:04:17 -0000	1.119
+++ configure.ac	8 Mar 2009 19:51:18 -0000
@@ -1251,6 +1251,49 @@
 		shl_load shl_unload shl_findsym)
 fi
 
+AH_TEMPLATE([XATTR_EXTRA_ARGS],
+Define if getxattr() etc. require additional MacOS-style arguments)
+if test x$ac_cv_func_getxattr = xyes && test x$ac_cv_header_sys_xattr_h = xyes
+then
+  AC_CACHE_CHECK(if getxattr etc. are Linux-like,
+  zsh_cv_getxattr_linux,
+  [AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <sys/types.h>
+#include <sys/xattr.h>]],
+  [[
+  (void)listxattr("", 0, 0);
+  (void)getxattr("", "", 0, 0);
+  (void)setxattr("", "", "", 0, 0);
+  (void)removexattr("", "");
+  ]])],
+  [zsh_cv_getxattr_linux=yes],
+  [zsh_cv_getxattr_linux=no])])
+
+  if test x$zsh_cv_getxattr_linux != xyes; then
+    AC_CACHE_CHECK(if getxattr etc. are MAC-like,
+    zsh_cv_getxattr_mac,
+    [AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <sys/types.h>
+#include <sys/xattr.h>]],
+    [[(void)listxattr("", 0, 0, 0);
+    (void)getxattr("", "", 0, 0, 0, 0);
+    (void)setxattr("", "", "", 0, 0, 0);
+    (void)removexattr("", "", 0);]])],
+    [zsh_cv_getxattr_mac=yes],
+    [zsh_cv_getxattr_mac=no])])
+
+    if test x$zsh_cv_getxattr_mac = xyes; then
+      AC_DEFINE(XATTR_EXTRA_ARGS)
+    fi
+  fi
+fi
+
+AC_CACHE_CHECK(if getxattr etc. are usable,
+zsh_cv_use_xattr,
+[if test x$zsh_cv_getxattr_linux = xyes || test x$zsh_cv_getxattr_mac = xyes
+then
+zsh_cv_use_xattr=yes
+else
+zsh_cv_use_xattr=no
+fi])
 
 dnl -------------
 dnl CHECK SIGNALS
Index: Src/Modules/attr.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/attr.c,v
retrieving revision 1.2
diff -u -r1.2 attr.c
--- Src/Modules/attr.c	3 Mar 2009 16:40:43 -0000	1.2
+++ Src/Modules/attr.c	8 Mar 2009 19:51:19 -0000
@@ -42,8 +42,16 @@
 
     unmetafy(*argv, &slen);
     unmetafy(*(argv+1), NULL);
-    if (listxattr(*argv, NULL, 0) > 0) {
-        if (0 < (len = getxattr(*argv, *(argv+1), value, 255))) {
+    if (listxattr(*argv, NULL, 0
+#ifdef XATTR_EXTRA_ARGS
+		  , 0
+#endif
+		  ) > 0) {
+        if (0 < (len = getxattr(*argv, *(argv+1), value, 255
+#ifdef XATTR_EXTRA_ARGS
+				, 0, 0
+#endif
+				))) {
             if (len < 256) {
                 value[len] = '\0';
                 if (*(argv+2))
@@ -67,7 +75,11 @@
     unmetafy(*argv, &slen);
     unmetafy(*(argv+1), NULL);
     unmetafy(*(argv+2), NULL);
-    if (setxattr(*argv, *(argv+1), *(argv+2), strlen(*(argv+2)), 0)) {
+    if (setxattr(*argv, *(argv+1), *(argv+2), strlen(*(argv+2)), 0
+#ifdef XATTR_EXTRA_ARGS
+						     , 0
+#endif
+		 )) {
         zwarnnam(nam, "%s: %e", metafy(*argv, slen, META_NOALLOC), errno);
         ret = 1;
     }
@@ -81,7 +93,11 @@
 
     unmetafy(*argv, &slen);
     unmetafy(*(argv+1), NULL);
-    if (removexattr(*argv, *(argv+1))) {
+    if (removexattr(*argv, *(argv+1)
+#ifdef XATTR_EXTRA_ARGS
+		    , 0
+#endif
+		    )) {
         zwarnnam(nam, "%s: %e", metafy(*argv, slen, META_NOALLOC), errno);
         ret = 1;
     }
@@ -96,7 +112,11 @@
     char value[256];
 
     unmetafy(*argv, &slen);
-    if (0 < (len = listxattr(*argv, value, 256))) {
+    if (0 < (len = listxattr(*argv, value, 256
+#ifdef XATTR_EXTRA_ARGS
+		  , 0
+#endif
+			     ))) {
         if (len < 256) {
             char *p = value;
             if (*(argv+1))
Index: Src/Modules/attr.mdd
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/attr.mdd,v
retrieving revision 1.1
diff -u -r1.1 attr.mdd
--- Src/Modules/attr.mdd	3 Mar 2009 15:04:28 -0000	1.1
+++ Src/Modules/attr.mdd	8 Mar 2009 19:51:19 -0000
@@ -1,5 +1,5 @@
 name=zsh/attr
-link='if test "x$ac_cv_func_getxattr" = xyes && test "x$ac_cv_header_sys_xattr_h" = xyes; then
+link='if test "x$zsh_cv_use_xattr" = xyes; then
   echo dynamic
 else
   echo no


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Modules/attr.c compile error on Mac OS X

[Bart Schaefer]

(Aside: Sourceforge CVS access is really slow right now.)

On Mar 8,  7:55pm, Peter Stephenson wrote:
} Subject: Re: Modules/attr.c compile error on Mac OS X
}
} On Sun, 8 Mar 2009 17:21:47 +0900
} Taro M <taromeister@gmail.com> wrote:
} > The new Modules/attr.c does not compile on Mac OS X (and probably others):
} 
} I don't know about the others, and you don't say, but a bit of research
} suggests Mac OS X takes some extra arguments.  I don't have Mac OS X so
} can't test that; if it still doesn't compile, please could one of the
} numerous Mac users provide a patch.

Compiling on MacOS now ... snippet of config.log:

configure:13232: checking if getxattr etc. are Linux-like
configure:13264: gcc -c  -Wall -Wmissing-prototypes -O2  conftest.c >&5
conftest.c: In function 'main':
conftest.c:211: error: too few arguments to function 'listxattr'
conftest.c:212: error: too few arguments to function 'getxattr'
conftest.c:213: error: too few arguments to function 'setxattr'
conftest.c:214: error: too few arguments to function 'removexattr'
configure:13270: $? = 1
configure: failed program was:
    (omitted)
configure:13285: result: no
configure:13289: checking if getxattr etc. are MAC-like
configure:13319: gcc -c  -Wall -Wmissing-prototypes -O2  conftest.c >&5
configure:13325: $? = 0
configure:13340: result: yes
configure:13352: checking if getxattr etc. are usable
configure:13364: result: yes

However, it didn't build the attr module.  Still didn't build it after
a "make clean".  Missing dependency?  Trying again from "make distclean":
OK, compiled the attr module that time.

} If it compiles but doesn't work,
} I will need someone else to research the problem.

I tried this:

% zmodload zsh/attr
% for x in /*; print -R $x $'\t= ' $(zlistattr $x)
/Applications   = 
/Desktop DB     = com.apple.FinderInfo
/Desktop DF     = com.apple.FinderInfo
/Developer      = com.apple.FinderInfo
/Library        = 
/Network        = com.apple.FinderInfo
/System         = 
/User Guides And Information    = com.apple.FinderInfo
/Users  = 
/Volumes        = com.apple.FinderInfo
/bin    = com.apple.FinderInfo
/cores  = com.apple.FinderInfo
/dev    = 
/etc    = 
/home   = 
/mach.sym       = com.apple.FinderInfo
/mach_kernel    = com.apple.FinderInfo
/mach_kernel.ctfsys     = com.apple.FinderInfo
/net    = 
/opt    = 
/private        = com.apple.FinderInfo
/sbin   = com.apple.FinderInfo
/tmp    = 
/usr    = com.apple.FinderInfo
/var    = com.apple.FinderInfo

Is there more that should be tested?
 
} ... patches are useful even if incomplete (and preferred to text
} statements along the lines of "wouldn't it be better to change ...").

Pardon for my tendency to do that lately.  My inclination is to avoid
sending a patch unless I actually intend for someone to apply it, so
if I'm not certain about a change ...

Re: Modules/attr.c compile error on Mac OS X

[Peter Stephenson]

Bart Schaefer wrote:
> However, it didn't build the attr module.  Still didn't build it after
> a "make clee an".  Missing dependency?  Trying again from "make distclean":
> OK, compiled the attr module that time.

I think that's because config.modules.sh doesn't get rerun
automatically.  I've come across the same thing.  config.modules should
probably depend on it somewhere.  I wonder if the following fixes it or
simply introduces some dependency loop...  It's odd that *nothing*
currently depends on config.modules, but maybe I just screwed up.

> I tried this:
>...
> Is there more that should be tested?

Looks pretty good to me; suggests that with the extra arguments it's not
fundamentally broken.

We could probably do with a Test module that only runs if zsh/attr is
loadable and the file system supports extended attributes (i.e. if it
doesn't load it should report as "couldn't test" rather than "test
failed").

Index: Makefile.in
===================================================================
RCS file: /cvsroot/zsh/zsh/Makefile.in,v
retrieving revision 1.19
diff -u -r1.19 Makefile.in
--- Makefile.in	1 Feb 2008 11:31:36 -0000	1.19
+++ Makefile.in	8 Mar 2009 21:45:59 -0000
@@ -41,7 +41,7 @@
 # ========== DEPENDENCIES FOR BUILDING ==========
 
 # default target
-all: config.h
+all: config.h config.modules
 	@for subdir in Src Doc; do \
 	  (cd $$subdir && $(MAKE) $(MAKEDEFS) $@) || exit 1; \
 	done
@@ -142,8 +142,9 @@
 stamp-h: $(sdir)/config.h.in config.status
 	cd $(dir_top) && $(SHELL) ./config.status config.h $@
 
-config.modules: $(sdir)/config.h.in config.status
-	cd $(dir_top) && $(SHELL) ./config.status $@
+config.modules: $(sdir)/config.h.in config.status config.modules.sh
+	cd $(dir_top) && $(SHELL) ./config.status $@ && \
+	$(SHELL) ./config.modules.sh
 
 $(sdir)/config.h.in: $(sdir)/stamp-h.in
 $(sdir)/stamp-h.in: $(sdir)/configure.ac \

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Modules/attr.c compile error on Mac OS X

[Bart Schaefer]

On Mar 8,  9:51pm, Peter Stephenson wrote:
}
} Bart Schaefer wrote:
} > However, it didn't build the attr module.  Still didn't build it after
} > a "make clean".  Missing dependency?
} 
} I think that's because config.modules.sh doesn't get rerun
} automatically.  I've come across the same thing.  config.modules should
} probably depend on it somewhere.  I wonder if the following fixes it or
} simply introduces some dependency loop... 

Seems to be OK (no loop) on Linux.  After applying the patch, "make"
rebuilds config.modules, all the Makefiles, and then init.o and relinks.
A subsquent "make" does all the stamp-modobjs.tmp steps but does not
rebuild anything.

I didn't try a from-scratch build, though.

zsh bug in . builtin

[Eric Blake]

According to POSIX,

A syntax error in a special built-in utility may cause a shell executing
that utility to abort, while a syntax error in a regular built-in utility
shall not cause a shell executing that utility to abort. (See Consequences
of Shell Errors for the consequences of errors on interactive and
non-interactive shells.) If a special built-in utility encountering a
syntax error does not abort the shell, its exit value shall be non-zero.
http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_14

However, zsh fails to recognize syntax errors encountered when sourcing a
file (tested with zsh 4.3.9):

$ echo 'if' > tmp
$ zsh -c '. ./tmp; echo $?'
0
$ zsh -c 'emulate sh; . ./tmp; echo $?'
./tmp:2: parse error near `\n'
0
$ zsh -c 'emulate sh; (. ./tmp); echo $?'
./tmp:2: parse error near `\n'
0

For the first test, you can do whatever (since zsh is not in sh-mode, so
it doesn't have to follow POSIX rules).  For the second line, it would be
feasible if $? is not echoed (because the entire shell was aborted), but
it is not acceptable to print 0.  And for the third line, the echo $?
should always be reached (regardless of whether the subshell aborted or
exited with failure), but again it is not acceptable to print 0.

-- 
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9@byu.net

Re: zsh bug in . builtin

[Peter Stephenson]

On Wed, 01 Jul 2009 07:14:54 -0600
Eric Blake <ebb9@byu.net> wrote:
> However, zsh fails to recognize syntax errors encountered when sourcing a
> file (tested with zsh 4.3.9):
> 
> $ echo 'if' > tmp
> $ zsh -c '. ./tmp; echo $?'
> 0
> $ zsh -c 'emulate sh; . ./tmp; echo $?'
> ./tmp:2: parse error near `\n'
> 0
> $ zsh -c 'emulate sh; (. ./tmp); echo $?'
> ./tmp:2: parse error near `\n'
> 0

Those are all bugs; it's not supposed to ignore errors even in native mode.
I've taken 128 to mean failed to execute file, 129 to mean failed to find
file---anyone can suggest better values if unhappy.

Index: Doc/Zsh/builtins.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/builtins.yo,v
retrieving revision 1.121
diff -u -r1.121 builtins.yo
--- Doc/Zsh/builtins.yo	2 Mar 2009 10:12:14 -0000	1.121
+++ Doc/Zsh/builtins.yo	1 Jul 2009 14:21:45 -0000
@@ -46,7 +46,9 @@
 If any arguments var(arg) are given,
 they become the positional parameters; the old positional
 parameters are restored when the var(file) is done executing.
-The exit status is the exit status of the last command executed.
+If var(file) was not found the return status is 129; if var(file) was found
+but contained a syntax error the return status is 128; else the return
+status is the exit status of the last command executed.
 )
 findex(NOTRANS(:))
 cindex(expanding parameters)
Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.228
diff -u -r1.228 builtin.c
--- Src/builtin.c	1 Jul 2009 09:07:08 -0000	1.228
+++ Src/builtin.c	1 Jul 2009 14:21:45 -0000
@@ -4701,9 +4701,10 @@
 bin_dot(char *name, char **argv, UNUSED(Options ops), UNUSED(int func))
 {
     char **old, *old0 = NULL;
-    int ret, diddot = 0, dotdot = 0;
+    int diddot = 0, dotdot = 0;
     char *s, **t, *enam, *arg0, *buf;
     struct stat st;
+    enum source_return ret;
 
     if (!*argv)
 	return 0;
@@ -4719,14 +4720,14 @@
     }
     s = unmeta(enam);
     errno = ENOENT;
-    ret = 1;
+    ret = SOURCE_NOT_FOUND;
     /* for source only, check in current directory first */
     if (*name != '.' && access(s, F_OK) == 0
 	&& stat(s, &st) >= 0 && !S_ISDIR(st.st_mode)) {
 	diddot = 1;
 	ret = source(enam);
     }
-    if (ret) {
+    if (ret == SOURCE_NOT_FOUND) {
 	/* use a path with / in it */
 	for (s = arg0; *s; s++)
 	    if (*s == '/') {
@@ -4739,7 +4740,8 @@
 		ret = source(arg0);
 		break;
 	    }
-	if (!*s || (ret && isset(PATHDIRS) && diddot < 2 && dotdot == 0)) {
+	if (!*s || (ret == SOURCE_NOT_FOUND &&
+		    isset(PATHDIRS) && diddot < 2 && dotdot == 0)) {
 	    pushheap();
 	    /* search path for script */
 	    for (t = path; *t; t++) {
@@ -4766,12 +4768,12 @@
 	freearray(pparams);
 	pparams = old;
     }
-    if (ret)
+    if (ret == SOURCE_NOT_FOUND)
 	zwarnnam(name, "%e: %s", errno, enam);
     zsfree(arg0);
     if (old0)
 	argzero = old0;
-    return ret ? ret : lastval;
+    return ret == SOURCE_OK ? lastval : 127 + ret;
 }
 
 /*
Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.100
diff -u -r1.100 init.c
--- Src/init.c	9 Mar 2009 15:57:59 -0000	1.100
+++ Src/init.c	1 Jul 2009 14:21:45 -0000
@@ -99,10 +99,11 @@
 /* keep executing lists until EOF found */
 
 /**/
-void
+int
 loop(int toplevel, int justonce)
 {
     Eprog prog;
+    int err;
 
     pushheap();
     if (!toplevel)
@@ -201,9 +202,12 @@
 	if (justonce)
 	    break;
     }
+    err = errflag;
     if (!toplevel)
 	lexrestore();
     popheap();
+
+    return err;
 }
 
 static char *cmd;
@@ -1049,10 +1053,13 @@
 	readhistfile(NULL, 0, HFILE_USE_OPTIONS);
 }
 
-/* source a file */
+/*
+ * source a file
+ * Returns one of the SOURCE_* enum values.
+ */
 
 /**/
-mod_export int
+mod_export enum source_return
 source(char *s)
 {
     Eprog prog;
@@ -1066,11 +1073,12 @@
     int ocsp;
     int otrap_return = trap_return, otrap_state = trap_state;
     struct funcstack fstack;
+    enum source_return ret = SOURCE_OK;
 
     if (!s || 
 	(!(prog = try_source_file((us = unmeta(s)))) &&
 	 (tempfd = movefd(open(us, O_RDONLY | O_NOCTTY))) == -1)) {
-	return 1;
+	return SOURCE_NOT_FOUND;
     }
 
     /* save the current shell state */
@@ -1121,8 +1129,13 @@
 	errflag = 0;
 	execode(prog, 1, 0);
 	popheap();
-    } else
-	loop(0, 0);		     /* loop through the file to be sourced  */
+	if (errflag)
+	    ret = SOURCE_ERROR;
+    } else {
+	/* loop through the file to be sourced  */
+	if (loop(0, 0))
+	    ret = SOURCE_ERROR;
+    }
     funcstack = funcstack->prev;
     sourcelevel--;
 
@@ -1152,7 +1165,7 @@
     cmdstack = ocs;
     cmdsp = ocsp;
 
-    return 0;
+    return ret;
 }
 
 /* Try to source a file in the home directory */
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.156
diff -u -r1.156 zsh.h
--- Src/zsh.h	5 Jun 2009 11:15:53 -0000	1.156
+++ Src/zsh.h	1 Jul 2009 14:21:45 -0000
@@ -1725,6 +1725,17 @@
 #define PRINT_WHENCE_FUNCDEF	(1<<9)
 #define PRINT_WHENCE_WORD	(1<<10)
 
+/* Return values from source() */
+
+enum source_return {
+    /* Source ran OK */
+    SOURCE_OK = 0,
+    /* Internal error sourcing file */
+    SOURCE_ERROR = 1,
+    /* File not found */
+    SOURCE_NOT_FOUND = 2
+};
+
 /***********************************/
 /* Definitions for history control */
 /***********************************/
Index: Src/Modules/newuser.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/newuser.c,v
retrieving revision 1.7
diff -u -r1.7 newuser.c
--- Src/Modules/newuser.c	11 Feb 2009 20:42:17 -0000	1.7
+++ Src/Modules/newuser.c	1 Jul 2009 14:21:45 -0000
@@ -97,7 +97,7 @@
 	VARARR(char, buf, strlen(*sp) + 9);
 	sprintf(buf, "%s/newuser", *sp);
 
-	if (!source(buf))
+	if (source(buf) != SOURCE_NOT_FOUND)
 	    break;
     }
 
Index: Test/A01grammar.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A01grammar.ztst,v
retrieving revision 1.19
diff -u -r1.19 A01grammar.ztst
--- Test/A01grammar.ztst	21 Apr 2009 10:38:00 -0000	1.19
+++ Test/A01grammar.ztst	1 Jul 2009 14:21:45 -0000
@@ -514,4 +514,11 @@
 >value
 >not#comment
 
-  
+  . ./nonexistent
+129: Attempt to "." non-existent file.
+?(eval):.:1: no such file or directory: ./nonexistent
+
+  echo '[[' >bad_syntax
+  . ./bad_syntax
+128: Attempt to "." file with bad syntax.
+?./bad_syntax:2: parse error near `\n'

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: zsh bug in . builtin

[Eric Blake]

Peter Stephenson <pws <at> csr.com> writes:

> Those are all bugs; it's not supposed to ignore errors even in native mode.
> I've taken 128 to mean failed to execute file, 129 to mean failed to find
> file---anyone can suggest better values if unhappy.

I'd much prefer 127 for a missing file (similar to exec), and something < 127
for syntax error (128 and above should typically be for signals).

Also, you may want to consistently detect syntax errors in eval and in the
action to trap:

$ zsh -c 'eval "if"; echo $?'
0
$ zsh -c 'emulate sh; eval "if"; echo $?'
zsh:1: parse error near `if'
1
$ zsh -c 'trap "if" 0'; echo $?
0
$ zsh -c 'emulate sh; trap "if" 0'; echo $?
zsh:1: parse error near `if'
zsh:trap:1: couldn't parse trap command
1
$ zsh -c 'trap "if" 1; kill -1 $$; echo $?'
0
$ zsh -c 'emulate sh; trap "if" 1; kill -1 $$; echo $?'
zsh:1: parse error near `if'
zsh:trap:1: couldn't parse trap command
$ echo $?
1

Re: zsh bug in . builtin

[Peter Stephenson]

On Thu, 2 Jul 2009 12:01:11 +0000 (UTC)
Eric Blake <ebb9@byu.net> wrote:
> Peter Stephenson <pws <at> csr.com> writes:
> 
> > Those are all bugs; it's not supposed to ignore errors even in native mode.
> > I've taken 128 to mean failed to execute file, 129 to mean failed to find
> > file---anyone can suggest better values if unhappy.
> 
> I'd much prefer 127 for a missing file (similar to exec), and something < 127
> for syntax error (128 and above should typically be for signals).

That seems reasonable.

Index: Doc/Zsh/builtins.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/builtins.yo,v
retrieving revision 1.122
diff -u -r1.122 builtins.yo
--- Doc/Zsh/builtins.yo	1 Jul 2009 15:07:32 -0000	1.122
+++ Doc/Zsh/builtins.yo	2 Jul 2009 13:00:31 -0000
@@ -46,8 +46,8 @@
 If any arguments var(arg) are given,
 they become the positional parameters; the old positional
 parameters are restored when the var(file) is done executing.
-If var(file) was not found the return status is 129; if var(file) was found
-but contained a syntax error the return status is 128; else the return
+If var(file) was not found the return status is 127; if var(file) was found
+but contained a syntax error the return status is 126; else the return
 status is the exit status of the last command executed.
 )
 findex(NOTRANS(:))
Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.229
diff -u -r1.229 builtin.c
--- Src/builtin.c	1 Jul 2009 15:07:32 -0000	1.229
+++ Src/builtin.c	2 Jul 2009 13:00:31 -0000
@@ -4773,7 +4773,7 @@
     zsfree(arg0);
     if (old0)
 	argzero = old0;
-    return ret == SOURCE_OK ? lastval : 127 + ret;
+    return ret == SOURCE_OK ? lastval : 128 - ret;
 }
 
 /*
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.157
diff -u -r1.157 zsh.h
--- Src/zsh.h	1 Jul 2009 15:07:33 -0000	1.157
+++ Src/zsh.h	2 Jul 2009 13:00:31 -0000
@@ -1730,10 +1730,10 @@
 enum source_return {
     /* Source ran OK */
     SOURCE_OK = 0,
-    /* Internal error sourcing file */
-    SOURCE_ERROR = 1,
     /* File not found */
-    SOURCE_NOT_FOUND = 2
+    SOURCE_NOT_FOUND = 1,
+    /* Internal error sourcing file */
+    SOURCE_ERROR = 2
 };
 
 /***********************************/
Index: Test/A01grammar.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A01grammar.ztst,v
retrieving revision 1.20
diff -u -r1.20 A01grammar.ztst
--- Test/A01grammar.ztst	1 Jul 2009 15:07:33 -0000	1.20
+++ Test/A01grammar.ztst	2 Jul 2009 13:00:31 -0000
@@ -515,10 +515,10 @@
 >not#comment
 
   . ./nonexistent
-129: Attempt to "." non-existent file.
+127: Attempt to "." non-existent file.
 ?(eval):.:1: no such file or directory: ./nonexistent
 
   echo '[[' >bad_syntax
   . ./bad_syntax
-128: Attempt to "." file with bad syntax.
+126: Attempt to "." file with bad syntax.
 ?./bad_syntax:2: parse error near `\n'

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: zsh bug in . builtin

[Bart Schaefer]

On Jul 1,  3:24pm, Peter Stephenson wrote:
}
} Those are all bugs; it's not supposed to ignore errors even in native mode.

I just got back from vacation and haven't had a chance to look at this in
any detail, but please make sure that these changes have not caused the
shell to refuse to start up when there is an error in ~/.zshrc et al.

Re: zsh bug in . builtin

[Peter Stephenson]

Bart Schaefer wrote:
> On Jul 1,  3:24pm, Peter Stephenson wrote:
> }
> } Those are all bugs; it's not supposed to ignore errors even in native mode.
> 
> I just got back from vacation and haven't had a chance to look at this in
> any detail, but please make sure that these changes have not caused the
> shell to refuse to start up when there is an error in ~/.zshrc et al.

The issue only affects "." and "source", and only changes the return
status.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zsh bug in . builtin

[Peter Stephenson]

On Thu, 2 Jul 2009 12:01:11 +0000 (UTC)
Eric Blake <ebb9@byu.net> wrote:
> Also, you may want to consistently detect syntax errors in eval and in the
> action to trap:
> 
> $ zsh -c 'eval "if"; echo $?'
> 0

The problem isn't in eval or trap.  The problem is that the short form
of "if" (which is an abomination anyway and no one should use) isn't
tested properly.

Adding the test below to the parser should fix it.  My heart sank when
this caused a failure in the test system---however, having read the
manual again, that test should never have worked, since the if-clause
wasn't delimited, which is the prerequisite for the syntax to work.  The
old if-clause therefore extended to the end of the evaluated block and
the then-clause was missing, so failing is correct.  (If you're not
convinced, try eval'ing "if false; print False is true" and you'll see
it gets to the "print", because it's still looking for the end of the
if-clause at that point.)  I've fixed the tests.

Index: Src/parse.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
retrieving revision 1.79
diff -u -r1.79 parse.c
--- Src/parse.c	25 Feb 2009 10:24:01 -0000	1.79
+++ Src/parse.c	6 Jul 2009 20:28:56 -0000
@@ -1199,6 +1199,10 @@
 	type = (xtok == IF ? WC_IF_IF : WC_IF_ELIF);
 	par_save_list(complex);
 	incmdpos = 1;
+	if (tok == ENDINPUT) {
+	    cmdpop();
+	    YYERRORV(oecused);
+	}
 	while (tok == SEPER)
 	    zshlex();
 	xtok = FI;
Index: Test/A01grammar.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A01grammar.ztst,v
retrieving revision 1.21
diff -u -r1.21 A01grammar.ztst
--- Test/A01grammar.ztst	2 Jul 2009 13:48:36 -0000	1.21
+++ Test/A01grammar.ztst	6 Jul 2009 20:28:56 -0000
@@ -378,10 +378,15 @@
 >true-2
 >false
 
-  if true; print true
+  if { true } print true
+  if { false } print false
 0:Short form of `if'
 >true
 
+  eval "if"
+1:Short form of `if' can't be too short
+?(eval):1: parse error near `if'
+
   for name ( word1 word2 word3 ) print $name
 0:Form of `for' with parentheses.
 >word1

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

non-interactive set -m

[Eric Blake]

As far as I can tell, POSIX requires 'set -m' to work even in non-interactive 
shells, if the shell implements the User Portability Utilities (UP) option.  
POSIX is also explicit that 'set -b' shall default to disabled.  Therefore, I 
claim that this demonstrates two bugs:

$ zsh -c 'emulate sh; echo $-; set -m && set +m'
b
zsh:set:1: can't change option: -m
$ echo 1

Of course, being a non-interactive environment, there are no prompts printed, 
so the rules about 'set -m' printing status changes prior to a prompt are 
ignored; and since 'set -b' should be disabled, no asynchronous status messages 
would be printed either.  But the important part is that 'set -m' also controls 
the fact that background jobs are created in their own process group, which IS 
something that a non-interactive script can usefully exploit.  Besides, POSIX 
is eplicit that jobs also work even when 'set +m' is in effect. 

Autoconf would LOVE to implement parallel testsuite support, but supporting 
clean signal-based testsuite abortion at the parent level without leaving hung 
children and without tickling fork bomb bugs in various ksh clones requires the 
use of job control in a non-interactive environment.  Thus, at the moment, bash 
appears to be the only shell capable of reliably running a parallel testsuite:
http://thread.gmane.org/gmane.comp.sysutils.autoconf.bugs/6776

-- 
Eric Blake

Re: non-interactive set -m

[Peter Stephenson]

Eric Blake wrote:
> As far as I can tell, POSIX requires 'set -m' to work even in non-interactive
>  
> shells, if the shell implements the User Portability Utilities (UP) option.  
> POSIX is also explicit that 'set -b' shall default to disabled.  Therefore, I
>  
> claim that this demonstrates two bugs:
> 
> $ zsh -c 'emulate sh; echo $-; set -m && set +m'
> b
> zsh:set:1: can't change option: -m
> $ echo 1

The -b behaviour isn't a bug: use "emulate -R sh" to get all options,
including ones that don't affect syntax, reset to the emulation
defaults.  However, making the shell start up in zsh mode and switching
is increasing the complexity of what's going on; something like

  ARGV0=sh zsh -c 'echo $-'

is easier and the output should be empty.  (Alternatively, if you're not
starting from zsh, use a symbolic link to zsh.)

Part of my reason for suggesting that is the following.  It's true you
can't change the MONITOR option after the shell starts.  It's probably
fixable, but being able to switch back and forth might be complicated
(though I haven't looked in detail).  However, making MONITOR work at
startup isn't so hard: the following simple patch stops it being tied to
interactive mode.  As this has no effect unless you request MONITOR, it
won't cause problems elsewhere.  This would mean that

  ARGV0=sh zsh -mc '...'

will enable monitor mode at startup.  Note that you do need a tty or pty
of some sort (that's what SHTTY != -1 is testing), but that's probably
obvious from the way job control works.

What I don't know and haven't tested is whether MONITOR actually works
in a non-interative shell.  A simple test suggested it was basically OK.
If someone can investigate whether there are any problems with the
following patch I will see if they are easy to fix.

Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.101
diff -u -r1.101 init.c
--- Src/init.c	1 Jul 2009 15:07:33 -0000	1.101
+++ Src/init.c	8 Jul 2009 08:49:21 -0000
@@ -486,7 +486,7 @@
      * process group leader.
      */
     mypid = (zlong)getpid();
-    if (opts[MONITOR] && interact && (SHTTY != -1)) {
+    if (opts[MONITOR] && (SHTTY != -1)) {
 	origpgrp = GETPGRP();
         acquire_pgrp(); /* might also clear opts[MONITOR] */
     } else


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: non-interactive set -m

[Eric Blake]

Peter Stephenson <pws <at> csr.com> writes:

> Part of my reason for suggesting that is the following.  It's true you
> can't change the MONITOR option after the shell starts.  It's probably
> fixable, but being able to switch back and forth might be complicated
> (though I haven't looked in detail).

POSIX requires that 'set -m'/'set -o monitor' be switchable on the fly, without
regards to whether the shell is interactive.  And your claim that MONITOR
cannot be changed on the fly is false: zsh already handles this for interactive
shells, with knock-on effects that fg and bg change behavior depending on the
current state of MONITOR (also as specified by POSIX).

$ ARGV0=sh zsh -ci 'echo $-; set +m; sleep 1& fg; echo $-; set -m; echo $-'
569XZim
zsh:fg:1: no job control in this shell.
569XZi
569XZim

>  However, making MONITOR work at
> startup isn't so hard: the following simple patch stops it being tied to
> interactive mode.  As this has no effect unless you request MONITOR, it
> won't cause problems elsewhere.  This would mean that
> 
>   ARGV0=sh zsh -mc '...'
> 
> will enable monitor mode at startup.

Unfortunately, for autoconf's purposes, that is not possible without
contortions of re-starting a script if it is detected that zsh is running the
script but not in sh mode.  Autoconf does not have control over how the user
started a script, but wants the shell to be in POSIX mode after the point that
the script is already started, as part of sanitizing the environment to be
portable across as many shells as possible, so autoconf is forced to use
'emulate sh' (although we may switch it to use 'emulate -R sh', based on your
advice).

>  Note that you do need a tty or pty
> of some sort (that's what SHTTY != -1 is testing), but that's probably
> obvious from the way job control works.

You are correct that you can't change interactive mode on the fly - whether $-
prints i is fixed from the moment the shell starts up.  You are also correct
that the initial state of MONITOR depends on whether the shell is interactive,
and that whether a shell is interactive depends, in part, on whether there is a
tty: POSIX requires that interactive shells start with 'set -m' and
non-interactive shells start with 'set +m' unless specified differently by
other command line options, so at startup, $- will usually contain either both
or neither of 'im'.  But while the presence of a tty is a factor for
determining whether the shell is interactive, it is NOT a factor in determining
whether job control works.  Job control works just fine without a tty, as
proved by bash and ksh.  Meanwhile, the presence of a tty is not the ONLY way
to get an interactive shell; POSIX requires that even without a tty, a shell
started with an explicit -i must be interactive (although in this condition,
the shell need not print prompts).

According to POSIX, job control does two things - it changes messages printed
before a prompt (related to interactivity and the presence of a tty), and it
makes it possible to spawn asynchronous processes in separate process groups
for independent tracking.  It is the latter part (separate process groups) that
autoconf wants to use to enable parallel testsuites, and which is unrelated to
whether there is a tty.  And in a non-interactive shell, the former part
(printing a message before prompts) does not matter, since non-interactive
shells never print prompts in the first place.

Basically, zsh cannot be used for parallel autoconf tests unless you fix the
bug of allowing job control support to be changed on the fly for non-interactive
shells.

-- 
Eric Blake

Re: non-interactive set -m

[Peter Stephenson]

Eric Blake wrote:
> POSIX requires that 'set -m'/'set -o monitor' be switchable on the fly

Unfortunately I have limited time even for fixing real bugs in the
shell, which was not originally designed to be a POSIX shell and is not
your best bet if POSIX compatibility is your bottom line, and we have
very few volunteers for fixing stuff in the core shell.  So arguments of
this kind on their own tend to fall a bit flat in these parts, for
purely practical reasons (we're not actually against providing POSIX
compatibility, of course).

"It'll work if you change *this*" is a much more enticing argument...

> And your claim that MONITOR
> cannot be changed on the fly is false: zsh already handles this for
> interactive shells

You're right---there's another test of the same kind there (i.e. that
we're interactive).  This removes it.  Let me know if you run up against
more, or if there's some hidden interaction with the
interactive/non-interactive mode (I wouldn't bet against it).

I think the code that does the run-time switching is a late addition,
but I haven't checked the history.

Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.101
diff -u -r1.101 init.c
--- Src/init.c	1 Jul 2009 15:07:33 -0000	1.101
+++ Src/init.c	8 Jul 2009 13:36:58 -0000
@@ -486,7 +486,7 @@
      * process group leader.
      */
     mypid = (zlong)getpid();
-    if (opts[MONITOR] && interact && (SHTTY != -1)) {
+    if (opts[MONITOR] && (SHTTY != -1)) {
 	origpgrp = GETPGRP();
         acquire_pgrp(); /* might also clear opts[MONITOR] */
     } else
Index: Src/options.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/options.c,v
retrieving revision 1.48
diff -u -r1.48 options.c
--- Src/options.c	3 Mar 2009 17:26:08 -0000	1.48
+++ Src/options.c	8 Jul 2009 13:36:58 -0000
@@ -730,7 +730,7 @@
     } else if (!force && optno == MONITOR && value) {
 	if (opts[optno] == value)
 	    return 0;
-	if (interact && (SHTTY != -1)) {
+	if (SHTTY != -1) {
 	    origpgrp = GETPGRP();
 	    acquire_pgrp();
 	} else


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: non-interactive set -m

[Eric Blake]

Peter Stephenson <pws <at> csr.com> writes:

> > POSIX requires that 'set -m'/'set -o monitor' be switchable on the fly
> 
> You're right---there's another test of the same kind there (i.e. that
> we're interactive).  This removes it.  Let me know if you run up against
> more, or if there's some hidden interaction with the
> interactive/non-interactive mode (I wouldn't bet against it).

I'd love to test this.  However, I'm behind a firewall, and can't access CVS. 
Is there a snapshot available? 
http://zsh.sourceforge.net/zsh-4.3-daily-snapshot.tar.bz2 appears to be a dead
link.  Or are there any plans for moving to a newer VCS, such as git, which
makes http access much easier?

-- 
Eric Blake

Re: non-interactive set -m

[Peter Stephenson]

Eric Blake wrote:
> Peter Stephenson <pws <at> csr.com> writes:
> 
> > > POSIX requires that 'set -m'/'set -o monitor' be switchable on the fly
> > 
> > You're right---there's another test of the same kind there (i.e. that
> > we're interactive).  This removes it.  Let me know if you run up against
> > more, or if there's some hidden interaction with the
> > interactive/non-interactive mode (I wouldn't bet against it).
> 
> I'd love to test this.  However, I'm behind a firewall, and can't access CVS.
>  
> Is there a snapshot available? 
> http://zsh.sourceforge.net/zsh-4.3-daily-snapshot.tar.bz2 appears to be a dea
> d
> link.  Or are there any plans for moving to a newer VCS, such as git, which
> makes http access much easier?

There's an almost up-to-date git mirror available; see

http://www.zsh.org/mla/workers/2009/msg00516.html

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: non-interactive set -m

[Eric Blake]

Peter Stephenson <pws <at> csr.com> writes:

> > Is there a snapshot available? 
> > http://zsh.sourceforge.net/zsh-4.3-daily-snapshot.tar.bz2 appears to be a dea
> > d
> > link.  Or are there any plans for moving to a newer VCS, such as git, which
> > makes http access much easier?
> 
> There's an almost up-to-date git mirror available; see
> 
> http://www.zsh.org/mla/workers/2009/msg00516.html

THANKS!  That's exactly what I was hoping for!  Coupled with repo.or.cz's
ability to create an http mirror of any git:// repository
(http://repo.or.cz/r/zsh/mirror.git), I now have enough to try out the recent
monitor patch.

Now, what would it take to get this documented in the FAQ and/or zsh homepage?
It might also be nice to document the sourceforge web-browseable interface,
http://zsh.git.sourceforge.net/git/gitweb.cgi?p=zsh

-- 
Eric Blake

Re: non-interactive set -m

[Eric Blake]

Peter Stephenson <pws <at> csr.com> writes:

> You're right---there's another test of the same kind there (i.e. that
> we're interactive).  This removes it.  Let me know if you run up against
> more, or if there's some hidden interaction with the
> interactive/non-interactive mode (I wouldn't bet against it).

As-is, your 2-hunk patch accidentally sets monitor mode by default for
non-interactive shells:

$ Src/zsh -c 'echo $-'
569Xm

But by reverting the hunk to Src/init.c, and using JUST the hunk in
Src/options.c, that particular snafu is fixed.  And for the simple tests that I
have run, I can confirm that 'set -m' and 'set +m' now work from a
non-interactive shell, that jobs appears to track background tasks correctly
given either setting of monitor in a non-interactive shell, and that background
processes spawned from a non-interactive shell are given their own process group
under 'set -m' but a common process group under 'set +m', as desired.

In all of my tests, running './sh -ci' instead of './sh -c' did not have any
problems (so it looks like your patch did not introduce any regressions to
interactive mode monitor handling, which is already working as desired).

However, I'm seeing an oddity with bg/fg that needs to be tracked down.  For an
example comparison against bash:

$ bash -c 'sleep 5& fg'
bash: line 0: fg: no job control
$ bash -c 'set -m; sleep 5& fg'
sleep 5
$ ln -s Src/zsh sh
$ ./sh -c 'sleep 5& fg'
zsh:fg:1: no job control in this shell.
$ ./sh -c 'set -m; sleep 5& fg'

[2]+  Stopped                 ./sh -c 'set -m; sleep 5& fg'
$ kill -s sigcont 50408
zsh:1: can't set tty pgrp: interrupt
$

It looks like the fg activated the sleep process, because it took five seconds
for the Stopped message to appear, but fg failed to print to stdout the process
that it just activated.  Also, when the sleep finally exited, zsh got in a weird
state, and ended up stopping itself; with the resulting message that printed
when I manually sent SIGCONT to the hung zsh process giving an indication why. 
It may be that all you need to do is add a check in the code that tries to
manipulate the tty pgrp on completion of a monitored task to skip the
manipulation if the shell is not interactive.  Here's my shot at that patch,
which passed the above test case for me (module the bug that fg still didn't
print anything to stdout), but I have no idea if it is doing the correct thing.

Meanwhile, Autoconf does not use fg or bg in implementing parallel tests, so I
will try to find time to see whether my patch below is sufficient to get through
a parallel testsuite run.



From: Eric Blake <ebb9@byu.net>
Date: Thu, 9 Jul 2009 12:04:58 -0600
Subject: [PATCH] 27100: Touch up non-interactive MONITOR handling.

---
 ChangeLog   |    6 ++++++
 Src/init.c  |    2 +-
 Src/utils.c |    2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 3a4ca1f..2797835 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2009-07-09  Eric Blake  <ebb9@byu.net>
+
+	* 27100: Touch up non-interactive MONITOR handling.
+	Don't enable MONITOR when initializing non-interactive shells.
+	Don't mess with tty when MONITOR but not interactive.
+
 2009-07-08  Peter Stephenson  <pws@csr.com>

 	* 27100: Allow MONITOR option in non-interactive shells.
diff --git a/Src/init.c b/Src/init.c
index 1894519..41e5ecf 100644
--- a/Src/init.c
+++ b/Src/init.c
@@ -486,7 +486,7 @@ init_io(void)
      * process group leader.
      */
     mypid = (zlong)getpid();
-    if (opts[MONITOR] && (SHTTY != -1)) {
+    if (opts[MONITOR] && interact && (SHTTY != -1)) {
 	origpgrp = GETPGRP();
         acquire_pgrp(); /* might also clear opts[MONITOR] */
     } else
diff --git a/Src/utils.c b/Src/utils.c
index 510a02b..ad4ffca 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -3590,7 +3590,7 @@ attachtty(pid_t pgrp)
 {
     static int ep = 0;

-    if (jobbing) {
+    if (jobbing && interact) {
 #ifdef HAVE_TCSETPGRP
 	if (SHTTY != -1 && tcsetpgrp(SHTTY, pgrp) == -1 && !ep)
 #else
-- 
1.6.3.2

Re: non-interactive set -m

[Peter Stephenson]

On Thu, 9 Jul 2009 18:13:15 +0000 (UTC)
Eric Blake <ebb9@byu.net> wrote:
> As-is, your 2-hunk patch accidentally sets monitor mode by default for
> non-interactive shells:
> 
> $ Src/zsh -c 'echo $-'
> 569Xm

This is tricky:  there's the knock-on effect from INTERACTIVE, and the
possibility it might be set or unset.  I think the following does it.

Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.102
diff -u -r1.102 init.c
--- Src/init.c	8 Jul 2009 17:07:13 -0000	1.102
+++ Src/init.c	9 Jul 2009 19:33:44 -0000
@@ -233,6 +233,11 @@
      * be changed.  At the end of the function, a value of 2 gets        *
      * changed to 1.                                                     */
     opts[INTERACTIVE] = isatty(0) ? 2 : 0;
+    /*
+     * MONITOR is similar:  we initialise it to 2, and if it's
+     * still 2 at the end, we set it to the value of INTERACTIVE.
+     */
+    opts[MONITOR] = 2;   /* may be unset in init_io() */
     opts[SHINSTDIN] = 0;
     opts[SINGLECOMMAND] = 0;
 
@@ -343,6 +348,8 @@
     if(isset(SINGLECOMMAND))
 	opts[INTERACTIVE] &= 1;
     opts[INTERACTIVE] = !!opts[INTERACTIVE];
+    if (opts[MONITOR] == 2)
+	opts[MONITOR] = opts[INTERACTIVE];
     pparams = x = (char **) zshcalloc((countlinknodes(paramlist) + 1) * sizeof(char *));
 
     while ((*x++ = (char *)getlinknode(paramlist)));
@@ -1395,7 +1402,6 @@
     createoptiontable();
     emulate(zsh_name, 1);   /* initialises most options */
     opts[LOGINSHELL] = (**argv == '-');
-    opts[MONITOR] = 1;   /* may be unset in init_io() */
     opts[PRIVILEGED] = (getuid() != geteuid() || getgid() != getegid());
     opts[USEZLE] = 1;   /* may be unset in init_io() */
     parseargs(argv);   /* sets INTERACTIVE, SHINSTDIN and SINGLECOMMAND */


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: non-interactive set -m

[Peter Stephenson]

On Thu, 9 Jul 2009 18:13:15 +0000 (UTC)
Eric Blake <ebb9@byu.net> wrote:
> It may be that all you need to do is add a check in the code that tries to
> manipulate the tty pgrp on completion of a monitored task to skip the
> manipulation if the shell is not interactive.  Here's my shot at that patch,
> which passed the above test case for me (module the bug that fg still didn't
> print anything to stdout), but I have no idea if it is doing the
> correct thing.

(This effectively disables attachtty() if non-interactive.)

It seems entirely reasonable not to try to take over the terminal if
you're in a shell that didn't think it was controlling the terminal in
the first place.  I'm not the job control expert, but since this doesn't
affect the usual case anyway I've committed it.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: non-interactive set -m

[Peter Stephenson]

On Thu, 9 Jul 2009 18:13:15 +0000 (UTC)
Eric Blake <ebb9@byu.net> wrote:
> It looks like the fg activated the sleep process, because it took five seconds
> for the Stopped message to appear, but fg failed to print to stdout
> the process that it just activated.

... and the third part: this seems to be another question of testing if
MONITOR is set (jobbing---I wish we didn't those #define's) but not if
INTERACTIVE is set (interact).  Also need to fix up the output to go to
stderr instead of the file (not) opened to the terminal.

I've done the same thing at another point, where the status of a job
changes.  You're the guinea pig so if you think this output should or
shouldn't be there I can leave it or change it.

As before, none of this should change the usual case where the shell is
also interactive.

Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.66
diff -u -r1.66 jobs.c
--- Src/jobs.c	16 Mar 2009 05:20:36 -0000	1.66
+++ Src/jobs.c	9 Jul 2009 20:16:47 -0000
@@ -828,7 +828,7 @@
     int job, len = 9, sig, sflag = 0, llen;
     int conted = 0, lineleng = columns, skip = 0, doputnl = 0;
     int doneprint = 0;
-    FILE *fout = (synch == 2) ? stdout : shout;
+    FILE *fout = (synch == 2) ? stdout : shout ? shout : stderr;
 
     if (oldjobtab != NULL)
 	job = jn - oldjobtab;
@@ -887,7 +887,7 @@
 /* print if necessary: ignore option state on explicit call to `jobs'. */
 
     if (synch == 2 || 
-	(interact && jobbing &&
+	(jobbing &&
 	 ((jn->stat & STAT_STOPPED) || sflag || job != thisjob))) {
 	int len2, fline = 1;
 	/* use special format for current job, except in `jobs' */
@@ -1379,12 +1379,13 @@
 	    setprevjob();
 	} else if (prevjob == -1 || !(jobtab[prevjob].stat & STAT_STOPPED))
 	    prevjob = thisjob;
-	if (interact && jobbing && jobtab[thisjob].procs) {
-	    fprintf(shout, "[%d]", thisjob);
+	if (jobbing && jobtab[thisjob].procs) {
+	    FILE *fout = shout ? shout : stderr;
+	    fprintf(fout, "[%d]", thisjob);
 	    for (pn = jobtab[thisjob].procs; pn; pn = pn->next)
-		fprintf(shout, " %ld", (long) pn->pid);
-	    fprintf(shout, "\n");
-	    fflush(shout);
+		fprintf(fout, " %ld", (long) pn->pid);
+	    fprintf(fout, "\n");
+	    fflush(fout);
 	}
     }
     if (!hasprocs(thisjob))
@@ -1907,7 +1908,8 @@
 		printjob(jobtab + job, (stopped) ? -1 : lng, 1);
 	    if (func != BIN_BG) {		/* fg or wait */
 		if (jobtab[job].pwd && strcmp(jobtab[job].pwd, pwd)) {
-		    FILE *fout = (func == BIN_JOBS) ? stdout : shout;
+		    FILE *fout = (func == BIN_JOBS) ? stdout : shout ?
+			shout : stderr;
 		    fprintf(fout, "(pwd : ");
 		    fprintdir(jobtab[job].pwd, fout);
 		    fprintf(fout, ")\n");


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: non-interactive set -m

[Eric Blake]

Peter Stephenson <p.w.stephenson <at> ntlworld.com> writes:

> > It looks like the fg activated the sleep process, because it took five
> > seconds for the Stopped message to appear, but fg failed to print to stdout
> > the process that it just activated.
> 
> ... and the third part: this seems to be another question of testing if
> MONITOR is set (jobbing---I wish we didn't those #define's) but not if
> INTERACTIVE is set (interact).  Also need to fix up the output to go to
> stderr instead of the file (not) opened to the terminal.

POSIX states that fg output goes to stdout, not stderr.  Thus, I should have a
silent wait for 5 seconds with the following command; although with this latest
round of patches, things are still noisy:

$ Src/zsh -c 'set -m; sleep 5& fg >/dev/null'
[1] 5804
[1]  + running    sleep 5
$

It looks like two different sources of noise - creating the background task is
outputting information about the job just created, which should really be
conditional on being interactive (a non-interactive shell will use 'jobs' to
learn this information, rather than having it printed to stderr).  The other
source of noise is that fg is writing to stderr instead of the required stdout.

Also, POSIX states that fg output should be the name of the command that was
brought into the foreground, and not the 'jobs' output describing that command.

Notice how bash does it:
$ bash -ci 'set -m; sleep 5& fg >/dev/null'
[1] 21428
$ bash -c 'set -m; sleep 5& fg'
sleep 5
$ bash -c 'set -m; sleep 5& fg >/dev/null'
$

> I've done the same thing at another point, where the status of a job
> changes.  You're the guinea pig so if you think this output should or
> shouldn't be there I can leave it or change it.

Not a problem - I'm actually having fun trying to test this.  At any rate, given
my (lack of) familiarity with the zsh code layout, reporting my tests is easier
than trying to propose patches, although I'll do what I can.

-- 
Eric Blake

Re: non-interactive set -m

[Eric Blake]

Peter Stephenson <p.w.stephenson <at> ntlworld.com> writes:

> I've done the same thing at another point, where the status of a job
> changes.  You're the guinea pig so if you think this output should or
> shouldn't be there I can leave it or change it.

Still another thing to fix:

$ Src/zsh -c 'set -m; echo $?'
0
$ Src/zsh -c '(set -m); echo $?'
zsh:set:1: can't change option: -m

So something about creating a subshell is interfering with the ability to enable
the monitor option.

-- 
Eric Blake

Re: non-interactive set -m

[Eric Blake]

Eric Blake <ebb9 <at> byu.net> writes:

> Still another thing to fix:
> 
> $ Src/zsh -c 'set -m; echo $?'
> 0
> $ Src/zsh -c '(set -m); echo $?'
> zsh:set:1: can't change option: -m
> 
> So something about creating a subshell is interfering with the ability to
> enable the monitor option.

This last finding is a pre-existing condition that afflicts interactive shells
too:

$ zsh --version
zsh 4.3.9 (i386-pc-cygwin)
$ zsh -ci '(set -m); echo $?'
zsh:set:1: can't change option: -m
1
$

-- 
Eric Blake

Re: non-interactive set -m

[Bart Schaefer]

On Jul 9,  8:56pm, Peter Stephenson wrote:
}
} It seems entirely reasonable not to try to take over the terminal if
} you're in a shell that didn't think it was controlling the terminal in
} the first place.  I'm not the job control expert, but since this doesn't
} affect the usual case anyway I've committed it.

I don't know if I qualify as the expert, either, but IMO this should
be OK.

Re: non-interactive set -m

[Peter Stephenson]

On Thu, 9 Jul 2009 21:40:19 +0000 (UTC)
Eric Blake <ebb9@byu.net> wrote:
> POSIX states that fg output goes to stdout, not stderr.
>...
> It looks like two different sources of noise - creating the background task is
> outputting information about the job just created, which should really be
> conditional on being interactive (a non-interactive shell will use 'jobs' to
> learn this information, rather than having it printed to stderr).  The
> other source of noise is that fg is writing to stderr instead of the
> required stdout.

It looks like you didn't want the other "if (interact)" removing.  This
should fix these two.

> Also, POSIX states that fg output should be the name of the command that was
> brought into the foreground, and not the 'jobs' output describing that
> command.

Haven't looked at this, but as an annoying fiddly difference it probably
needs to be an option.

Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.67
diff -u -r1.67 jobs.c
--- Src/jobs.c	9 Jul 2009 20:33:56 -0000	1.67
+++ Src/jobs.c	10 Jul 2009 08:56:26 -0000
@@ -828,7 +828,7 @@
     int job, len = 9, sig, sflag = 0, llen;
     int conted = 0, lineleng = columns, skip = 0, doputnl = 0;
     int doneprint = 0;
-    FILE *fout = (synch == 2) ? stdout : shout ? shout : stderr;
+    FILE *fout = (synch == 2 || !shout) ? stdout : shout;
 
     if (oldjobtab != NULL)
 	job = jn - oldjobtab;
@@ -886,8 +886,8 @@
 
 /* print if necessary: ignore option state on explicit call to `jobs'. */
 
-    if (synch == 2 || 
-	(jobbing &&
+    if (synch == 2 ||
+	(interact && jobbing &&
 	 ((jn->stat & STAT_STOPPED) || sflag || job != thisjob))) {
 	int len2, fline = 1;
 	/* use special format for current job, except in `jobs' */
@@ -1380,7 +1380,7 @@
 	} else if (prevjob == -1 || !(jobtab[prevjob].stat & STAT_STOPPED))
 	    prevjob = thisjob;
 	if (jobbing && jobtab[thisjob].procs) {
-	    FILE *fout = shout ? shout : stderr;
+	    FILE *fout = shout ? shout : stdout;
 	    fprintf(fout, "[%d]", thisjob);
 	    for (pn = jobtab[thisjob].procs; pn; pn = pn->next)
 		fprintf(fout, " %ld", (long) pn->pid);
@@ -1908,8 +1908,7 @@
 		printjob(jobtab + job, (stopped) ? -1 : lng, 1);
 	    if (func != BIN_BG) {		/* fg or wait */
 		if (jobtab[job].pwd && strcmp(jobtab[job].pwd, pwd)) {
-		    FILE *fout = (func == BIN_JOBS) ? stdout : shout ?
-			shout : stderr;
+		    FILE *fout = (func == BIN_JOBS || !shout) ? stdout : shout;
 		    fprintf(fout, "(pwd : ");
 		    fprintdir(jobtab[job].pwd, fout);
 		    fprintf(fout, ")\n");


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: non-interactive set -m

[Peter Stephenson]

On Fri, 10 Jul 2009 09:58:13 +0100
Peter Stephenson <pws@csr.com> wrote:
> On Thu, 9 Jul 2009 21:40:19 +0000 (UTC)
> > Also, POSIX states that fg output should be the name of the command that was
> > brought into the foreground, and not the 'jobs' output describing that
> > command.
> 
> Haven't looked at this, but as an annoying fiddly difference it probably
> needs to be an option.

Straightforward if an annoying increase in mess.

I've partly gone back on the previous going back on the previous patch that
printed a message if non-interactive but with job-control---I think we need
to do it if synchronous, so we get the output from "fg" and "bg".

Index: Doc/Zsh/options.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/options.yo,v
retrieving revision 1.82
diff -u -r1.82 options.yo
--- Doc/Zsh/options.yo	17 May 2009 18:23:10 -0000	1.82
+++ Doc/Zsh/options.yo	10 Jul 2009 10:48:57 -0000
@@ -1274,6 +1274,17 @@
 Report the status of background jobs immediately, rather than
 waiting until just before printing a prompt.
 )
+pindex(POSIX_JOBS)
+pindex(POSIXJOBS)
+pindex(NO_POSIX_JOBS)
+pindex(NOPOSIXJOBS)
+cindex(bg, output in POSIX format)
+cindex(fg, output in POSIX format)
+item(tt(POSIX_JOBS) <K> <S>)(
+When putting jobs in the background or foreground with tt(bg) or tt(fg),
+just print the text of the job as required by POSIX,
+rather than the full information that would be output by tt(jobs).
+)
 enditem()
 
 subsect(Prompting)
Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.68
diff -u -r1.68 jobs.c
--- Src/jobs.c	10 Jul 2009 09:11:12 -0000	1.68
+++ Src/jobs.c	10 Jul 2009 10:48:57 -0000
@@ -813,6 +813,7 @@
  * synch = 0 means asynchronous
  * synch = 1 means synchronous
  * synch = 2 means called synchronously from jobs
+ * synch = 3 means called synchronously from bg or fg
  *
  * Returns 1 if some output was done.
  *
@@ -884,12 +885,18 @@
 	}
     }
 
-/* print if necessary: ignore option state on explicit call to `jobs'. */
-
+    /*
+     * - Always print if called from jobs
+     * - Otherwise, require MONITOR option ("jobbing") and some
+     *   change of state
+     * - also either the shell is interactive or this is synchronous.
+     */
     if (synch == 2 ||
-	(interact && jobbing &&
+	((interact || synch) && jobbing &&
 	 ((jn->stat & STAT_STOPPED) || sflag || job != thisjob))) {
 	int len2, fline = 1;
+	/* POSIX requires just the job text for bg and fg */
+	int plainfmt = (synch == 3) && isset(POSIXJOBS);
 	/* use special format for current job, except in `jobs' */
 	int thisfmt = job == thisjob && synch != 2;
 	Process qn;
@@ -908,54 +915,60 @@
 		for (qn = pn->next; qn; qn = qn->next) {
 		    if (qn->status != pn->status)
 			break;
-		    if ((int)strlen(qn->text) + len2 + ((qn->next) ? 3 : 0) > lineleng)
+		    if ((int)strlen(qn->text) + len2 + ((qn->next) ? 3 : 0)
+			> lineleng)
 			break;
 		    len2 += strlen(qn->text) + 2;
 		}
 	    doneprint = 1;
-	    if (!thisfmt || lng) {
-		if (fline)
-		    fprintf(fout, "[%ld]  %c ",
-			    (long)job,
-			    (job == curjob) ? '+'
-			    : (job == prevjob) ? '-' : ' ');
-		else
-		    fprintf(fout, (job > 9) ? "        " : "       ");
-	    } else
-		fprintf(fout, "zsh: ");
-	    if (lng & 1)
-		fprintf(fout, "%ld ", (long) pn->pid);
-	    else if (lng & 2) {
-		pid_t x = jn->gleader;
-
-		fprintf(fout, "%ld ", (long) x);
-		do
+	    if (!plainfmt) {
+		if (!thisfmt || lng) {
+		    if (fline)
+			fprintf(fout, "[%ld]  %c ",
+				(long)job,
+				(job == curjob) ? '+'
+				: (job == prevjob) ? '-' : ' ');
+		    else
+			fprintf(fout, (job > 9) ? "        " : "       ");
+		} else
+		    fprintf(fout, "zsh: ");
+		if (lng & 1)
+		    fprintf(fout, "%ld ", (long) pn->pid);
+		else if (lng & 2) {
+		    pid_t x = jn->gleader;
+
+		    fprintf(fout, "%ld ", (long) x);
+		    do
+			skip++;
+		    while ((x /= 10));
 		    skip++;
-		while ((x /= 10));
-		skip++;
-		lng &= ~3;
-	    } else
-		fprintf(fout, "%*s", skip, "");
-	    if (pn->status == SP_RUNNING) {
-		if (!conted)
-		    fprintf(fout, "running%*s", len - 7 + 2, "");
+		    lng &= ~3;
+		} else
+		    fprintf(fout, "%*s", skip, "");
+		if (pn->status == SP_RUNNING) {
+		    if (!conted)
+			fprintf(fout, "running%*s", len - 7 + 2, "");
+		    else
+			fprintf(fout, "continued%*s", len - 9 + 2, "");
+		}
+		else if (WIFEXITED(pn->status)) {
+		    if (WEXITSTATUS(pn->status))
+			fprintf(fout, "exit %-4d%*s", WEXITSTATUS(pn->status),
+				len - 9 + 2, "");
+		    else
+			fprintf(fout, "done%*s", len - 4 + 2, "");
+		} else if (WIFSTOPPED(pn->status))
+		    fprintf(fout, "%-*s", len + 2,
+			    sigmsg(WSTOPSIG(pn->status)));
+		else if (WCOREDUMP(pn->status))
+		    fprintf(fout, "%s (core dumped)%*s",
+			    sigmsg(WTERMSIG(pn->status)),
+			    (int)(len - 14 + 2 -
+				  strlen(sigmsg(WTERMSIG(pn->status)))), "");
 		else
-		    fprintf(fout, "continued%*s", len - 9 + 2, "");
+		    fprintf(fout, "%-*s", len + 2,
+			    sigmsg(WTERMSIG(pn->status)));
 	    }
-	    else if (WIFEXITED(pn->status)) {
-		if (WEXITSTATUS(pn->status))
-		    fprintf(fout, "exit %-4d%*s", WEXITSTATUS(pn->status),
-			    len - 9 + 2, "");
-		else
-		    fprintf(fout, "done%*s", len - 4 + 2, "");
-	    } else if (WIFSTOPPED(pn->status))
-		fprintf(fout, "%-*s", len + 2, sigmsg(WSTOPSIG(pn->status)));
-	    else if (WCOREDUMP(pn->status))
-		fprintf(fout, "%s (core dumped)%*s",
-			sigmsg(WTERMSIG(pn->status)),
-			(int)(len - 14 + 2 - strlen(sigmsg(WTERMSIG(pn->status)))), "");
-	    else
-		fprintf(fout, "%-*s", len + 2, sigmsg(WTERMSIG(pn->status)));
 	    for (; pn != qn; pn = pn->next) {
 		char *txt = dupstring(pn->text);
 		int txtlen;
@@ -1904,7 +1918,7 @@
 	    }
 	    if (func != BIN_WAIT)
 		/* for bg and fg -- show the job we are operating on */
-		printjob(jobtab + job, (stopped) ? -1 : lng, 1);
+		printjob(jobtab + job, (stopped) ? -1 : lng, 3);
 	    if (func != BIN_BG) {		/* fg or wait */
 		if (jobtab[job].pwd && strcmp(jobtab[job].pwd, pwd)) {
 		    FILE *fout = (func == BIN_JOBS || !shout) ? stdout : shout;
Index: Src/options.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/options.c,v
retrieving revision 1.49
diff -u -r1.49 options.c
--- Src/options.c	8 Jul 2009 17:07:14 -0000	1.49
+++ Src/options.c	10 Jul 2009 10:48:57 -0000
@@ -201,6 +201,7 @@
 {{NULL, "posixaliases",       OPT_EMULATE|OPT_BOURNE},	 POSIXALIASES},
 {{NULL, "posixbuiltins",      OPT_EMULATE|OPT_BOURNE},	 POSIXBUILTINS},
 {{NULL, "posixidentifiers",   OPT_EMULATE|OPT_BOURNE},	 POSIXIDENTIFIERS},
+{{NULL, "posixjobs",          OPT_EMULATE|OPT_BOURNE},	 POSIXJOBS},
 {{NULL, "printeightbit",      0},                        PRINTEIGHTBIT},
 {{NULL, "printexitvalue",     0},			 PRINTEXITVALUE},
 {{NULL, "privileged",	      OPT_SPECIAL},		 PRIVILEGED},
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.158
diff -u -r1.158 zsh.h
--- Src/zsh.h	2 Jul 2009 13:48:36 -0000	1.158
+++ Src/zsh.h	10 Jul 2009 10:48:57 -0000
@@ -1955,6 +1956,7 @@
     POSIXALIASES,
     POSIXBUILTINS,
     POSIXIDENTIFIERS,
+    POSIXJOBS,
     PRINTEIGHTBIT,
     PRINTEXITVALUE,
     PRIVILEGED,

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: non-interactive set -m

[Peter Stephenson]

On Fri, 10 Jul 2009 03:32:59 +0000 (UTC)
Eric Blake <ebb9@byu.net> wrote:
> Eric Blake <ebb9 <at> byu.net> writes:
> > > $ Src/zsh -c '(set -m); echo $?'
> > zsh:set:1: can't change option: -m
>
> $ zsh -ci '(set -m); echo $?'
> zsh:set:1: can't change option: -m

This is getting murkier.

When we enter a subshell, we close the file descriptor pointing to the
terminal.  It's not clear to me this is strictly necessary: if we have a
non-interactive shell, we will open that file descriptor if we can and
keep it around permanently (that's why the initial patch to allow
setting -m non-interatively was so simple), so obviously it's not
strictly necessary to close it when a shell isn't interactive / doesn't
have job control.

What I think is necessary is getting rid of "shout", the FILE opened to
the terminal, which is strictly for interactive use only and hence
inappropriate in a subshell (we don't want the subshell trying to do
line editing, etc.).  So it's possible the following patch makes things
work.  (The other chunk is because we allow "jobs" to work in a subshell
so you can pipe its output---if you "set -m" you're effectively making
the shell the output for "jobs" twice.)

Again, this passed a trivial smoke test.  From the non-interactive case
I think keeping SHTTY around is benign, but we're heading into
more difficult territory.

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.168
diff -u -r1.168 exec.c
--- Src/exec.c	11 Jul 2009 16:43:00 -0000	1.168
+++ Src/exec.c	11 Jul 2009 18:52:56 -0000
@@ -931,11 +931,7 @@
     zsh_subshell++;
     if ((flags & ESUB_REVERTPGRP) && getpid() == mypgrp)
 	release_pgrp();
-    if (SHTTY != -1) {
-	shout = NULL;
-	zclose(SHTTY);
-	SHTTY = -1;
-    }
+    shout = NULL;
     if (isset(MONITOR)) {
 	signal_default(SIGTTOU);
 	signal_default(SIGTTIN);
Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.69
diff -u -r1.69 jobs.c
--- Src/jobs.c	10 Jul 2009 11:08:48 -0000	1.69
+++ Src/jobs.c	11 Jul 2009 18:52:56 -0000
@@ -1307,7 +1307,8 @@
 
     if (monitor && oldmaxjob) {
 	int sz = oldmaxjob * sizeof(struct job);
-	DPUTS(oldjobtab != NULL, "BUG: saving job table twice\n");
+	if (oldjobtab)
+	    free(oldjobtab);
 	oldjobtab = (struct job *)zalloc(sz);
 	memcpy(oldjobtab, jobtab, sz);
 
 
-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: non-interactive set -m

[Peter Stephenson]

Peter Stephenson wrote:
> (The other chunk is because we allow "jobs" to work in a subshell
> so you can pipe its output---if you "set -m" you're effectively making
> the shell the output for "jobs" twice.)
           ^

The missing word is "save".  When you do "set -m" in the subshell you're
wiping the record of the parent shell's job control---that's inevitable
if you now want the subshell to do its own job control.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: non-interactive set -m

[Eric Blake]

Peter Stephenson <p.w.stephenson <at> ntlworld.com> writes:

> 
> Peter Stephenson wrote:
> > (The other chunk is because we allow "jobs" to work in a subshell
> > so you can pipe its output---if you "set -m" you're effectively making
> > the shell the output for "jobs" twice.)
>            ^
> 
> The missing word is "save".  When you do "set -m" in the subshell you're
> wiping the record of the parent shell's job control---that's inevitable
> if you now want the subshell to do its own job control.
> 

POSIX says the following about sh:

A shell execution environment consists of the following:
...
Options turned on at invocation or by set
...
Process IDs of the last commands in asynchronous lists known to this shell
environment;
...
A subshell environment shall be created as a duplicate of the shell environment,
except that signal traps set by that shell environment shall be set to the
default values. 


It also says, for wait:

If it is called in a subshell or separate utility execution environment, such as
one of the following:
(wait)
...
it returns immediately because there are no known process IDs to wait for in
those environments.


I interpret this to mean that subshells of a parent process with job control on
must also have job control on, but that their list of tracked processes is nuked.

So we are getting closer, but still not there.

$ zsh -ci 'echo $-; (echo $-)'

should have identical output on both lines (the subshell should not lose the
fact that -m was set in the parent shell).

Likewise, I argue that this should follow bash's behavior, and not print
anything between 'got' and 'here' since there are no jobs known to the subshell:

$ zsh -ci 'sleep 1& (echo got; jobs; echo here)'

-- 
Eric Blake

Re: non-interactive set -m

[Peter Stephenson]

On Sat, 11 Jul 2009 19:57:02 +0100
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> Again, this passed a trivial smoke test.  From the non-interactive case
> I think keeping SHTTY around is benign, but we're heading into
> more difficult territory.

We should make sure it's closed on an exec, however.  This maybe what
the previous code was trying to do.

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.169
diff -u -r1.169 exec.c
--- Src/exec.c	11 Jul 2009 19:06:52 -0000	1.169
+++ Src/exec.c	12 Jul 2009 14:38:36 -0000
@@ -595,6 +595,12 @@
      * here, which should be visible to external processes.
      */
     closem(FDT_XTRACE);
+#ifndef FD_CLOEXEC
+    if (SHTTY != -1) {
+	close(SHTTY);
+	SHTTY = -1;
+    }
+#endif
     child_unblock();
     if ((int) strlen(arg0) >= PATH_MAX) {
 	zerr("command too long: %s", arg0);
Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.105
diff -u -r1.105 init.c
--- Src/init.c	11 Jul 2009 16:43:00 -0000	1.105
+++ Src/init.c	12 Jul 2009 14:38:36 -0000
@@ -480,8 +480,16 @@
     if (SHTTY == -1) {
 	zsfree(ttystrname);
 	ttystrname = ztrdup("");
-    } else if (!ttystrname) {
-	ttystrname = ztrdup("/dev/tty");
+    } else {
+#ifdef FD_CLOEXEC
+	long fdflags = fcntl(SHTTY, F_GETFD, 0);
+	if (fdflags != (long)-1) {
+	    fdflags |= FD_CLOEXEC;
+	    fcntl(SHTTY, F_SETFD, fdflags);
+	}
+#endif
+	if (!ttystrname)
+	    ttystrname = ztrdup("/dev/tty");
     }
 
     /* We will only use zle if shell is interactive, *

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: non-interactive set -m

[Peter Stephenson]

On Sat, 11 Jul 2009 23:16:51 +0000 (UTC)
Eric Blake <ebb9@byu.net> wrote:
> I interpret this to mean that subshells of a parent process with job
> control on must also have job control on, but that their list of
> tracked processes is nuked.

These look fairly straightforward, if we don't run into problems later
on.  They can be attached to POSIX_JOBS---that's why I gave it a generic
name originally.

This looks to me like quite a big change in the way background processes
from subshells work, but if it's what POSIX users expects that's
fine---I'm not expecting anyone to turn the option on just for fun.

Index: Doc/Zsh/options.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/options.yo,v
retrieving revision 1.83
diff -u -r1.83 options.yo
--- Doc/Zsh/options.yo	10 Jul 2009 11:08:48 -0000	1.83
+++ Doc/Zsh/options.yo	12 Jul 2009 14:59:11 -0000
@@ -1280,10 +1280,26 @@
 pindex(NOPOSIXJOBS)
 cindex(bg, output in POSIX format)
 cindex(fg, output in POSIX format)
+cindex(job control, in subshell)
+cindex(jobs, output in subshell)
 item(tt(POSIX_JOBS) <K> <S>)(
-When putting jobs in the background or foreground with tt(bg) or tt(fg),
-just print the text of the job as required by POSIX,
-rather than the full information that would be output by tt(jobs).
+This option makes job control more compliant with the POSIX standard.
+
+When the option is not set, the tt(MONITOR) option is unset on entry to
+subshells, so that job control is no longer active.  When the option is
+set, the tt(MONITOR) option and job control remain active in the
+subshell, but note that the subshell has no access to jobs in the parent
+shell.
+
+When the option is not set, jobs put in the background or foreground
+with tt(bg) or tt(fg) are displayed with the same information that would
+be reported by tt(jobs).  When the option is set, only the text is
+printed.  The output from tt(jobs) itself is not affected by the option.
+
+When the option is not set, job information from the parent
+shell is saved for output within a subshell (for example, within a
+pipeline).  When the option is set, the output of tt(jobs) is empty
+unless job control has been activated within the subshell.
 )
 enditem()
 
Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.169
diff -u -r1.169 exec.c
--- Src/exec.c	11 Jul 2009 19:06:52 -0000	1.169
+++ Src/exec.c	12 Jul 2009 14:59:12 -0000
@@ -944,7 +950,9 @@
     }
     if (!(sigtrapped[SIGQUIT] & ZSIG_IGNORED))
 	signal_default(SIGQUIT);
-    opts[MONITOR] = opts[USEZLE] = 0;
+    if (!isset(POSIXJOBS))
+	opts[MONITOR] = 0;
+    opts[USEZLE] = 0;
     zleactive = 0;
     if (flags & ESUB_PGRP)
 	clearjobtab(monitor);
Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.70
diff -u -r1.70 jobs.c
--- Src/jobs.c	11 Jul 2009 19:06:52 -0000	1.70
+++ Src/jobs.c	12 Jul 2009 14:59:12 -0000
@@ -1292,6 +1292,8 @@
 {
     int i;
 
+    if (isset(POSIXJOBS))
+	oldmaxjob = 0;
     for (i = 1; i <= maxjob; i++) {
 	/*
 	 * See if there is a jobtable worth saving.
@@ -1299,7 +1301,7 @@
 	 * once for each subshell of a shell with job control,
 	 * so doesn't create a leak.
 	 */
-	if (monitor && jobtab[i].stat)
+	if (monitor && !isset(POSIXJOBS) && jobtab[i].stat)
 	    oldmaxjob = i+1;
 	else if (jobtab[i].stat & STAT_INUSE)
 	    freejob(jobtab + i, 0);


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: non-interactive set -m

[Bart Schaefer]

On Jul 12,  4:01pm, Peter Stephenson wrote:
}
} This looks to me like quite a big change in the way background processes
} from subshells work, but if it's what POSIX users expects that's
} fine---I'm not expecting anyone to turn the option on just for fun.

If zsh was previously in the wrong here, then so is pdksh:

schaefer[507] ksh                                                         4.2.0
$ sleep 30 &
[1] 14322
$ (jobs)
[1] + Running              sleep 30 
$ jobs | head
[1] + Running              sleep 30 
$ (wait)
$ (fg)
ksh: fg: job control not enabled
$ [1] + Done                 sleep 30 

Now, in pdksh "jobs | head" the "jobs" command is in the foreground
shell and "head" is forked off, whereas in zsh both "jobs" and "head"
will have been forked because of handling of this case ...

$ jobs | read line
$ echo $line

$ jobs | { read line; echo $line;}
[1] + Running sleep 30
$ 

... where in zsh the first "echo $line" will work (but won't any more
with this patch and POSIX_JOBS set).

POSIX doesn't specify whether the left or right side of a pipeline
will run in the current shell, which means that with POSIX_JOBS set
and this patch applied zsh will produce different results for the
piping of "jobs" to something.  Since it's only been about a year (?)
since we went through all the rigamarole of making "jobs" in a sub-
shell have access to (a snapshot of) the parent's job table, it seems
hasty to break that again even an as option.

Also look here:

$ sleep 30 &
[1] 14341
$ ( sleep 40 & jobs )
[2] + Running              sleep 40 
[1] - Running              sleep 30 
$ 

Note that in pdksh the job in the subshell is added to the table of
jobs inherited from the parent shell (testing with pdksh 5.2.14).

This may be something we should ask about on the austin-group list
before we select an interpretation and implement it.  I'm currently
not able to post there because of some problem with the listserv's
interpretation of the "+" in the email address I used to sign up.

Re: non-interactive set -m

[Peter Stephenson]

Bart Schaefer wrote:
> POSIX doesn't specify whether the left or right side of a pipeline
> will run in the current shell, which means that with POSIX_JOBS set
> and this patch applied zsh will produce different results for the
> piping of "jobs" to something.  Since it's only been about a year (?)
> since we went through all the rigamarole of making "jobs" in a sub-
> shell have access to (a snapshot of) the parent's job table, it seems
> hasty to break that again even an as option.

It seems to me it should one thing or the other---either have job
control in the shell with its own separate job table, or report the
parent's.  A mixture is madness.

> Also look here:
> 
> $ sleep 30 &
> [1] 14341
> $ ( sleep 40 & jobs )
> [2] + Running              sleep 40 
> [1] - Running              sleep 30 
> $ 
> 
> Note that in pdksh the job in the subshell is added to the table of
> jobs inherited from the parent shell (testing with pdksh 5.2.14).

Yuk.  So you've got a real job you can manipulate, and one phantom job
from the parent shell you can't and which is presumably fixed like that
for eternity.  In other words, pdksh (unlike zsh) doesn't clear the job
table while keeping a copy of the parent's around, it just uses the same
job table.  This doesn't strike me as the right way to do it.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

Hi,

I've found a problem with zsh 4.3.10 (and some previous versions, as
this bug has occurred for quite a long time, and I've identified it
only now). I could trigger it only under Mac OS X (version 10.4.11).

Consider a script with a line like

  /usr/bin/emacs -Q -nw

or

  /Applications/MacPorts/Emacs.app/Contents/MacOS/Emacs -Q -nw

(the problem doesn't depend on the version of emacs, e.g. 21 or 22).

Run it with:

  zsh -f ./script

and type Ctrl-G in Emacs. Then zsh terminates with a SIGINT, killing
Emacs at the same time.

This problem also occurs with ksh 1993-12-28 p, but not with bash.
With zsh 4.2.3, the SINGINT occurs only after Emacs terminates (e.g.
by quitting it with C-x C-c).

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: non-interactive set -m

[Bart Schaefer]

On Jul 12,  8:35pm, Peter Stephenson wrote:
} Subject: Re: non-interactive set -m
}
} Bart Schaefer wrote:
} > POSIX doesn't specify whether the left or right side of a pipeline
} > will run in the current shell, which means that with POSIX_JOBS set
} > and this patch applied zsh will produce different results for the
} > piping of "jobs" to something.
} 
} It seems to me it should one thing or the other---either have job
} control in the shell with its own separate job table, or report the
} parent's.  A mixture is madness.

I agree with *that*, but I'm wondering if the end result is that the
POSIX spec is self-contradictory [it wouldn't be the first time].
 
} > Note that in pdksh the job in the subshell is added to the table of
} > jobs inherited from the parent shell (testing with pdksh 5.2.14).
} 
} Yuk.  So you've got a real job you can manipulate, and one phantom job
} from the parent shell you can't and which is presumably fixed like that
} for eternity.

No, actually, you have two jobs neither of which you can manipulate.
PDKSH doesn't do job control in subshells.  It turns off -m when the
subshell starts, and silently ignores it if you manually turn it back
on again.

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

On Jul 12, 10:59pm, Vincent Lefevre wrote:
}
} Consider a script with a line like
} 
}   /usr/bin/emacs -Q -nw
} 
} Run it with:
} 
}   zsh -f ./script
} 
} and type Ctrl-G in Emacs. Then zsh terminates with a SIGINT, killing
} Emacs at the same time.

This must be happening because either (1) emacs resets the TTY intr
character to ^G or (2) emacs is sending a SIGINT to the terminal pgrp
when it sees ^G.  Experimenting with "trap 'stty -a' INT" seems to
point to the latter, but the SIGINT only makes it through to zsh once
(or at least the trap fires only on the first ^G).

Either way, I think this is a problem with emacs rather than zsh.  Try
replacing emacs with a command that doesn't do its own input handling
("sleep 10", perhaps) and then type the normal intr character (^C).
Do you expect the script to exit on *that* SIGINT?  [It does.]

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

On 2009-07-12 14:50:01 -0700, Bart Schaefer wrote:
> This must be happening because either (1) emacs resets the TTY intr
> character to ^G or (2) emacs is sending a SIGINT to the terminal pgrp
> when it sees ^G.  Experimenting with "trap 'stty -a' INT" seems to
> point to the latter, but the SIGINT only makes it through to zsh once
> (or at least the trap fires only on the first ^G).

With zsh 4.3.10, all of them: with the script

trap 'echo INT' INT
/usr/bin/emacs -Q -nw
echo OK

if I hit C-g 4 times, then C-x C-c to quit emacs, I get:

INT
INT
INT
INT
OK

when I run the script with zsh 4.3.10, but just

INT
OK

when I run the script with bash, ksh or zsh 4.2.3.
Why does zsh 4.3.10 behave differently?

> Either way, I think this is a problem with emacs rather than zsh.  Try
> replacing emacs with a command that doesn't do its own input handling
> ("sleep 10", perhaps) and then type the normal intr character (^C).
> Do you expect the script to exit on *that* SIGINT?  [It does.]

With bash, the script exits, even though it doesn't exit in the
case of Ctrl-g in Emacs.

I've searched on Google, and found the explanations of the various
behaviors:

  http://www.cons.org/cracauer/sigint.html

IMHO, zsh should implement WCE, just like bash.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: non-interactive set -m

[Eric Blake]

Bart Schaefer <schaefer <at> brasslantern.com> writes:

> } > Note that in pdksh the job in the subshell is added to the table of
> } > jobs inherited from the parent shell (testing with pdksh 5.2.14).
> } 
> } Yuk.  So you've got a real job you can manipulate, and one phantom job
> } from the parent shell you can't and which is presumably fixed like that
> } for eternity.
> 
> No, actually, you have two jobs neither of which you can manipulate.
> PDKSH doesn't do job control in subshells.  It turns off -m when the
> subshell starts, and silently ignores it if you manually turn it back
> on again.

My opinion is that pdksh is somewhat buggy when it comes to job handling, and
that you are better off trying to emulate David Korn's ksh93 than the buggy
pdksh; other better examples are dash and bash.  I already quoted the parts of
POSIX that mention that subshells inherit the same options as the parent (so the
pdksh behavior of silently disabling -m in a subshell violates that rule).

-- 
Eric Blake

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

On Jul 13,  2:43am, Vincent Lefevre wrote:
} 
} Why does zsh 4.3.10 behave differently?

Hmm.  There was a change to trap handling, but that should only matter
if LOCAL_TRAPS is set.  I also wanted there to have been some change
related to configure detecting restartable syscalls, but I can't find
anything in the ChangeLog about that.

IMO 4.3.10 has got it right and the older zsh et al. do not.

} I've searched on Google, and found the explanations of the various
} behaviors:
} 
}   http://www.cons.org/cracauer/sigint.html
} 
} IMHO, zsh should implement WCE, just like bash.

I'm not so sure.  You can emulate WCE with WUE and a trap, but there is
no way to make WCE do the right thing in the case of an ill-behaved
child that does not propagate the signal state through exit status.

Re: non-interactive set -m

[Eric Blake]

Bart Schaefer <schaefer <at> brasslantern.com> writes:

> POSIX doesn't specify whether the left or right side of a pipeline
> will run in the current shell, which means that with POSIX_JOBS set
> and this patch applied zsh will produce different results for the
> piping of "jobs" to something.

This is similar to the issue of traps being reset in a subshell - see
http://austingroupbugs.net/view.php?id=53 for recent Austin group comments on
that related topic.  I've gone ahead and added a comment to that report about
the question on shell options and jobs across subshells.

-- 
Eric Blake

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Peter Stephenson]

Bart Schaefer wrote:
> } I've searched on Google, and found the explanations of the various
> } behaviors:
> } 
> }   http://www.cons.org/cracauer/sigint.html
> } 
> } IMHO, zsh should implement WCE, just like bash.
> 
> I'm not so sure.  You can emulate WCE with WUE and a trap, but there is
> no way to make WCE do the right thing in the case of an ill-behaved
> child that does not propagate the signal state through exit status.

Could be an option (but I'm not writing it).

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

On 2009-07-13 19:39:24 +0100, Peter Stephenson wrote:
> Bart Schaefer wrote:
> > I'm not so sure.  You can emulate WCE with WUE and a trap,

I'll try to look at that.

> > but there is no way to make WCE do the right thing in the case of
> > an ill-behaved child that does not propagate the signal state
> > through exit status.

Well, since bash implements WCE and is the default shell on Linux and
Mac OS X machines, I'd say that such ill-behaved programs should not
be common (and should be fixed). So, IMHO, there would be more benefit
to have WCE in zsh.

> Could be an option (but I'm not writing it).

Yes, that would be nice, even though a trap emulating WCE could be as
simple (unless there are drawbacks).

In any case, I think that the behavior should be documented.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

On Jul 16,  6:24pm, Vincent Lefevre wrote:
}
} Well, since bash implements WCE and is the default shell on Linux and
} Mac OS X machines, I'd say that such ill-behaved programs should not
} be common (and should be fixed). So, IMHO, there would be more benefit
} to have WCE in zsh.

So I investigated this a bit further and re-learned something I'd long
forgotten:

When running a shell script (not interactively), zsh does NO HANDLING
of SIGINT whatsoever.  It's implementing IUE, not WUE.  Or to be more
precise, it's implementing the default behavior on SIGINT, which is
usually to exit.

In reminding myself of this, I believe I've also remembered the reason
that it ended up this way, which leads to points of disagreement with
the analysis in http://www.cons.org/cracauer/sigint.html (source of
the IUE/WUE/WCE abbreviations).

The point I believe Cracauer is overlooking is that the CALLER of the
shell script may have specified that SIGINT is to be ignored.  SIG_IGN
is supposed to be inherited by child processes; but to implement WUE
or WCE, the shell must itself either ignore or handle SIGINT, which
may contradict what the caller has requested.

My recollection now, which may be wrong, is that there was a considerable
debate about this several years ago, leading to the decision to simply
stick with the default signal handling when the shell is not interactive,
for any signal that it isn't otherwise absolutely necessary to handle.

Further I think Cracauer is very wrong here:

  Do nothing special when SIGINT appears while you wait for a child. You
  don't even have to remember that one happened.
  ...
  Look at WIFSIGNALED(status) and WTERMSIG(status) to tell whether the
  child says "I exited on SIGINT: in my opinion the user wants the
  shellscript to be discontinued".

This is plain nonsense.

Not only does this potentially contradict a caller's explicit request
to ignore SIGINT, but the script should not exit 130 every time any
child exits 130.  It should exit only when SIGINT was received *by the
script*.  "kill -INT ..." of the child should not cause the shell to
behave as if it was interrupted.  Try it with bash.

Having said all that, I also looked at what it would take to implement
WCE as an option.  There are two complications here:

(1) There's a race condition as to whether the shell receives SIGINT
or SIGCHLD first.  With enough trials I can get it to happen either
way, though the vast majority of the time the SIGINT wins.

(2) Both the exiting and the waiting normally take place in zhandler()
when the signal in question arrives.  This means that to get the option
right, we have to record the state of both signals and then find a
common place guaranteed to be outside the race where we can examine
the combined state.

There might be enough state currently available in zhandler() to test
it on either signal and do the correct thing, but I'm not sure yet.

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

On 2009-07-17 22:29:36 -0700, Bart Schaefer wrote:
> When running a shell script (not interactively), zsh does NO HANDLING
> of SIGINT whatsoever.  It's implementing IUE, not WUE.  Or to be more
> precise, it's implementing the default behavior on SIGINT, which is
> usually to exit.

Yes, this is what I can see with zsh 4.3.10 (but zsh 4.2.3 implemented
WUE).

> The point I believe Cracauer is overlooking is that the CALLER of the
> shell script may have specified that SIGINT is to be ignored.  SIG_IGN
> is supposed to be inherited by child processes; but to implement WUE
> or WCE, the shell must itself either ignore or handle SIGINT, which
> may contradict what the caller has requested.

I don't think that the caller should enforce what its children should
do. Many programs will set their own SIGINT handler, even if the caller
chose to ignore this signal. Anyway, even though bash implements WCE,
it ignores SIGINT if the caller chose to ignore it; also when executing
a process, it restores the trap to the default behavior, in case it was
ignored (that's another difference with zsh).

For instance, you can try with the following script with both bash
and zsh:

------------------------------------------------------------
echo "Caller: $$"
cmd="echo \"Callee: \$\$ ($SHELL)\"; sleep 6; echo OK"
trp="trap 'echo SIGINT; exit' INT; $cmd"

echo "Send SIGINT to each callee's PID"

$SHELL -c "$cmd"; echo "Callee's exit status: $?"
$SHELL -c "$trp"; echo "Callee's exit status: $?"

trap ''  INT
echo "SIGINT ignored"

$SHELL -c "$cmd"; echo "Callee's exit status: $?"
$SHELL -c "$trp"; echo "Callee's exit status: $?"

trap 'echo "Caller received SIGINT"'  INT
echo "Type Ctrl-C in the terminal"

$SHELL -c "$cmd"; echo "Callee's exit status: $?"
$SHELL -c "$trp"; echo "Callee's exit status: $?"
------------------------------------------------------------

Try also "killall -INT sleep" in the 6 cases.

> Further I think Cracauer is very wrong here:
> 
>   Do nothing special when SIGINT appears while you wait for a child. You
>   don't even have to remember that one happened.
>   ...
>   Look at WIFSIGNALED(status) and WTERMSIG(status) to tell whether the
>   child says "I exited on SIGINT: in my opinion the user wants the
>   shellscript to be discontinued".
> 
> This is plain nonsense.
> 
> Not only does this potentially contradict a caller's explicit request
> to ignore SIGINT, but the script should not exit 130 every time any
> child exits 130.  It should exit only when SIGINT was received *by the
> script*.  "kill -INT ..." of the child should not cause the shell to
> behave as if it was interrupted.  Try it with bash.

It seems that bash behaves correctly in all those cases. If the caller
ignores SIGINT, then bash also ignores it. Moreover bash terminates
with exit status 130 only when it receives SIGINT and the child
terminates with exit status 130. (I've tried on a Linux machine with
bash 3.2 because I don't have access to my Mac OS X machine right
now.)

> Having said all that, I also looked at what it would take to implement
> WCE as an option.  There are two complications here:
> 
> (1) There's a race condition as to whether the shell receives SIGINT
> or SIGCHLD first.  With enough trials I can get it to happen either
> way, though the vast majority of the time the SIGINT wins.

How does bash handle that?

BTW, I wonder why typing Ctrl-g in emacs sends SIGINT to the parent
under Mac OS X, but not under Linux.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

I'm going to send two replies to this part of the thread for different
subsets of Vincent's remarks, so that neither message gets too long.
This first one has a bunch of strace output in it.  Jump towards the
end for the punchline.

On Jul 18, 12:16pm, Vincent Lefevre wrote:
} Subject: Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in em
}
} On 2009-07-17 22:29:36 -0700, Bart Schaefer wrote:
} > When running a shell script (not interactively), zsh does NO HANDLING
} > of SIGINT whatsoever.  It's implementing IUE, not WUE.  Or to be more
} > precise, it's implementing the default behavior on SIGINT, which is
} > usually to exit.
} 
} Yes, this is what I can see with zsh 4.3.10 (but zsh 4.2.3 implemented
} WUE).

Let's look at that a bit more closely.

Here's 4.2.6 (slight difference in OS version may account for the
missing RT_1 in the sigprocmask):

pipe([3, 4])                            = 0
gettimeofday({1247938636, 336849}, {480, 0}) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0xb75e4868) = 27231
close(4)                                = 0
read(3, "", 1)                          = 0
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigsuspend(~[HUP CHLD RTMIN] <unfinished ...>
--- SIGCHLD (Child exited) @ 0 (0) ---
<... rt_sigsuspend resumed> )           = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_BLOCK, ~[RTMIN], ~[HUP KILL STOP RTMIN], 8) = 0
rt_sigprocmask(SIG_SETMASK, ~[HUP KILL STOP RTMIN], ~[KILL STOP RTMIN], 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WUNTRACED,
{ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 27231
gettimeofday({1247938666, 347837}, {480, 0}) = 0
wait4(-1, 0xbfffd14c, WNOHANG|WUNTRACED, 0xbfffd154) = -1 ECHILD (No child
processes)
sigreturn()                             = ? (mask now [CHLD])
--- SIGINT (Interrupt) @ 0 (0) ---

Now here's 4.2.6 with the TRAPS_ASYNC option set:

pipe([3, 4])                            = 0
gettimeofday({1247939573, 896574}, {480, 0}) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0xb75e0868) = 27252
close(4)                                = 0
read(3, "", 1)                          = 0
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigsuspend([] <unfinished ...>
--- SIGINT (Interrupt) @ 0 (0) ---

And finally 4.3.10-dev-1, doesn't matter whether TRAPS_ASYNC is set:

pipe([3, 4])                            = 0
gettimeofday({1247938336, 854096}, {420, 0}) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0xb7f6c708) = 10263
close(4)                                = 0
read(3, "", 1)                          = 0
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigsuspend([] <unfinished ...>
--- SIGINT (Interrupt) @ 0 (0) ---

This was changed in this patch ...

        * 23067: Doc/Zsh/builtins.yo, Src/jobs.c, Src/signals.c:
        queue traps but handle signals when waiting for jobs or processes,
        unless TRAPSASYNC is set or the wait builtin is in use, so as
        to handle untrapped signals in a timely fashion; document that
        negative or zero process IDs after kill may be handled specially
        by the OS.

... when the signal_suspend_setup() function was removed from signals.c
because of a problem with interrupting "wait".  This began in 23053 as
a MacOS bug report and relevant dicussion continues in 23055 and 23061.

Apparently you darn Mac users can't make up your mind. :-)

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

Part 2 of my reply, in which I argue philosophy rather than address the
specific shell problem.

On Jul 18, 12:16pm, Vincent Lefevre wrote:
} Subject: Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in em
}
} On 2009-07-17 22:29:36 -0700, Bart Schaefer wrote:
} > The point I believe Cracauer is overlooking is that the CALLER of the
} > shell script may have specified that SIGINT is to be ignored.  SIG_IGN
} > is supposed to be inherited by child processes; but to implement WUE
} > or WCE, the shell must itself either ignore or handle SIGINT, which
} > may contradict what the caller has requested.
} 
} I don't think that the caller should enforce what its children should
} do. Many programs will set their own SIGINT handler, even if the caller
} chose to ignore this signal.

You've missed the point.  SIG_IGN is by definition been inherited from
parent to child, and this is deliberately preserved across exec.  The
parent doesn't *enforce* this -- the child is *allowed* to set up a
handler of its own once it gets control -- but the mere fact that the
execv'd program is a shell should not void the inherited signal settings.
The shell might merely be an intermediary that's going to exec itself
out of existence, and the programmer of the parent should be able to
expect that the inherited signal settings make it through.

} Anyway, even though bash implements WCE,

Except that bash *DOESN'T* implement WCE, at least not the way that it
is defined in Cracauer's web page.  See below.

} it ignores SIGINT if the caller chose to ignore it; also when executing
} a process, it restores the trap to the default behavior, in case it was
} ignored (that's another difference with zsh).

And that's a case where I think the zsh behavior is in fact preferable.
 
} > Further I think Cracauer is very wrong here:
} > 
} >   Do nothing special when SIGINT appears while you wait for a child. You
} >   don't even have to remember that one happened.
} >   ...
} >   Look at WIFSIGNALED(status) and WTERMSIG(status) to tell whether the
} >   child says "I exited on SIGINT: in my opinion the user wants the
} >   shellscript to be discontinued".
} > 
} > This is plain nonsense.
} > 
} > Not only does this potentially contradict a caller's explicit request
} > to ignore SIGINT, but the script should not exit 130 every time any
} > child exits 130.  It should exit only when SIGINT was received *by the
} > script*.  "kill -INT ..." of the child should not cause the shell to
} > behave as if it was interrupted.  Try it with bash.
} 
} It seems that bash behaves correctly in all those cases.

Exactly -- which it would not if it followed Cracauer's advice; he's
wrong, and I'm not going to advocate that zsh adopt the behavior that
he describes.

} > Having said all that, I also looked at what it would take to implement
} > WCE as an option.  There are two complications here:
} > 
} > (1) There's a race condition as to whether the shell receives SIGINT
} > or SIGCHLD first.  With enough trials I can get it to happen either
} > way, though the vast majority of the time the SIGINT wins.
} 
} How does bash handle that?

It must handle it the way zsh 4.2.6 NO_TRAPS_ASYNC does, that is, by
blocking SIGINT (without changing the handler behavior) while waiting for
a child, then unblocking so the signal comes through after the waited-
for child exits.  This resolves the race in favor of always getting the
SIGCHLD first, and therefore the shell can always exit when it receives
the SIGINT (or never exit when it does not receive it).

} BTW, I wonder why typing Ctrl-g in emacs sends SIGINT to the parent
} under Mac OS X, but not under Linux.

Some ancestry of BSD vs. SYSV tty process group semantics, I expect.

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

On 2009-07-18 11:35:09 -0700, Bart Schaefer wrote:
> On Jul 18, 12:16pm, Vincent Lefevre wrote:
> } On 2009-07-17 22:29:36 -0700, Bart Schaefer wrote:
> } > The point I believe Cracauer is overlooking is that the CALLER of the
> } > shell script may have specified that SIGINT is to be ignored.  SIG_IGN
> } > is supposed to be inherited by child processes; but to implement WUE
> } > or WCE, the shell must itself either ignore or handle SIGINT, which
> } > may contradict what the caller has requested.
> } 
> } I don't think that the caller should enforce what its children should
> } do. Many programs will set their own SIGINT handler, even if the caller
> } chose to ignore this signal.
> 
> You've missed the point.  SIG_IGN is by definition been inherited from
> parent to child, and this is deliberately preserved across exec.  The
> parent doesn't *enforce* this -- the child is *allowed* to set up a
> handler of its own once it gets control -- but the mere fact that the
> execv'd program is a shell should not void the inherited signal settings.
> The shell might merely be an intermediary that's going to exec itself
> out of existence, and the programmer of the parent should be able to
> expect that the inherited signal settings make it through.

I didn't miss anything: the shell can still restore the previous
status before an exec.

> } Anyway, even though bash implements WCE,
> 
> Except that bash *DOESN'T* implement WCE, at least not the way that it
> is defined in Cracauer's web page.  See below.

According to my tests, bash *does* implement WCE. See below.

> } it ignores SIGINT if the caller chose to ignore it; also when executing
> } a process, it restores the trap to the default behavior, in case it was
> } ignored (that's another difference with zsh).
> 
> And that's a case where I think the zsh behavior is in fact preferable.

Is there any reason?

> } > Further I think Cracauer is very wrong here:
> } > 
> } >   Do nothing special when SIGINT appears while you wait for a child. You
> } >   don't even have to remember that one happened.
> } >   ...
> } >   Look at WIFSIGNALED(status) and WTERMSIG(status) to tell whether the
> } >   child says "I exited on SIGINT: in my opinion the user wants the
> } >   shellscript to be discontinued".
> } > 
> } > This is plain nonsense.
> } > 
> } > Not only does this potentially contradict a caller's explicit request
> } > to ignore SIGINT, but the script should not exit 130 every time any
> } > child exits 130.  It should exit only when SIGINT was received *by the
> } > script*.  "kill -INT ..." of the child should not cause the shell to
> } > behave as if it was interrupted.  Try it with bash.
> } 
> } It seems that bash behaves correctly in all those cases.
> 
> Exactly -- which it would not if it followed Cracauer's advice; he's
> wrong, and I'm not going to advocate that zsh adopt the behavior that
> he describes.

Well, after re-reading, I agree Cracauer's advice is wrong here.
But it is fixable and I think that the other parts of his page are
correct. At least, IUE/WUE/WCE are defined under the context "There
are several ways to handle abortion of shell scripts when SIGINT is
received while a foreground child runs", so it is clear that the shell
*must* receive a SIGINT (this is also said near the beginning of the
page).

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

In fact, it seems that zsh implements some form of WCE in
interactive scripts, whereas bash implements it in any case.
Consider the following shell script:

sigint() { echo "SIGINT (sh)"; }
for i in 0 1 2 3
do
  trap - INT
  [ "$i" -eq 1 ] && trap sigint INT
  perl -e $i' or $SIG{"INT"} = sub { print "SIGINT (perl)\n" }; print "'$i' - $$\n"; sleep 30; $SIG{"INT"} = "DEFAULT"; kill "INT", $$'
done

Under Linux, whether this script is run in an interactive shell
or not (zsh or another one), I always get something like:

0 - 7387
^CSIGINT (perl)

and the script is immediately interrupted. Now, remove the

  $SIG{"INT"} = "DEFAULT"; kill "INT", $$

so that in the case i = 0, the Perl script is no longer killed
by SIGINT. Now, depending on the shell and how the script is
started, the script is interrupted either for i = 0 or i = 2
(the latter more of less corresponds to WCE, i.e. the sh script
is interrupted only if the Perl script is killed by SIGINT).

I considered the following two cases:
  1. "<shell> script.sh"
  2. ". script.sh" from the interactive shell

and I get the following results with:

  bash 3.2-5 (whether executed as bash or sh)
  dash 0.5.5.1-2
  ksh 93t+-2 (ksh93)
  pdksh 5.2.14-23
  posh 0.6.18
  zsh 4.3.10-3

under Debian/unstable.

* bash[1], dash[2], ksh93[1], pdksh[2], zsh[2]

0 - 7062
^CSIGINT (perl)
1 - 7063
^CSIGINT (sh)
2 - 7066
^C

* bash[2]

0 - 7241
^CSIGINT (perl)
1 - 7242
^C
2 - 7243
^C

* dash[1], ksh93[2], pdksh[1], posh[1], posh[2], zsh[1]

0 - 7387
^CSIGINT (perl)

Note that ksh93 does the opposite to pdksh and zsh!

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

On Jul 19, 11:51am, Vincent Lefevre wrote:
}
} In fact, it seems that zsh implements some form of WCE in
} interactive scripts, whereas bash implements it in any case.

I think you'll find that if you "setopt restricted" you'll get yet
another set of behaviors.

This really all comes down to what I said in zsh-workers/27159.

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

On Jul 18, 11:16am, Bart Schaefer wrote:
}
} This was changed in this patch ...
} 
}         * 23067: Doc/Zsh/builtins.yo, Src/jobs.c, Src/signals.c:
}         queue traps but handle signals when waiting for jobs or processes,
}         unless TRAPSASYNC is set or the wait builtin is in use, so as
}         to handle untrapped signals in a timely fashion; document that
}         negative or zero process IDs after kill may be handled specially
}         by the OS.

The following small change appears to restore the old behavior for SIGINT.
The question is whether this should be applied to other signals that are
known to cause process termination, such as SIGQUIT.

I'm also not sure whether it's really necessary to test sigtrapped[], or
whether I've got the sense of that test right.

Index: Src/signals.c
===================================================================
diff -c -r1.22 signals.c
--- Src/signals.c	28 Feb 2009 07:13:37 -0000	1.22
+++ Src/signals.c	19 Jul 2009 17:52:28 -0000
@@ -353,6 +353,8 @@
 #endif /* BROKEN_POSIX_SIGSUSPEND */
 
     sigemptyset(&set);
+    if (!(isset(TRAPSASYNC) || (sigtrapped[SIGINT] & ~ZSIG_IGNORED)))
+	sigaddset(&set, SIGINT);
 #ifdef BROKEN_POSIX_SIGSUSPEND
     sigprocmask(SIG_SETMASK, &set, &oset);
     pause();


-- 

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

On Jul 19,  1:09am, Vincent Lefevre wrote:
} Subject: Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in em
}
} > } [bash] ignores SIGINT if the caller chose to ignore it; also when
} > } executing a process, it restores the trap to the default behavior,
} > } in case it was ignored (that's another difference with zsh).
} > 
} > And that's a case where I think the zsh behavior is in fact preferable.
} 
} Is there any reason?

If bash really "restores the trap to the default behavior, in case it was
ignored" then I think bash is wrong because "the default behavior" is to
exit on the signal.  It should instead continue to ignore it, as requested
by the caller, as would have happened if the caller directly exec'd the
process instead of passing through bash along the way.

The patch I just sent for Src/signals.c sidesteps the issue in the way I
previously mentioned: without changing the eventual handling, it holds
SIGINT undelivered until some other signal interrupts signal_suspend().

Hmm, looking at the #ifdef flow and considering that this affects MacOS,
maybe the change in my patch needs to be in the BSD_SIGNALS section,
either instead or as well.

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

On Jul 19, 11:03am, Bart Schaefer wrote:
}
} The following small change appears to restore the old behavior for SIGINT.

Unfortunately it does so too well.  It makes "wait" uninterruptible again,
which was the problem in the original thread at zsh-workers/23053.

The only way I can see to fix this is to re-introduce the wait_cmd second
argument to signal_suspend().  PWS?  Any thoughts?

} Hmm, looking at the #ifdef flow and considering that this affects MacOS,
} maybe the change in my patch needs to be in the BSD_SIGNALS section,
} either instead or as well.

I checked config.h on my iMac at work and it does not have BSD_SIGNALS,
so this isn't necessary for that reason, but it still might be a good
idea to have it in both branches.

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Bart Schaefer]

On Jul 19, 12:15pm, Bart Schaefer wrote:
}
} The only way I can see to fix this is to re-introduce the wait_cmd second
} argument to signal_suspend().  PWS?  Any thoughts?

Here's a patch in case there's agreement this is the right approach.
Apply *instead of* zsh-workers/27165.  Line numbers may be wonky in
the jobs.c patch.

Index: Src/jobs.c
===================================================================
--- Src/jobs.c	17 Apr 2009 18:57:22 -0000	1.27
+++ Src/jobs.c	19 Jul 2009 19:19:10 -0000
@@ -1178,7 +1191,7 @@
 	    kill(pid, SIGCONT);
 
 	last_signal = -1;
-	signal_suspend(SIGCHLD);
+	signal_suspend(SIGCHLD, wait_cmd);
 	if (last_signal != SIGCHLD && wait_cmd && last_signal >= 0 &&
 	    (sigtrapped[last_signal] & ZSIG_TRAPPED)) {
 	    /* wait command interrupted, but no error: return */
@@ -1217,7 +1230,7 @@
 	while (!errflag && jn->stat &&
 	       !(jn->stat & STAT_DONE) &&
 	       !(interact && (jn->stat & STAT_STOPPED))) {
-	    signal_suspend(SIGCHLD);
+	    signal_suspend(SIGCHLD, wait_cmd);
 	    if (last_signal != SIGCHLD && wait_cmd && last_signal >= 0 &&
 		(sigtrapped[last_signal] & ZSIG_TRAPPED))
 	    {

Index: Src/signals.c
===================================================================
--- Src/signals.c	28 Feb 2009 07:13:37 -0000	1.22
+++ Src/signals.c	19 Jul 2009 20:07:04 -0000
@@ -342,29 +342,32 @@
 
 /**/
 int
-signal_suspend(UNUSED(int sig))
+signal_suspend(UNUSED(int sig), int wait_cmd)
 {
     int ret;
- 
-#ifdef POSIX_SIGNALS
+
+#if defined(POSIX_SIGNALS) || defined(BSD_SIGNALS)
     sigset_t set;
-#ifdef BROKEN_POSIX_SIGSUSPEND
+# if defined(POSIX_SIGNALS) && defined(BROKEN_POSIX_SIGSUSPEND)
     sigset_t oset;
-#endif /* BROKEN_POSIX_SIGSUSPEND */
+# endif
 
     sigemptyset(&set);
-#ifdef BROKEN_POSIX_SIGSUSPEND
+    if (!(wait_cmd || isset(TRAPSASYNC) ||
+	  (sigtrapped[SIGINT] & ~ZSIG_IGNORED)))
+	sigaddset(&set, SIGINT);
+#endif /* POSIX_SIGNALS || BSD_SIGNALS */
+
+#ifdef POSIX_SIGNALS
+# ifdef BROKEN_POSIX_SIGSUSPEND
     sigprocmask(SIG_SETMASK, &set, &oset);
     pause();
     sigprocmask(SIG_SETMASK, &oset, NULL);
-#else /* not BROKEN_POSIX_SIGSUSPEND */
+# else /* not BROKEN_POSIX_SIGSUSPEND */
     ret = sigsuspend(&set);
-#endif /* BROKEN_POSIX_SIGSUSPEND */
+# endif /* BROKEN_POSIX_SIGSUSPEND */
 #else /* not POSIX_SIGNALS */
 # ifdef BSD_SIGNALS
-    sigset_t set;
-
-    sigemptyset(&set);
     ret = sigpause(set);
 # else
 #  ifdef SYSV_SIGNALS

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Peter Stephenson]

Bart Schaefer wrote:
> On Jul 19, 12:15pm, Bart Schaefer wrote:
> }
> } The only way I can see to fix this is to re-introduce the wait_cmd second
> } argument to signal_suspend().  PWS?  Any thoughts?
> 
> Here's a patch in case there's agreement this is the right approach.

If it fixes both problems, it'll certainly do for now, until we work out
what it breaks.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

On 2009-07-19 09:32:31 -0700, Bart Schaefer wrote:
> On Jul 19, 11:51am, Vincent Lefevre wrote:
> }
> } In fact, it seems that zsh implements some form of WCE in
> } interactive scripts, whereas bash implements it in any case.
> 
> I think you'll find that if you "setopt restricted" you'll get yet
> another set of behaviors.

I don't find any difference.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

On 2009-07-19 11:31:47 -0700, Bart Schaefer wrote:
> On Jul 19,  1:09am, Vincent Lefevre wrote:
> } Subject: Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in em
> }
> } > } [bash] ignores SIGINT if the caller chose to ignore it; also when
> } > } executing a process, it restores the trap to the default behavior,
> } > } in case it was ignored (that's another difference with zsh).
> } > 
> } > And that's a case where I think the zsh behavior is in fact preferable.
> } 
> } Is there any reason?
> 
> If bash really "restores the trap to the default behavior, in case it was
> ignored" then I think bash is wrong because "the default behavior" is to
> exit on the signal.

Sorry, I was not enough clear. I meant that bash restores the trap
to the behavior at the time that bash was called. But it seems to
be something different, something like if SIGINT is ignored when
bash is executed, a "trap command INT" has no effect unless both
bash and its child receive a SIGINT. Now, this behavior is a bit
strange, maybe a bug.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Eric Blake]

Vincent Lefevre <vincent <at> vinc17.org> writes:

> 
> Sorry, I was not enough clear. I meant that bash restores the trap
> to the behavior at the time that bash was called. But it seems to
> be something different, something like if SIGINT is ignored when
> bash is executed, a "trap command INT" has no effect unless both
> bash and its child receive a SIGINT. Now, this behavior is a bit
> strange, maybe a bug.

POSIX requires that if a shell starts life with a signal ignored, that the shell
should silently reject attempts to alter the fact that the signal is ignored
(either back to a default, or to a user-specified handler).  Which is very
annoying if your shell ever gets started with SIGPIPE ignored, but that's life.
 In other words, if you start bash with SIGINT ignored, you can't undo that from
within the bash shell.

-- 
Eric Blake

Re: zsh 4.3.10 terminates with SIGINT when one types Ctrl-G in emacs under Mac OS X

[Vincent Lefevre]

On 2009-07-22 02:58:37 +0000, Eric Blake wrote:
> POSIX requires that if a shell starts life with a signal ignored,
> that the shell should silently reject attempts to alter the fact
> that the signal is ignored (either back to a default, or to a
> user-specified handler). Which is very annoying if your shell ever
> gets started with SIGPIPE ignored, but that's life. In other words,
> if you start bash with SIGINT ignored, you can't undo that from
> within the bash shell.

OK, thanks for the explanations. So, the following script shows a bug
in zsh (typing Ctrl-C immediately interrupts the sleep and the script
and SIGINT is output):

#!/bin/sh

trap ''  INT
zsh -fc "emulate sh; trap 'echo SIGINT; exit' INT; sleep 6"

I wonder whether zsh should behave in the same way without
"emulate sh".

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

FEATURES description wrong line

[Maddi Kopfermann]

Hi Zsh-Workers

I read in "FEATURES" and found some wrong line:

"ls *(om[2])  matches the two most recently modified files"

I am very thankful that this FEATURE isn't there ;)
Would confuse me very much :)


Maddi

Re: FEATURES description wrong line

[Bart Schaefer]

On Nov 8, 12:49pm, Maddi Kopfermann wrote:
}
} I read in "FEATURES" and found some wrong line:
} 
} "ls *(om[2])  matches the two most recently modified files"
} 
} I am very thankful that this FEATURE isn't there ;)
} Would confuse me very much :)

Good catch, that matches the second-most-recently modified file.  To get
the two most recently modified, you need *(om[1,2]) ... but that's not
exemplary enough if you ask me, because it makes it appear you have to
list all the indices.

Index: FEATURES
===================================================================
RCS file: /extra/cvsroot/zsh/zsh-4.0/FEATURES,v
retrieving revision 1.2
diff -c -r1.2 FEATURES
--- FEATURES	24 Sep 2005 17:48:33 -0000	1.2
+++ FEATURES	8 Nov 2009 17:07:22 -0000
@@ -28,7 +28,7 @@
 qualifiers in parentheses after globbing expression:
   ls *(*@)  matches executable files/directories or symlinks
   ls *(L0f.go-w.)  matches all zero-length files not group or world writable
-  ls *(om[2])  matches the two most recently modified files
+  ls *(om[1,3])  matches the three most recently modified files
 null command shorthands:
   "< file" is same as "more <file"
   "> file" is same as "cat >file"

Re: FEATURES description wrong line

[Peter Stephenson]

Maddi Kopfermann wrote:
> I read in "FEATURES" and found some wrong line:
> 
> "ls *(om[2])  matches the two most recently modified files"

Yes, that's not very helpful.  I'll change it to...

Index: FEATURES
===================================================================
RCS file: /cvsroot/zsh/zsh/FEATURES,v
retrieving revision 1.2
diff -u -r1.2 FEATURES
--- FEATURES	1 Aug 2005 09:54:56 -0000	1.2
+++ FEATURES	8 Nov 2009 18:43:31 -0000
@@ -28,7 +28,7 @@
 qualifiers in parentheses after globbing expression:
   ls *(*@)  matches executable files/directories or symlinks
   ls *(L0f.go-w.)  matches all zero-length files not group or world writable
-  ls *(om[2])  matches the two most recently modified files
+  ls *(om[2])  matches the second most recently modified file
 null command shorthands:
   "< file" is same as "more <file"
   "> file" is same as "cat >file"

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: FEATURES description wrong line

[Stephane Chazelas]

2009-11-08 09:08:15 -0800, Bart Schaefer:
> On Nov 8, 12:49pm, Maddi Kopfermann wrote:
> }
> } I read in "FEATURES" and found some wrong line:
> } 
> } "ls *(om[2])  matches the two most recently modified files"
> } 
> } I am very thankful that this FEATURE isn't there ;)
> } Would confuse me very much :)
> 
> Good catch, that matches the second-most-recently modified file.  To get
> the two most recently modified, you need *(om[1,2]) ... but that's not
> exemplary enough if you ask me, because it makes it appear you have to
> list all the indices.
[...]

By the way, what about having the empty string (or any sequence
of white space) resolve to 0 in numeric context everywhere?

We already have:

$ echo $(())
0
$ (()) || echo null
null
$ [ 0 -eq "" ] && echo yes
yes
$ a=; echo $(( a == 0 ))
1

AT&T ksh and pdksh have:

$ a=1 pdksh -c 'echo ${a[]}'
1

zsh could have the same and also *(om[,2]) would be another way
to get the 2 most recent files, what do you think?

-- 
Stephane

Re: FEATURES description wrong line

[Bart Schaefer]

On Nov 8,  6:44pm, Peter Stephenson wrote:
}
} > "ls *(om[2])  matches the two most recently modified files"
} 
} Yes, that's not very helpful.  I'll change it to...
} 
} +  ls *(om[2])  matches the second most recently modified file

So we've got two competing patches for this and neither has been checked
in yet.  Shall I go ahead and put *both* of the examples in FEATURES?

Re: FEATURES description wrong line

[Peter Stephenson]

Bart Schaefer wrote:
> On Nov 8,  6:44pm, Peter Stephenson wrote:
> }
> } > "ls *(om[2])  matches the two most recently modified files"
> } 
> } Yes, that's not very helpful.  I'll change it to...
> } 
> } +  ls *(om[2])  matches the second most recently modified file
> 
> So we've got two competing patches for this and neither has been checked
> in yet.  Shall I go ahead and put *both* of the examples in FEATURES?

Sounds good.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

${(q)...} for newline

[Peter Stephenson]

Param-style simple (i.e. backslash) quoting of a newline appends a real
newline in double quotes.  It seems to me it would occasion far fewer
surprises if this never output a literal newline, so this uses $'\n'.  We
already bargain for up to 7 output characters per input character.  The
first two hunks are just a tidy up (written before I realised I didn't
need to do any more counting).

Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.242
diff -p -u -r1.242 utils.c
--- Src/utils.c	27 Mar 2010 19:04:36 -0000	1.242
+++ Src/utils.c	23 May 2010 19:54:21 -0000
@@ -4585,11 +4585,12 @@ quotestring(const char *s, char **e, int
      * quotesub = 2:  mechanism active, added opening "'"; need
      *                closing "'".
      */
-    int quotesub = 0;
+    int quotesub = 0, slen;
     char *quotestart;
     convchar_t cc;
     const char *uend;
 
+    slen = strlen(s);
     switch (instring)
     {
     case QT_BACKSLASH:
@@ -4598,21 +4599,22 @@ quotestring(const char *s, char **e, int
 	 * Keep memory usage within limits by allocating temporary
 	 * storage and using heap for correct size at end.
 	 */
-	alloclen = strlen(s) * 7 + 1;
+	alloclen = slen * 7 + 1;
 	break;
 
     case QT_SINGLE_OPTIONAL:
 	/*
 	 * Here, we may need to add single quotes.
 	 */
-	alloclen = strlen(s) * 4 + 3;
+	alloclen = slen * 4 + 3;
 	quotesub = 1;
 	break;
 
     default:
-	alloclen = strlen(s) * 4 + 1;
+	alloclen = slen * 4 + 1;
 	break;
     }
+
     tt = quotestart = v = buf = zshcalloc(alloclen);
 
     DPUTS(instring < QT_BACKSLASH || instring == QT_BACKTICK ||
@@ -4771,15 +4773,19 @@ quotestring(const char *s, char **e, int
 		    continue;
 		} else if (*u == '\n' ||
 			   (instring == QT_SINGLE && *u == '\'')) {
-		    if (unset(RCQUOTES)) {
+		    if (*u == '\n') {
+			*v++ = '$';
+			*v++ = '\'';
+			*v++ = '\\';
+			*v++ = 'n';
+			*v++ = '\'';
+		    } else if (unset(RCQUOTES)) {
 			*v++ = '\'';
 			if (*u == '\'')
 			    *v++ = '\\';
 			*v++ = *u;
 			*v++ = '\'';
-		    } else if (*u == '\n')
-			*v++ = '"', *v++ = '\n', *v++ = '"';
-		    else
+		    } else
 			*v++ = '\'', *v++ = '\'';
 		    u++;
 		    continue;

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: ${(q)...} for newline

[Peter Stephenson]

Peter Stephenson wrote:
> Param-style simple (i.e. backslash) quoting of a newline appends a real
> newline in double quotes.  It seems to me it would occasion far fewer
> surprises if this never output a literal newline, so this uses $'\n'.

Also, would I be right in thinking this is an accident waiting to
happen?

% foo=(one '' three)
% print ${(q)foo}
one three

Certainly

% printf "%q\n" ''


which uses the same quoting internally is incompatible with what bash
does (it outputs '').  It looks like in both cases (and all explicit
uses of backslash-quoting --- completion is different) empty strings
might be better turned into an explicit ''.

Is there a case where this breaks something?  I can't see one unless
you're explicitly relying on a particular result from quoting, which
seems an unhealthy thing to do.  However, there may be special cases I
haven't thought of (there usually are with quoting).

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: ${(q)...} for newline

[Stephane Chazelas]

2010-05-23 21:45:39 +0100, Peter Stephenson:
> Peter Stephenson wrote:
> > Param-style simple (i.e. backslash) quoting of a newline appends a real
> > newline in double quotes.  It seems to me it would occasion far fewer
> > surprises if this never output a literal newline, so this uses $'\n'.
> 
> Also, would I be right in thinking this is an accident waiting to
> happen?
> 
> % foo=(one '' three)
> % print ${(q)foo}
> one three
> 
> Certainly
> 
> % printf "%q\n" ''
> 
> 
> which uses the same quoting internally is incompatible with what bash
> does (it outputs '').  It looks like in both cases (and all explicit
> uses of backslash-quoting --- completion is different) empty strings
> might be better turned into an explicit ''.
[...]

Not in the first case though, as array expansion removes the
empty elements. But in "${(@q)foo}" yes maybe, at least to be
consistent with bash and ksh93 printf %q.

$ zsh -c 'a=(a "" b); printf "<%s>\n" ${(q)a}'
<a>
<b>
$ zsh -c 'a=(a "" b); printf "<%s>\n" "${(q@)a}"'
<a>
<>
<b>

-- 
Stephane

Re: ${(q)...} for newline

[Bart Schaefer]

On Sun, May 23, 2010 at 12:58 PM, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
> Param-style simple (i.e. backslash) quoting of a newline appends a real
> newline in double quotes.  It seems to me it would occasion far fewer
> surprises if this never output a literal newline, so this uses $'\n'.

I let this stew for a couple of days before commenting ... and after
some initial vague unease about this I guess I'm over it, since $'...'
is already used for unprintable or invalid characters.

However, I suggest that (q-) continue to use the literal quoted newline.

Re: ${(q)...} for newline

[Peter Stephenson]

On Tue, 25 May 2010 14:25:13 +0100
Stephane Chazelas <stephane_chazelas@yahoo.fr> wrote:
> 2010-05-23 21:45:39 +0100, Peter Stephenson:
> > Also, would I be right in thinking this is an accident waiting to
> > happen?
> > 
> > % foo=(one '' three)
> > % print ${(q)foo}
> > one three
>
> Not in the first case though, as array expansion removes the
> empty elements.

I don't think that argument really works... consider:

% foo=(one '' three)
% print ${(j.X.)foo}
oneXXthree

In other words, flags are applied before empty elements are
stripped.  The same goes for most transformations on array elements,
whether zsh-specific or otherwise; I don't see why (q) should be
different.

However, we don't actually say in the (already long) manual entry giving
the rules for variable expansion at what part either quoting or empty
element removal takes place.  The latter appears to be the last thing of all,
i.e. at the point where we decide to insert this into the list of arguments
(or otherwise).  I should probably concoct something.


Bart wrote
> However, I suggest that (q-) continue to use the literal quoted newline.

Right, it's really there as minimal quoting for human readability.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: ${(q)...} for newline

[Peter Stephenson]

On Tue, 25 May 2010 15:48:24 +0100
Peter Stephenson <Peter.Stephenson@csr.com> wrote:
> However, we don't actually say in the (already long) manual entry giving
> the rules for variable expansion at what part either quoting or empty
> element removal takes place.  The latter appears to be the last thing of all,
> i.e. at the point where we decide to insert this into the list of arguments
> (or otherwise).  I should probably concoct something.

Index: Doc/Zsh/expn.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/expn.yo,v
retrieving revision 1.113
diff -p -u -r1.113 expn.yo
--- Doc/Zsh/expn.yo	12 May 2010 10:07:01 -0000	1.113
+++ Doc/Zsh/expn.yo	26 May 2010 19:51:21 -0000
@@ -1172,7 +1172,15 @@ Any modifiers, as specified by a trailin
 noderef(Modifiers) in noderef(History Expansion)), are applied to the words
 of the value at this level.
 )
-item(tt(8.) em(Forced Joining))(
+item(tt(8.) em(Character evaluation))(
+Any tt((#)) flag is applied, evaluating the result so far numerically
+as a character.
+)
+item(tt(9.) em(Length))(
+Any initial tt(#) modifier, i.e. in the form tt(${#)var(var)tt(}), is
+used to evaluate the length of the expression so far.
+)
+item(tt(10.) em(Forced Joining))(
 If the `tt((j))' flag is present, or no `tt((j))' flag is present but
 the string is to be split as given by rules tt(8.) or tt(9.), and joining
 did not take place at step tt(4.), any words in the value are joined
@@ -1180,36 +1188,51 @@ together using the given string or the f
 Note that the `tt((F))' flag implicitly supplies a string for joining in this
 manner.
 )
-item(tt(9.) em(Forced Splitting))(
+item(tt(11.) em(Case modification))(
+Any case modification from one of the flags tt((L)), tt((U)) or tt((C))
+is applied.
+)
+item(tt(12.) em(Prompt evaluation))(
+Any prompt-style formatting from the tt((%)) family of flags is applied.
+)
+item(tt(13.) em(Quote application))(
+Any quoting or unquoting using tt((q)) and tt((Q)) and related flags
+is applied.
+)
+item(tt(14.) em(Visibility enhancment))(
+Any modifications to make characters visible using the tt((V)) flag
+are applied.
+)
+item(tt(15.) em(Forced Splitting))(
 If one of the `tt((s))', `tt((f))' or `tt((z))' flags are present, or the `tt(=)'
 specifier was present (e.g. tt(${=)var(var)tt(})), the word is split on
 occurrences of the specified string, or (for tt(=) with neither of the two
 flags present) any of the characters in tt($IFS).
 )
-item(tt(10.) em(Shell Word Splitting))(
+item(tt(16.) em(Shell Word Splitting))(
 If no `tt((s))', `tt((f))' or `tt(=)' was given, but the word is not
 quoted and the option tt(SH_WORD_SPLIT) is set, the word is split on
 occurrences of any of the characters in tt($IFS).  Note this step, too,
 takes place at all levels of a nested substitution.
 )
-item(tt(11.) em(Uniqueness))(
+item(tt(17.) em(Uniqueness))(
 If the result is an array and the `tt((u))' flag was present, duplicate
 elements are removed from the array.
 )
-item(tt(12.) em(Ordering))(
+item(tt(18.) em(Ordering))(
 If the result is still an array and one of the `tt((o))' or `tt((O))' flags
 was present, the array is reordered.
 )
-item(tt(13.) em(Re-Evaluation))(
+item(tt(19.) em(Re-Evaluation))(
 Any `tt((e))' flag is applied to the value, forcing it to be re-examined
 for new parameter substitutions, but also for command and arithmetic
 substitutions.
 )
-item(tt(14.) em(Padding))(
+item(tt(20.) em(Padding))(
 Any padding of the value by the `tt(LPAR()l.)var(fill)tt(.RPAR())' or
 `tt(LPAR()r.)var(fill)tt(.RPAR())' flags is applied.
 )
-item(tt(15.) em(Semantic Joining))(
+item(tt(21.) em(Semantic Joining))(
 In contexts where expansion semantics requires a single word to
 result, all words are rejoined with the first character of tt(IFS)
 between.  So in `tt(${LPAR()P)tt(RPAR()${LPAR()f)tt(RPAR()lines}})'
@@ -1218,6 +1241,15 @@ joined again before the tt(P) flag can b
 
 If a single word is not required, this rule is skipped.
 )
+item(tt(22.) em(Empty argument removal))(
+If the substitution does not appear in double quotes, any resulting
+zero-length argument, whether from a scalar or an element of an array,
+is elided from the list of arguments inserted into the command line.
+
+Strictly speaking, the removal happens later as the same happens with
+other forms of substitution; the point to note here is simply that
+it occurs after any of the above parameter operations.
+)
 enditem()
 
 subsect(Examples)

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: environment variables

[Peter Stephenson]

On Tue, 31 Aug 2010 17:54:25 +0200
Joke de Buhr <joke@seiken.de> wrote:
> ## 4: HELLO not exported to shell function, different from 2 and 3
> $ call() { env | grep '^HELLO' }
> $ HELLO=world
> $ HELLO=$HELLO call
>     HELLO=      ## <-- empty

(Moved to zsh-workers immediately, this time.)

That's a bug.  When HELLO needs to be restored explicitly because the
code where it's set to something else is running in the main shell, we
remove the parameter from the table to be restored later.  The
assignment than can't see the value it's temporarily modifying.  You
don't need to search the environment for this; zsh consistently screws
up the parameter within the shell, too.

I think we can just copy the parameter instead of removing it, so long
as we copy it properly.

valgrind says this doesn't cause any leaks.

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.182
diff -p -u -r1.182 exec.c
--- Src/exec.c	22 Aug 2010 20:08:57 -0000	1.182
+++ Src/exec.c	31 Aug 2010 19:11:41 -0000
@@ -3313,13 +3313,17 @@ save_params(Estate state, Wordcode pc, L
     while (wc_code(ac = *pc) == WC_ASSIGN) {
 	s = ecrawstr(state->prog, pc + 1, NULL);
 	if ((pm = (Param) paramtab->getnode(paramtab, s))) {
+	    Param tpm;
 	    if (pm->env)
 		delenv(pm);
 	    if (!(pm->node.flags & PM_SPECIAL)) {
-		paramtab->removenode(paramtab, s);
+		tpm = (Param) zshcalloc(sizeof *tpm);
+		tpm->node.nam = ztrdup(pm->node.nam);
+		copyparam(tpm, pm, 0);
+		pm = tpm;
 	    } else if (!(pm->node.flags & PM_READONLY) &&
 		       (unset(RESTRICTED) || !(pm->node.flags & PM_RESTRICTED))) {
-		Param tpm = (Param) hcalloc(sizeof *tpm);
+		tpm = (Param) hcalloc(sizeof *tpm);
 		tpm->node.nam = pm->node.nam;
 		copyparam(tpm, pm, 1);
 		pm = tpm;
@@ -3383,8 +3387,9 @@ restore_params(LinkList restorelist, Lin
 		    break;
 		}
 		pm = tpm;
-	    } else
+	    } else {
 		paramtab->addnode(paramtab, pm->node.nam, pm);
+	    }
 	    if ((pm->node.flags & PM_EXPORTED) && ((s = getsparam(pm->node.nam))))
 		addenv(pm, s);
 	}
Index: Src/params.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/params.c,v
retrieving revision 1.161
diff -p -u -r1.161 params.c
--- Src/params.c	3 Jun 2010 13:38:17 -0000	1.161
+++ Src/params.c	31 Aug 2010 19:11:41 -0000
@@ -927,11 +927,17 @@ createspecialhash(char *name, GetNodeFun
 }
 
 
-/* Copy a parameter */
+/*
+ * Copy a parameter
+ *
+ * If fakecopy is set, we are just saving the details of a special
+ * parameter.  Otherwise, the result will be used as a real parameter
+ * and we need to do more work.
+ */
 
 /**/
 void
-copyparam(Param tpm, Param pm, int toplevel)
+copyparam(Param tpm, Param pm, int fakecopy)
 {
     /*
      * Note that tpm, into which we're copying, may not be in permanent
@@ -942,7 +948,8 @@ copyparam(Param tpm, Param pm, int tople
     tpm->node.flags = pm->node.flags;
     tpm->base = pm->base;
     tpm->width = pm->width;
-    if (!toplevel)
+    tpm->level = pm->level;
+    if (!fakecopy)
 	tpm->node.flags &= ~PM_SPECIAL;
     switch (PM_TYPE(pm->node.flags)) {
     case PM_SCALAR:
@@ -963,13 +970,15 @@ copyparam(Param tpm, Param pm, int tople
 	break;
     }
     /*
-     * If called from inside an associative array, that array is later going
-     * to be passed as a real parameter, so we need the gets and sets
-     * functions to be useful.  However, the saved associated array is
-     * not itself special, so we just use the standard ones.
-     * This is also why we switch off PM_SPECIAL.
+     * If the value is going to be passed as a real parameter (e.g. this is
+     * called from inside an associative array), we need the gets and sets
+     * functions to be useful.
+     *
+     * In this case we assume the the saved parameter is not itself special,
+     * so we just use the standard functions.  This is also why we switch off
+     * PM_SPECIAL.
      */
-    if (!toplevel)
+    if (!fakecopy)
 	assigngetset(tpm);
 }
 
Index: Test/A06assign.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A06assign.ztst,v
retrieving revision 1.5
diff -p -u -r1.5 A06assign.ztst
--- Test/A06assign.ztst	23 Sep 2006 06:55:29 -0000	1.5
+++ Test/A06assign.ztst	31 Aug 2010 19:11:41 -0000
@@ -277,3 +277,18 @@
 >
 >
 >
+
+  call() { print $HELLO; }
+  export HELLO=world
+  call
+  HELLO=universe call
+  call
+  HELLO=${HELLO}liness call
+  call
+  unset HELLO
+0:save and restore when using original value in temporary
+>world
+>universe
+>world
+>worldliness
+>world


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: environment variables

[Bart Schaefer]

On Aug 31,  8:18pm, Peter Stephenson wrote:
} Subject: Re: environment variables
}
} On Tue, 31 Aug 2010 17:54:25 +0200
} Joke de Buhr <joke@seiken.de> wrote:
} > ## 4: HELLO not exported to shell function, different from 2 and 3
} > $ call() { env | grep '^HELLO' }
} > $ HELLO=world
} > $ HELLO=$HELLO call
} >     HELLO=      ## <-- empty
} 
} That's a bug.

Have you been following the austin-group discussion about the behavior
of assignments prefixing builtin vs. external commands?

} I think we can just copy the parameter instead of removing it, so long
} as we copy it properly.

What did the name "toplevel" signify before?  Obviously it's semantically
the same as "fakecopy", but it must have meant something to someone ...

Re: environment variables

[Peter Stephenson]

Bart Schaefer wrote:
> Have you been following the austin-group discussion about the behavior
> of assignments prefixing builtin vs. external commands?

Only very vaguely.  It was about some fairly abstruse behaviour that's
not yet standardised, so I didn't particularly sit up and take notice,
but I've forgotten the details.
 
> } I think we can just copy the parameter instead of removing it, so long
> } as we copy it properly.
> 
> What did the name "toplevel" signify before?  Obviously it's semantically
> the same as "fakecopy", but it must have meant something to someone ...

It was named because of the use it was put to rather than the function
it had (poor interface design on my part).  It refers to the top level
of parameter handling when paramtab is the "real" parameter table, as
opposed to when it's the parameter table contained within an associative
array (when we're somewhere within the bowels of parameter code).  In
the former case it happened we never needed to do a real copy because we
only used it for partially saving special parameters (but now we use it
for normal parameters at the same "level", which wasn't even a
particularly descriptive word).

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: killing suspended jobs makes zsh hang after 47d1215

[Bart Schaefer]

On Jun 12,  8:40am, Pan Tsu wrote:
} Subject: killing suspended jobs makes zsh hang after 47d1215
}
}   % man man
}   ^Z
}   zsh: suspended  man man
}   % kill %
}   % %
}   [1]  + continued  man man
}   load: 0.04  cmd: zsh 47317 [pause] 222.05r 0.01u 0.00s 0% 2612k

I am able to reproduce this.

kill -CLD wakes up the shell, showing this:

torch% jobs
[1]  + running    man man
torch% 

The bug appears to have been introduced here:

----------------------------
revision 1.80
date: 2011/04/01 11:02:16;  author: pws;  state: Exp;  lines: +18 -1
Stef van Vlierberghe: 28965 (as posted in 28967):
findproc() should not return processes not marked as SP_RUNNING
----------------------------

Evidently there are in fact cases where we need to find jobs that are
not running.

-- 

Re: killing suspended jobs makes zsh hang after 47d1215

[Bart Schaefer]

On Jun 12,  7:22am, Bart Schaefer wrote:
}
} ----------------------------
} revision 1.80
} date: 2011/04/01 11:02:16;  author: pws;  state: Exp;  lines: +18 -1
} Stef van Vlierberghe: 28965 (as posted in 28967):
} findproc() should not return processes not marked as SP_RUNNING
} ----------------------------
} 
} Evidently there are in fact cases where we need to find jobs that are
} not running.

The specific bug is fixed by the patch below, but I wonder if there are
other job status that need to be tested here as well.

Index: jobs.c
===================================================================
RCS file: /extra/cvsroot/zsh/zsh-4.0/Src/jobs.c,v
retrieving revision 1.30
diff -c -r1.30 jobs.c
--- jobs.c	1 Jun 2011 06:40:00 -0000	1.30
+++ jobs.c	12 Jun 2011 14:55:25 -0000
@@ -189,7 +189,8 @@
 	     * the termination of the process which pid we were supposed
 	     * to return in a different job.
 	     */
-	    if (pn->pid == pid && pn->status == SP_RUNNING) {
+	    if (pn->pid == pid && (pn->status == SP_RUNNING ||
+				   WIFSTOPPED(pn->status))) {
 		*pptr = pn;
 		*jptr = jobtab + i;
 		return 1;


Here are backtraces showing how we reach findproc() with the patch above
NOT YET applied [line number in findproc is inconsequential, I stepped
through to make sure of the return value].

When the job gets stopped:

(gdb) where
#0  findproc (pid=26597, jptr=0xbff08180, pptr=0xbff0817c, aux=0)
    at ../../zsh-4.0/Src/jobs.c:193
#1  0x080b175b in wait_for_processes () at ../../zsh-4.0/Src/signals.c:493
#2  0x080b19aa in zhandler (sig=17) at ../../zsh-4.0/Src/signals.c:584
#3  <signal handler called>
#4  0x007067a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#5  0x00747e6c in sigsuspend () from /lib/tls/libc.so.6
#6  0x080b15f6 in signal_suspend (sig=17, wait_cmd=0)
    at ../../zsh-4.0/Src/signals.c:373
#7  0x0808160e in zwaitjob (job=1, wait_cmd=0)
    at ../../zsh-4.0/Src/jobs.c:1317
#8  0x080817e5 in waitjobs () at ../../zsh-4.0/Src/jobs.c:1362
#9  0x08061ee0 in execpline (state=0xbff08cf0, slcode=4098, how=18, last1=0)
    at ../../zsh-4.0/Src/exec.c:1500
#10 0x080613e8 in execlist (state=0xbff08cf0, dont_change_job=0, exiting=0)
    at ../../zsh-4.0/Src/exec.c:1207
#11 0x08060e7e in execode (p=0xb7d834b8, dont_change_job=0, exiting=0, 
    context=0x813359b "toplevel") at ../../zsh-4.0/Src/exec.c:1028
#12 0x0807ac02 in loop (toplevel=1, justonce=0)
    at ../../zsh-4.0/Src/init.c:185
#13 0x0807d9db in zsh_main (argc=2, argv=0xbff08e44)
    at ../../zsh-4.0/Src/init.c:1528
#14 0x0804b386 in main (argc=2, argv=0xbff08e44)
    at ../../zsh-4.0/Src/main.c:93

Here findproc() returns 1.

After the job is killed with "kill %" there are two findproc() calls,
both of which return 0.

192		    if (pn->pid == pid && pn->status == SP_RUNNING) {
(gdb) p pn->status
$2 = 5247

(gdb) where
#0  findproc (pid=26597, jptr=0xbffe0d18, pptr=0xbffe0d14, aux=0)
    at ../../zsh-4.0/Src/jobs.c:200
#1  0x080b175b in wait_for_processes () at ../../zsh-4.0/Src/signals.c:493
#2  0x080b19aa in zhandler (sig=17) at ../../zsh-4.0/Src/signals.c:584
#3  <signal handler called>
#4  0x007067a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#5  0x00747d39 in sigprocmask () from /lib/tls/libc.so.6
#6  0x080b1533 in signal_unblock (set={__val = {65536, 0 <repeats 31 times>}})
    at ../../zsh-4.0/Src/signals.c:254
#7  0x080624f3 in execpline (state=0xbffe1830, slcode=4098, how=18, last1=0)
    at ../../zsh-4.0/Src/exec.c:1587
#8  0x080613e8 in execlist (state=0xbffe1830, dont_change_job=0, exiting=0)
    at ../../zsh-4.0/Src/exec.c:1207
#9  0x08060e7e in execode (p=0xb7d6a4d0, dont_change_job=0, exiting=0, 
    context=0x813359b "toplevel") at ../../zsh-4.0/Src/exec.c:1028
#10 0x0807ac02 in loop (toplevel=1, justonce=0)
    at ../../zsh-4.0/Src/init.c:185
#11 0x0807d9db in zsh_main (argc=2, argv=0xbffe1984)
    at ../../zsh-4.0/Src/init.c:1528
#12 0x0804b386 in main (argc=2, argv=0xbffe1984)
    at ../../zsh-4.0/Src/main.c:93

(gdb) where
#0  findproc (pid=26597, jptr=0xbffe0d18, pptr=0xbffe0d14, aux=1)
    at ../../zsh-4.0/Src/jobs.c:201
#1  0x080b17c7 in wait_for_processes () at ../../zsh-4.0/Src/signals.c:503
#2  0x080b19aa in zhandler (sig=17) at ../../zsh-4.0/Src/signals.c:584
#3  <signal handler called>
#4  0x007067a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#5  0x00747d39 in sigprocmask () from /lib/tls/libc.so.6
#6  0x080b1533 in signal_unblock (set={__val = {65536, 0 <repeats 31 times>}})
    at ../../zsh-4.0/Src/signals.c:254
#7  0x080624f3 in execpline (state=0xbffe1830, slcode=4098, how=18, last1=0)
    at ../../zsh-4.0/Src/exec.c:1587
#8  0x080613e8 in execlist (state=0xbffe1830, dont_change_job=0, exiting=0)
    at ../../zsh-4.0/Src/exec.c:1207
#9  0x08060e7e in execode (p=0xb7d6a4d0, dont_change_job=0, exiting=0, 
    context=0x813359b "toplevel") at ../../zsh-4.0/Src/exec.c:1028
#10 0x0807ac02 in loop (toplevel=1, justonce=0)
    at ../../zsh-4.0/Src/init.c:185
#11 0x0807d9db in zsh_main (argc=2, argv=0xbffe1984)
    at ../../zsh-4.0/Src/init.c:1528
#12 0x0804b386 in main (argc=2, argv=0xbffe1984)
    at ../../zsh-4.0/Src/main.c:93

With my patch then applied, findproc() returns 1 in the middle trace above,
which causes the job status to be updated.

-- 

Re: killing suspended jobs makes zsh hang after 47d1215

[Peter Stephenson]

Bart Schaefer wrote:
> The specific bug is fixed by the patch below, but I wonder if there are
> other job status that need to be tested here as well.

Hmm... the original bug was if the (original) job had gone completely.
With the change, the test now covers running and stopped (but still
recognised by the system, i.e. not terminated) jobs.  What else is
there?

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: killing suspended jobs makes zsh hang after 47d1215

[Bart Schaefer]

On Jun 12,  6:02pm, Peter Stephenson wrote:
}
} Hmm... the original bug was if the (original) job had gone completely.
} With the change, the test now covers running and stopped (but still
} recognised by the system, i.e. not terminated) jobs.  What else is
} there?

WIFSIGNALED()?  WIFCONTINUED()?

There's this hunk & comment in Src/jobs.c:

#ifndef WIFCONTINUED
            else if (sig == SIGCONT) {
                Job jn;
                Process pn;
                /* With WIFCONTINUED we find this out properly */
                if (findproc(pid, &jn, &pn, 0)) {
                    if (WIFSTOPPED(pn->status))
                        pn->status = SP_RUNNING;
                }
            }
#endif

But WIFCONTINUED() is tested only in update_job() which in turn is
called only when findproc() finds something (Src/signals.c):

        if (findproc(pid, &jn, &pn, 0)) {
#if defined(HAVE_WAIT3) && defined(HAVE_GETRUSAGE)
            struct timezone dummy_tz;
            gettimeofday(&pn->endtime, &dummy_tz);
            pn->status = status;
            pn->ti = ru;
#else
            update_process(pn, status);
#endif
            update_job(jn);
        } else if (findproc(pid, &jn, &pn, 1)) {
            pn->status = status;
            update_job(jn);

We were collectively wrong about SP_RUNNING; perhaps its the case that
there's a single state this ought to be avoiding, rather than a list of
them that it should be accepting.

Re: killing suspended jobs makes zsh hang after 47d1215

[Pan Tsu]

Bart Schaefer <schaefer@brasslantern.com> writes:

> On Jun 12,  7:22am, Bart Schaefer wrote:
> }
> } ----------------------------
> } revision 1.80
> } date: 2011/04/01 11:02:16;  author: pws;  state: Exp;  lines: +18 -1
> } Stef van Vlierberghe: 28965 (as posted in 28967):
> } findproc() should not return processes not marked as SP_RUNNING
> } ----------------------------
> } 
> } Evidently there are in fact cases where we need to find jobs that are
> } not running.
>
> The specific bug is fixed by the patch below, but I wonder if there are
> other job status that need to be tested here as well.

After playing a bit more with pn->status...

  % cat | cat &
  [1] 91580 97756
  [1]  + suspended (tty input)  cat | cat
  % %
  [1]  + continued  cat | cat
  ^C
  load: 0.06 no foreground process group

or when less(1) in `cat /dev/random | less' pipeline exits before
cat(1) is done

  Breakpoint 1, findproc (pid=97756, jptr=0x7fffffff4c08, pptr=0x7fffffff4c00, aux=0) at jobs.c:192
  192                 if (pn->pid == pid && pn->status == SP_RUNNING) {
  (gdb) p pn->status
  $1 = 19
  (gdb) p WIFSIGNALED(pn->status)
  $2 = 1
  (gdb) bt
  #0  findproc (pid=97756, jptr=0x7fffffff4c08, pptr=0x7fffffff4c00, aux=0) at jobs.c:192
  #1  0x000000000047d83c in wait_for_processes () at signals.c:493
  #2  0x000000000047d9f4 in zhandler (sig=20) at signals.c:584
  #3  0x00007ffffffff003 in ?? ()
  #4  0x000000000047d900 in wait_for_processes () at signals.c:524
  #5  0x00000000004481e8 in zwaitjob (job=1, wait_cmd=0) at jobs.c:1317
  #6  0x0000000000448390 in waitjobs () at jobs.c:1362
  #7  0x000000000044a1d9 in bin_fg (name=0x80f318738 "fg", argv=0x7fffffff52c0, ops=0x7fffffff5390, func=2) at jobs.c:2058
  #8  0x000000000040ed73 in execbuiltin (args=0x80f318700, bn=0x6af3c0) at builtin.c:450
  #9  0x000000000042ab5d in execcmd (state=0x7fffffff59a0, input=0, output=0, how=2, last1=2) at exec.c:3182
  #10 0x00000000004265a0 in execpline2 (state=0x7fffffff59a0, pcode=195, how=18, input=0, output=0, last1=0) at exec.c:1640
  #11 0x0000000000425aef in execpline (state=0x7fffffff59a0, slcode=3074, how=18, last1=0) at exec.c:1424
  #12 0x000000000042531e in execlist (state=0x7fffffff59a0, dont_change_job=0, exiting=0) at exec.c:1207
  #13 0x0000000000424d63 in execode (p=0x80f318660, dont_change_job=0, exiting=0, context=0x49b217 "toplevel") at exec.c:1028
  #14 0x00000000004412f3 in loop (toplevel=1, justonce=0) at init.c:185
  #15 0x000000000044431d in zsh_main (argc=2, argv=0x7fffffff5b00) at init.c:1528
  #16 0x000000000040e12b in main (argc=2, argv=0x7fffffff5b00) at ./main.c:93

Re: killing suspended jobs makes zsh hang after 47d1215

[Bart Schaefer]

On Jun 13,  4:02am, Pan Tsu wrote:
}
}   % cat | cat &
}   [1] 91580 97756
}   [1]  + suspended (tty input)  cat | cat
}   % %
}   [1]  + continued  cat | cat
}   ^C
}   load: 0.06 no foreground process group
} 
}   (gdb) p pn->status
}   $1 = 19
}   (gdb) p WIFSIGNALED(pn->status)

So I begin to suspect that what we really want here is

Index: Src/jobs.c
--- Src/jobs.c.~1.83.~	2011-06-12 08:05:58.000000000 -0700
+++ Src/jobs.c	2011-06-12 18:51:37.000000000 -0700
@@ -190,7 +190,7 @@
 	     * to return in a different job.
 	     */
 	    if (pn->pid == pid && (pn->status == SP_RUNNING ||
-				   WIFSTOPPED(pn->status))) {
+				   !WIFEXITED(pn->status))) {
 		*pptr = pn;
 		*jptr = jobtab + i;
 		return 1;

Re: killing suspended jobs makes zsh hang after 47d1215

[Peter Stephenson]

On Sun, 12 Jun 2011 18:53:39 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jun 13,  4:02am, Pan Tsu wrote:
> }
> }   % cat | cat &
> }   [1] 91580 97756
> }   [1]  + suspended (tty input)  cat | cat
> }   % %
> }   [1]  + continued  cat | cat
> }   ^C
> }   load: 0.06 no foreground process group
> } 
> }   (gdb) p pn->status
> }   $1 = 19
> }   (gdb) p WIFSIGNALED(pn->status)
> 
> So I begin to suspect that what we really want here is
> 
> Index: Src/jobs.c
> --- Src/jobs.c.~1.83.~	2011-06-12 08:05:58.000000000 -0700
> +++ Src/jobs.c	2011-06-12 18:51:37.000000000 -0700
> @@ -190,7 +190,7 @@
>  	     * to return in a different job.
>  	     */
>  	    if (pn->pid == pid && (pn->status == SP_RUNNING ||
> -				   WIFSTOPPED(pn->status))) {
> +				   !WIFEXITED(pn->status))) {
>  		*pptr = pn;
>  		*jptr = jobtab + i;
>  		return 1;

This is getting confusing.  WIFSIGNALED() indicates the process exited,
but on a signal rather than due to an exit call.  Why would we need to
distinguish those two cases?

It may be we need to distinguish the callers.  The original bug was when
a process that had exited, that was part of a job that had not yet
terminated, was being used inappropriately (see zsh-workers/28965).  In
this case it appears that under similar circumstances we need the
terminated job.  What is it in the current case that means we need the
process number even though the process has exited?

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: killing suspended jobs makes zsh hang after 47d1215

[Bart Schaefer]

On Jun 13, 12:07pm, Peter Stephenson wrote:
} Subject: Re: killing suspended jobs makes zsh hang after 47d1215
}
} It may be we need to distinguish the callers.  The original bug was when
} a process that had exited, that was part of a job that had not yet
} terminated, was being used inappropriately (see zsh-workers/28965).  In
} this case it appears that under similar circumstances we need the
} terminated job.  What is it in the current case that means we need the
} process number even though the process has exited?

In the original bug (28965 and long ago 12814) we have two "procs"
with the same PID, one (call it X) exited but not yet removed from
the job table and the other one (Y) still running.  When the state
of Y changes we want to update the job table for Y, but because we
search linearly by zsh job number we find X first and improperly
return that.  In this scenario we do not want to find a job that
isn't running, because we may confuse it with a job that IS running.

In the new bug we have no duplicate PIDs but we have a job whose state
is changing twice.  In one case (users/16092) it was stopped (so we
update the state to say it's not running) and then killed behind our
back (so the state changed again without first becoming running).  In
another case (workers/29475) the job really did exit, but we haven't
removed it from the job table because it is part of a pipeline which
has not yet finished.  We need to find it in order to finally clean
it up.

The first bug is obviously a whole lot more rare than the second, and
it's pretty unfortunate that we've broken a common case to fix one
that almost never happens.

It's possible that differentiating at the call to wait_for_processes()
between whether we're called from a signal handler or whether we are
called from bin_fg() might resolve the deadlock, but I don't think
that's the correct fix.  In the original bug we deadlock when called
from a signal handler, and in the new bug we deadlock later because
we did not update the job state correctly when called from a signal
handler (we skipped the job because it's "not running" even though
its state is changing a second time).

Changing the way bin_fg() works might detect the latter mistake after
the fact, but it will leave a bug that the job status is not being
correctly reflected in the output of "jobs".  (*Maybe* that's only a
problem in the WIFSTOPPED() case and combining 29472 with a flag
passed in through wait_for_processes() would suffice.)

In the 28965 case we might be able to fix it by having findproc()
continue to scan the table for running jobs any time it encounters
one that matches but is not running, as long as it eventually does
return the first one it found if there are no others.  That may be
a lot of overhead for a case that almost never happens.

Re: killing suspended jobs makes zsh hang after 47d1215

[Peter Stephenson]

On Mon, 13 Jun 2011 07:37:48 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> In the 28965 case we might be able to fix it by having findproc()
> continue to scan the table for running jobs any time it encounters
> one that matches but is not running, as long as it eventually does
> return the first one it found if there are no others.

Possibly I'm being dozy but this is the first thing I've heard that
sounds like a robust fix, if it's the case that we always find an
appropriate running process in the case that was causing the original
problem.

I don't think efficiency is a problem.  zsh makes no attempt to ensure
the process you're actually looking for is found quickly in any case.
If it is a problem, surely that means the process search is
intrinsically inefficient and we need to do something more basic about
it such as putting the processes in a hash table, or whatever.  Generally,
I suspect it's rare for the shell to have enough processes to make the
time is important, and if it is important this change is a minor
effect.  Of course, if anyone wants to profile it they're welcome.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: killing suspended jobs makes zsh hang after 47d1215

[Bart Schaefer]

On Jun 14,  7:54pm, Peter Stephenson wrote:
} Subject: Re: killing suspended jobs makes zsh hang after 47d1215
}
} On Mon, 13 Jun 2011 07:37:48 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > In the 28965 case we might be able to fix it by having findproc()
} > continue to scan the table for running jobs any time it encounters
} > one that matches but is not running, as long as it eventually does
} > return the first one it found if there are no others.
} 
} Possibly I'm being dozy but this is the first thing I've heard that
} sounds like a robust fix, if it's the case that we always find an
} appropriate running process in the case that was causing the original
} problem.

The only remaining glitch could be that we find the wrong process in
the event that somehow we recycled the whole range of PID values
without ever managing to handle the signal for the state change of
the first one to exit.  I suppose one could concoct a scenario in
which that's possible, but it'd be even harder to reproduce than the
original bug from years ago.

Anyway, that change looks something like this (second hunk just for
completeness):

Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.83
diff -u -r1.83 jobs.c
--- Src/jobs.c	12 Jun 2011 15:06:37 -0000	1.83
+++ Src/jobs.c	15 Jun 2011 02:56:08 -0000
@@ -160,6 +160,8 @@
     Process pn;
     int i;
 
+    *jptr = NULL;
+    *pptr = NULL;
     for (i = 1; i <= maxjob; i++)
     {
 	/*
@@ -189,16 +191,16 @@
 	     * the termination of the process which pid we were supposed
 	     * to return in a different job.
 	     */
-	    if (pn->pid == pid && (pn->status == SP_RUNNING ||
-				   WIFSTOPPED(pn->status))) {
+	    if (pn->pid == pid) {
 		*pptr = pn;
 		*jptr = jobtab + i;
-		return 1;
+		if (pn->status == SP_RUNNING) 
+		    return 1;
 	    }
 	}
     }
 
-    return 0;
+    return (*pptr && *jptr);
 }
 
 /* Does the given job number have any processes? */
Index: Src/signals.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/signals.c,v
retrieving revision 1.61
diff -u -r1.61 signals.c
--- Src/signals.c	14 Sep 2010 12:52:31 -0000	1.61
+++ Src/signals.c	15 Jun 2011 02:56:08 -0000
@@ -489,7 +489,6 @@
 	 * Find the process and job containing this pid and
 	 * update it.
 	 */
-	pn = NULL;
 	if (findproc(pid, &jn, &pn, 0)) {
 #if defined(HAVE_WAIT3) && defined(HAVE_GETRUSAGE)
 	    struct timezone dummy_tz;

-- 

It's time for 5.0.1

[Peter Stephenson]

I'd really like to get 5.0.1 out before Christmas, since there are a
number of fixes since 5.0.0 and nothing I can see that's likely to be
worse, even if it doesn't fix everything properly.  If no one can think of
a reason for waiting I'll make a test build in the next few days.
Typically, test builds are only hairy if there are configurational
changes that need trying on lots of architectures, which I don't think
is the case at the moment, so I'd expect to turn that into a release
fairly quickly.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: It's time for 5.0.1

[Bart Schaefer]

On Dec 6,  7:44pm, Peter Stephenson wrote:
} Subject: It's time for 5.0.1
}
} I'd really like to get 5.0.1 out before Christmas, since there are a
} number of fixes since 5.0.0 and nothing I can see that's likely to be
} worse, even if it doesn't fix everything properly.

I've committed users/17445 (leading zeroes in floats).  The only other
thing we might want to consider is [tweaked slightly] workers/30818.
The trouble is that as noted in 30826, it doesn't entirely solve the
underlying problem.  On the other hand, it might improve common cases
enough to be worth applying while we think about optimizing freeheap().

Re: It's time for 5.0.1

[Peter Stephenson]

On Sat, 08 Dec 2012 11:54:16 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> The only other
> thing we might want to consider is [tweaked slightly] workers/30818.
> The trouble is that as noted in 30826, it doesn't entirely solve the
> underlying problem.  On the other hand, it might improve common cases
> enough to be worth applying while we think about optimizing freeheap().

Yes, I'd say it's worth committing, although I don't know where we go next.

pws

Re: It's time for 5.0.1

[Bart Schaefer]

On Dec 9,  7:10pm, Peter Stephenson wrote:
} Subject: Re: It's time for 5.0.1
}
} On Sat, 08 Dec 2012 11:54:16 -0800
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > The only other
} > thing we might want to consider is [tweaked slightly] workers/30818.
} > The trouble is that as noted in 30826, it doesn't entirely solve the
} > underlying problem.  On the other hand, it might improve common cases
} > enough to be worth applying while we think about optimizing freeheap().
} 
} Yes, I'd say it's worth committing, although I don't know where we go next.

In workers/29175 I wrote this comment:

     * When pushheap() is called, it sweeps over the entire heaps list of       
     * arenas and marks every one of them with the amount of free space in      
     * that arena at that moment.  zhalloc() is then allowed to grab bits       
     * out of any of those arenas that have free space.                         

I'm now pretty sure that's completely wrong.  Does anyone recall what is
really supposed to be going on in pushheap() ?

On further examination the flow looks to me to go something like this:

1. We start out with heaps = NULL. zhalloc() is called, allocates a Heap
   struct, and points heaps at it.

2. Until the memory represented by that Heap struct is insufficient,
   zhalloc() carves slices off it.  So far so good.

3. When more memory is needed, zhalloc() allocates another Heap struct
   and chains it off the first one's next pointer.  Repeat as neeeded.

4. Now pushheap() is called.  For each Heap in the linked list, a new
   Heapstack is allocated and pushed into the Heap structs sp list.
   Each Heap object has some amount (possibly zero) of free space
   "above" the already used space in its arena.  The Heapstack tracks
   the starting position of this free space.

5. See steps 2 and 3.

6. Eventually freeheap() is called.  It walks through the Heapstack and
   resets the free space position in each corresponding Heap arena.

7. At last popheap() is called.  It both resets the free space like
   freeheap() and also discards the top Heapstack in each sp list.

If the above is all correct, then what pushheap()/freeheap()/popheap()
are designed to do is maximize re-use of the existing Heap arenas for
nested scopes; attempt to avoid allocating new Heap objects at the cost
of tracking/resetting the free space in every existing Heap arena.  This
puts conservation of space at a large premium over speed of operation.
(Which, given how profligate things like parameter expansion are with
their use of space, is maybe not entirely out of line.)

The problem with this is that the better zhalloc() is at filling up the
Heap arenas, the harder freeheap() has to work -- because it ends up
managing a Heapstack for each of a bunch of useless [that is, full] Heap
arenas.  At that point it'd be much better if it just pushed the entire
heaps list and started over from scratch.

Re: It's time for 5.0.1

[Peter Stephenson]

On Wed, 12 Dec 2012 00:14:11 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> The problem with this is that the better zhalloc() is at filling up
> the Heap arenas, the harder freeheap() has to work -- because it ends
> up managing a Heapstack for each of a bunch of useless [that is,
> full] Heap arenas.  At that point it'd be much better if it just
> pushed the entire heaps list and started over from scratch.

I think this all fits with what I roughly thought was going on.

One point about the above is that if we keep the heap size small, 16384
instead of ~ 10 times as much, it's much less wasteful to forget about
heapstacks altogether and simply mark the current heap and start a new
one when you push.  Then you just have the one linked list and
everything stays linear.  I think wasting something of order 16 kB each
time we push (and getting it back when we pop) is probably neither here
nor there these days.  Given that, so far as I know, actually allocating
a new heap isn't a bottlenec, that might be a way forward.

I'm not 100% sure what's going on with switch_heaps(), but you might
hope it was agnostic about the underlying heap implementation.

pws

Re: It's time for 5.0.1

[Bart Schaefer]

On Dec 12,  9:56am, Peter Stephenson wrote:
}
} On Wed, 12 Dec 2012 00:14:11 -0800
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > The problem with this is that the better zhalloc() is at filling up
} > the Heap arenas, the harder freeheap() has to work -- because it ends
} > up managing a Heapstack for each of a bunch of useless [that is,
} > full] Heap arenas.  At that point it'd be much better if it just
} > pushed the entire heaps list and started over from scratch.
} 
} One point about the above is that if we keep the heap size small, 16384
} instead of ~ 10 times as much

Regarding that ... there are some comments in zsh.h that make it appear
we should stick with multiples of 8, so 131072 or 262144 if we were in
fact to make it "about 10" times bigger.

} it's much less wasteful to forget about
} heapstacks altogether and simply mark the current heap and start a new
} one when you push.  Then you just have the one linked list and
} everything stays linear.

Well, there are still two linked lists: the currently growing heap and
the stack of pushed heaps.  I think we can do away with the Heapstack
struct and use the heapstack pointer in the Heap struct to manage the
pushed heaps.

} I think wasting something of order 16 kB each
} time we push (and getting it back when we pop) is probably neither here
} nor there these days.

Well ... in a pathological case (lots of allocations of 8k+1 bytes) you
might be wasting as much space as was already in use.  But you could
construct a similar pathological case for the current algorithm, so I
don't think it's worth solving right now.

} I'm not 100% sure what's going on with switch_heaps(), but you might
} hope it was agnostic about the underlying heap implementation.

I briefly considered NEWHEAP/OLDHEAP or SWITCHHEAPS/SWITCHBACKHEAPS as
a stand-in for pushheap/popheap in selected contexts, but concluded
that I don't understand well enough to initialize them ... nor do I
recall why the completion system is using them, except that it seems
to have something to do with completion-specific special parameters.

But I don't think they fiddle with how push/free/pop work underneath.

Re: It's time for 5.0.1

[Peter Stephenson]

On Wed, 12 Dec 2012 11:29:13 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> } it's much less wasteful to forget about
> } heapstacks altogether and simply mark the current heap and start a new
> } one when you push.  Then you just have the one linked list and
> } everything stays linear.
> 
> Well, there are still two linked lists: the currently growing heap and
> the stack of pushed heaps.  I think we can do away with the Heapstack
> struct and use the heapstack pointer in the Heap struct to manage the
> pushed heaps.

Right, there's the full linked list and what is essentially a parallel
linked list with each link pointing some way back into the full list.
So I suppose by default you'd just duplicate the second link from the
previous node when you pushed something onto the main list.

> } I think wasting something of order 16 kB each
> } time we push (and getting it back when we pop) is probably neither here
> } nor there these days.
> 
> Well ... in a pathological case (lots of allocations of 8k+1 bytes) you
> might be wasting as much space as was already in use.  But you could
> construct a similar pathological case for the current algorithm, so I
> don't think it's worth solving right now.

Actually, given that the whole point of the heap is so you don't have to
free things immediately and hence you're already sacrificing memory you
don't strictly need in favour of speed, I don't regard even that case as
particularly high up the pathology scale; it registers as a moderate
cold rather than the black death.

Anyway, it sounds to me like this is a post-5.0.1 project and there's
not a lot of point in short term fiddling, given that the problems
aren't new.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: It's time for 5.0.1

[Bart Schaefer]

On Dec 15,  6:16pm, Peter Stephenson wrote:
}
} Anyway, it sounds to me like this is a post-5.0.1 project and there's
} not a lot of point in short term fiddling, given that the problems
} aren't new.

Which brings us back to, do we bump HEAPSIZE by a factor of 8 for now,
or just punt entirely?

Re: It's time for 5.0.1

[Peter Stephenson]

Bart Schaefer wrote:
> On Dec 15,  6:16pm, Peter Stephenson wrote:
> }
> } Anyway, it sounds to me like this is a post-5.0.1 project and there's
> } not a lot of point in short term fiddling, given that the problems
> } aren't new.
> 
> Which brings us back to, do we bump HEAPSIZE by a factor of 8 for now,
> or just punt entirely?

If we've got a plan we can implement soon after 5.0.1, I'd be inclined
to say "leave it".  Maybe there's some pathological case where you get
lots of pushes and this makes the memory shoot up: given we know there
are pathologies, it's better to stick with the ones we've got rather
than force people to work around a whole new set that are about to
disappear.  That's an opinion based largely on ignorance rather than
knowledge, however.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: It's time for 5.0.1

[Peter Stephenson]

I've uploaded zsh-5.0.0-test-1 to ftp.zsh.org in the zsh/development
directory.  If you get a chance please see if there's anything wrong
with this before I release 5.0.1.

Thanks
pws

Re: It's time for 5.0.1

[Bart Schaefer]

On Dec 16,  8:03pm, Peter Stephenson wrote:
}
} I've uploaded zsh-5.0.0-test-1 to ftp.zsh.org in the zsh/development

Compiles and passes "make check" on MacOS Lion with macports.

Re: It's time for 5.0.1

[Axel Beckert]

Hi,

On Sun, Dec 16, 2012 at 08:03:56PM +0000, Peter Stephenson wrote:
> I've uploaded zsh-5.0.0-test-1 to ftp.zsh.org in the zsh/development
> directory.  If you get a chance please see if there's anything wrong
> with this before I release 5.0.1.

Looks fine for the debian builds on a first glance, too.

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.asciiribbon.org/              | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://noone.org/abe/ (Web)

Re: It's time for 5.0.1

[S. Cowles]

On Sun, 16 Dec 2012, Peter Stephenson wrote:

> I've uploaded zsh-5.0.0-test-1 to ftp.zsh.org in the zsh/development
> directory.  If you get a chance please see if there's anything wrong
> with this before I release 5.0.1.

Compile and link are fine on osx mountain lion (12.2.0) with homebrew, and ubu
12.04 32bit and 64bit.

I did run into an install problem that I have never encountered before; if it is
documented and I missed it, I apologize for the noise.

Just recently, I have begun running with a umask of 0077.  So, when the install
procedure creates new directories, the mode of the new dirs is 0077.  Since I install
as root, this creates the obvious suite of problems when running as a regular user:
permission denied errors for various shared libraries and functions (e.g. vcs_info).
To fix the install, I clean house and build from a zsh-clean environment with a
umask of 0022.

My current workaround is to force the umask to 0022 during build and install.  Would
it be appropriate to put mode setting code into mkinstalldirs?

Re: It's time for 5.0.1

[Peter Stephenson]

On Tue, 18 Dec 2012 08:22:58 -0800 (PST)
"S. Cowles" <scowles@ckhb.org> wrote:
> Just recently, I have begun running with a umask of 0077.  So, when
> the install procedure creates new directories, the mode of the new
> dirs is 0077.  Since I install as root, this creates the obvious
> suite of problems when running as a regular user: permission denied
> errors for various shared libraries and functions (e.g. vcs_info). To
> fix the install, I clean house and build from a zsh-clean environment
> with a umask of 0022.

I don't see why zsh shouldn't assume a default mode of 755 for
installation directories it creates.

Index: mkinstalldirs
===================================================================
RCS file: /cvsroot/zsh/zsh/mkinstalldirs,v
retrieving revision 1.2
diff -p -u -r1.2 mkinstalldirs
--- mkinstalldirs	10 Jun 2007 20:46:02 -0000	1.2
+++ mkinstalldirs	18 Dec 2012 16:42:47 -0000
@@ -15,7 +15,8 @@ nl='
 '
 IFS=" ""	$nl"
 errstatus=0
-dirmode=
+# Default directory mode for all zsh files is world read- and executable
+dirmode=755
 
 usage="\
 Usage: mkinstalldirs [-h] [--help] [--version] [-m MODE] DIR ...


-- 
Peter Stephenson <p.stephenson@samsung.com>       Consultant, Software
Tel: +44 (0)1223 434724              Samsung Cambridge Solution Centre
St John's House, St John's Innovation Park,
Cowley Road, Cambridge, CB4 0ZT, UK

bug with eval, proc-subst and pipes

[Stephane Chazelas]

Under:

ZDOTDIR=/some/new/dir zsh -f
autoload compinit
compinit

OK$ eval 'paste <(echo 1) <(echo 2) <(echo 3)' | cat
1       2       3

OK$ eval 'paste <(echo 1) <(echo 2) <(echo 3) <(echo 4)'
1       2       3       4

OK$ paste <(echo 1) <(echo 2) <(echo 3) <(echo 4) | cat
1       2       3       4

NOK$ eval 'paste <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
paste: /proc/self/fd/13: No such file or directory

That is, it occurs only with the "eval" with the "| something"
and with 4 process substitutions or more.

I can't reproduce it with zsh -f, and it starts to occur after
running "compinit". strangely not when compinit is called under
"set -x".

Reproduced with Zsh 5.0.2 on Debian amd64 on two different
machines, not with 4.3.17 on Ubuntu 12.04 amd64

-- 
Stephane

Re: bug with eval, proc-subst and pipes

[Bart Schaefer]

On Jul 15,  2:35pm, Stephane Chazelas wrote:
} Subject: bug with eval, proc-subst and pipes
}
} NOK$ eval 'paste <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
} paste: /proc/self/fd/13: No such file or directory

It's a race condition of some kind.  I can reproduce reliably with

% (){ sleep 5; paste "$@" } <(echo 1)

even without an "eval".

I believe this is happening because zsh asynchronously gets the child
exited signal for (echo 1) and immediately reaps it and therefore closes
its descriptors, but I don't have direct evidence.

Re[2]: bug with eval, proc-subst and pipes

[Manuel Presnitz]

SC> NOK$ eval 'paste <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
SC> paste: /proc/self/fd/13: No such file or directory
SC> Reproduced with Zsh 5.0.2 on Debian amd64 on two different
SC> machines, not with 4.3.17 on Ubuntu 12.04 amd64

BS> It's a race condition of some kind.  I can reproduce reliably with
BS> 
BS> % (){ sleep 5; paste "$@" } <(echo 1)
BS> 
BS> even without an "eval".

I don't know if it helps to get an idea what's going on, but
both variants work as expected with zsh 5.0.2 under the cygwin
environment:

$ uname -smo  
CYGWIN_NT-6.1-WOW64 i686 Cygwin
$ mkdir /tmp/empty
$ ZDOTDIR=/tmp/empty zsh -f
$ autoload compinit
$ compinit
$ eval 'paste <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
1       2       3       4
$ (){ sleep 5; paste "$@" } <(echo 1)
1
$ print $ZSH_VERSION $ZSH_PATCHLEVEL 
5.0.2 1.5778


Regards,
Manuel.

Re: Re[2]: bug with eval, proc-subst and pipes

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

On Mon, Jul 15, 2013 at 12:14 PM, Manuel Presnitz <mpy@gmx.net> wrote:

>
> I don't know if it helps to get an idea what's going on, but
> both variants work as expected with zsh 5.0.2 under the cygwin
> environment:
>

I suspect this is because Cygwin does not support the /proc/fd magic
filesystem or its equivalent.

Indeed, if I remove PATH_DEV_FD from config.h and recompile, I can no
longer reproduce the error.

There has to be something more going on, though, because moving the call
that closes the /proc/fd descriptors from exec.c into jobs.c:deletefilelist
does not resolve the problem.

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Mon, 15 Jul 2013 10:06:24 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jul 15,  2:35pm, Stephane Chazelas wrote:
> } Subject: bug with eval, proc-subst and pipes
> }
> } NOK$ eval 'paste <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
> } paste: /proc/self/fd/13: No such file or directory
> 
> It's a race condition of some kind.  I can reproduce reliably with
> 
> % (){ sleep 5; paste "$@" } <(echo 1)
> 
> even without an "eval".

That's not a race; it looks like it because it relies on a command
running before the one that's using the command substitution, but in
fact the bug in this case is completely deterministic.  It may therefore
be a different problem from Stephane's.

The shell executes the () { ... }; it creates fd 11 (in this case, the
first beyond 10 which is SHTTY).  It marks it in the global fdtable
array as FDT_PROC_SUBST.

Now it starts executing inside the anonymous function.  It starts processing the
sleep and gets to the chunk in execcmd() where it handles the fork.  At
line 2864 of exec.c, as it's in the function that's calling sleep, it
closes all the file descriptors of type FDT_PROC_SUBST that it thinks
are private to the sleep.  Unfortunately as fdtable is global it has no
way of knowing that actually fd 11 is associated with the current
function, not the sleep, so it closes it anyway.  So when it gets to the
paste fd 11 is already closed.

I suppose a smarter way of marking such file descriptors is needed,
perhaps one associated with the appropriate job in the same way as
auxiliary processes are remembered.  Good luck avoiding leaks...

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: bug with eval, proc-subst and pipes

[Bart Schaefer]

On Jul 16,  9:55pm, Peter Stephenson wrote:
}
} On Mon, 15 Jul 2013 10:06:24 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > 
} > It's a race condition of some kind.  I can reproduce reliably with
} > 
} > % (){ sleep 5; paste "$@" } <(echo 1)
} > 
} > even without an "eval".
} 
} That's not a race; it looks like it because it relies on a command
} running before the one that's using the command substitution, but in
} fact the bug in this case is completely deterministic.  It may therefore
} be a different problem from Stephane's.

I don't think so, fundamentally.  The problem in both cases is that the
file descriptor for the process substitution is closed before the data
is read.  This has the unintended side-effect of removing the special
device file.  The reasons for the premature descriptor close may be
different, but the end result is the same; I believe fixing either one
would fix both.

If, instead, a fifo file created by zsh itself is used, it's not removed 
until the associated job is deleted from the job table.

} At line 2864 of exec.c, as it's in the function that's calling sleep,
} it closes all the file descriptors of type FDT_PROC_SUBST that it
} thinks are private to the sleep. Unfortunately as fdtable is global
} it has no way of knowing that actually fd 11 is associated with the
} current function, not the sleep, so it closes it anyway.

I think you've just explained why simply moving the closem() call is
not sufficient.
 
} I suppose a smarter way of marking such file descriptors is needed,
} perhaps one associated with the appropriate job in the same way as
} auxiliary processes are remembered.  Good luck avoiding leaks...

Not in the way auxiliary processes are remembered; in the way temporary
files are remembered.  With #undef PATH_DEV_FD, zsh seems to do a very
good job of cleaning up those FIFOs.

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Wed, 17 Jul 2013 00:00:27 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jul 16,  9:55pm, Peter Stephenson wrote:
> } It may therefore be a different problem from Stephane's.
> 
> I don't think so, fundamentally.  The problem in both cases is that the
> file descriptor for the process substitution is closed before the data
> is read.  This has the unintended side-effect of removing the special
> device file.  The reasons for the premature descriptor close may be
> different, but the end result is the same; I believe fixing either one
> would fix both.

We can see...
 
> } I suppose a smarter way of marking such file descriptors is needed,
> } perhaps one associated with the appropriate job in the same way as
> } auxiliary processes are remembered.  Good luck avoiding leaks...
> 
> Not in the way auxiliary processes are remembered; in the way temporary
> files are remembered.  With #undef PATH_DEV_FD, zsh seems to do a very
> good job of cleaning up those FIFOs.

In that case, this patch does the exact equivalent.  I've made the
entries in the temporary file list larger rather than adding a separate
entry to the job since it should use less memory in general.

It seems to fix the completely reproducible variant.

diff --git a/Src/exec.c b/Src/exec.c
index 75805d3..d462d97 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -2860,9 +2860,6 @@ execcmd(Estate state, int input, int output, int how, int last1)
 	    close(synch[1]);
 	    read_loop(synch[0], &dummy, 1);
 	    close(synch[0]);
-#ifdef PATH_DEV_FD
-	    closem(FDT_PROC_SUBST);
-#endif
 	    if (how & Z_ASYNC) {
 		lastpid = (zlong) pid;
 		/* indicate it's possible to set status for lastpid */
@@ -3247,32 +3244,16 @@ execcmd(Estate state, int input, int output, int how, int last1)
 	    if (is_shfunc) {
 		/* It's a shell function */
 
-#ifdef PATH_DEV_FD
-		int i;
-
-		for (i = 10; i <= max_zsh_fd; i++)
-		    if (fdtable[i] >= FDT_PROC_SUBST)
-			fdtable[i]++;
-#endif
 		if (subsh_close >= 0)
 		    zclose(subsh_close);
 		subsh_close = -1;
 
 		execshfunc((Shfunc) hn, args);
-#ifdef PATH_DEV_FD
-		for (i = 10; i <= max_zsh_fd; i++)
-		    if (fdtable[i] >= FDT_PROC_SUBST)
-			if (--(fdtable[i]) <= FDT_PROC_SUBST)
-			    zclose(i);
-#endif
 	    } else {
 		/* It's a builtin */
 		if (forked)
 		    closem(FDT_INTERNAL);
 		lastval = execbuiltin(args, (Builtin) hn);
-#ifdef PATH_DEV_FD
-		closem(FDT_PROC_SUBST);
-#endif
 		fflush(stdout);
 		if (save[1] == -2) {
 		    if (ferror(stdout)) {
@@ -3887,9 +3868,7 @@ getoutputfile(char *cmd, char **eptr)
 	    untokenize(s);
     }
 
-    if (!jobtab[thisjob].filelist)
-	jobtab[thisjob].filelist = znewlinklist();
-    zaddlinknode(jobtab[thisjob].filelist, nam);
+    addfilelist(nam, 0);
 
     if (!s)
 	child_block();
@@ -3975,9 +3954,7 @@ getproc(char *cmd, char **eptr)
 	return NULL;
     if (!(prog = parsecmd(cmd, eptr)))
 	return NULL;
-    if (!jobtab[thisjob].filelist)
-	jobtab[thisjob].filelist = znewlinklist();
-    zaddlinknode(jobtab[thisjob].filelist, ztrdup(pnam));
+    addfilelist(pnam, 0);
 
     if ((pid = zfork(&bgtime))) {
 	if (pid == -1)
@@ -3995,7 +3972,7 @@ getproc(char *cmd, char **eptr)
     entersubsh(ESUB_ASYNC|ESUB_PGRP);
     redup(fd, out);
 #else /* PATH_DEV_FD */
-    int pipes[2];
+    int pipes[2], fd;
 
     if (thisjob == -1)
 	return NULL;
@@ -4012,7 +3989,9 @@ getproc(char *cmd, char **eptr)
 	    zclose(pipes[!out]);
 	    return NULL;
 	}
-	fdtable[pipes[!out]] = FDT_PROC_SUBST;
+	fd = pipes[!out];
+	fdtable[fd] = FDT_PROC_SUBST;
+	addfilelist(NULL, fd);
 	if (!out)
 	{
 	    addproc(pid, NULL, 1, &bgtime);
diff --git a/Src/jobs.c b/Src/jobs.c
index 0dbb10b..a1955bb 100644
--- a/Src/jobs.c
+++ b/Src/jobs.c
@@ -1113,16 +1113,48 @@ printjob(Job jn, int lng, int synch)
     return doneprint;
 }
 
+/* Add a file to be deleted or fd to be closed to the current job */
+
+/**/
+void
+addfilelist(const char *name, int fd)
+{
+    Jobfile jf = (Jobfile)zalloc(sizeof(struct jobfile));
+    LinkList ll = jobtab[thisjob].filelist;
+
+    if (!ll)
+	ll = jobtab[thisjob].filelist = znewlinklist();
+    if (name)
+    {
+	jf->u.name = ztrdup(name);
+	jf->is_fd = 0;
+    }
+    else
+    {
+	jf->u.fd = fd;
+	jf->is_fd = 1;
+    }
+    zaddlinknode(ll, jf);
+}
+
+/* Finished with list of files for a job */
+
 /**/
 void
 deletefilelist(LinkList file_list, int disowning)
 {
-    char *s;
+    Jobfile jf;
     if (file_list) {
-	while ((s = (char *)getlinknode(file_list))) {
-	    if (!disowning)
-		unlink(s);
-	    zsfree(s);
+	while ((jf = (Jobfile)getlinknode(file_list))) {
+	    if (jf->is_fd) {
+		if (!disowning)
+		    zclose(jf->u.fd);
+	    } else {
+		if (!disowning)
+		    unlink(jf->u.name);
+		zsfree(jf->u.name);
+	    }
+	    zfree(jf, sizeof(*jf));
 	}
 	zfree(file_list, sizeof(struct linklist));
     }
diff --git a/Src/zsh.h b/Src/zsh.h
index 299357d..ebd3cb7 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -374,9 +374,8 @@ enum {
 #ifdef PATH_DEV_FD
 /*
  * Entry used by a process substition.
- * The value will be incremented on entering a function and
- * decremented on exit; we don't close entries greater than
- * FDT_PROC_SUBST except when closing everything.
+ * This marker is not tested internally as we associated the file
+ * descriptor with a job for closing.
  */
 #define FDT_PROC_SUBST		6
 #endif
@@ -422,6 +421,7 @@ typedef struct heap      *Heap;
 typedef struct heapstack *Heapstack;
 typedef struct histent   *Histent;
 typedef struct hookdef   *Hookdef;
+typedef struct jobfile   *Jobfile;
 typedef struct job       *Job;
 typedef struct linkedmod *Linkedmod;
 typedef struct linknode  *LinkNode;
@@ -878,6 +878,18 @@ struct eccstr {
 /* Definitions for job table and job control */
 /********************************************/
 
+/* Entry in filelist linked list in job table */
+
+struct jobfile {
+    /* Record to be deleted or closed */
+    union {
+	char *name; /* Name of file to delete */
+	int fd;	    /* File descriptor to close */
+    } u;
+    /* Discriminant */
+    int is_fd;
+};
+
 /* entry in the job table */
 
 struct job {
@@ -889,6 +901,7 @@ struct job {
     struct process *procs;	/* list of processes                 */
     struct process *auxprocs;	/* auxiliary processes e.g multios   */
     LinkList filelist;		/* list of files to delete when done */
+				/* elements are struct jobfile */
     int stty_in_env;		/* if STTY=... is present            */
     struct ttyinfo *ty;		/* the modes specified by STTY       */
 };

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: bug with eval, proc-subst and pipes

[Vin Shelton]

On Wed, Jul 17, 2013 at 3:17 PM, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
> On Wed, 17 Jul 2013 00:00:27 -0700
> Bart Schaefer <schaefer@brasslantern.com> wrote:
>> On Jul 16,  9:55pm, Peter Stephenson wrote:
>> } It may therefore be a different problem from Stephane's.
>>
>> I don't think so, fundamentally.  The problem in both cases is that the
>> file descriptor for the process substitution is closed before the data
>> is read.  This has the unintended side-effect of removing the special
>> device file.  The reasons for the premature descriptor close may be
>> different, but the end result is the same; I believe fixing either one
>> would fix both.
>
> We can see...

"make check" now hangs and cannot be interrupted with ^C.  This is on
a Fedora 18 system with the following configure options:

/opt/src/zsh-2013-07-17/configure'  '--prefix=/opt/zsh-2013-07-17'
'--datadir=/opt/share' '--enable-etcdir=/opt/etc' '--with-tcsetpgrp'
'--enable-zsh-mem' '--enable-pcre' 'CC=gcc'

Let me know if you need me to run any more tests.

Here's a verbose trace:

ZTST_verbose=3 make TESTNUM=D03 check
cd Test ; make check
make[1]: Entering directory `/opt/build/zsh-2013-07-17/Test'
if test -n "gcc"; then \
  cd .. && DESTDIR= \
  make MODDIR=`pwd`/Test/Modules install.modules > /dev/null; \
fi
if ZTST_testlist="`for f in /opt/src/zsh-2013-07-17/Test/D03*.ztst; \
           do echo $f; done`" \
 ZTST_srcdir="/opt/src/zsh-2013-07-17/Test" \
 ZTST_exe=../Src/zsh \
 ../Src/zsh +Z -f /opt/src/zsh-2013-07-17/Test/runtests.zsh; then \
 stat=0; \
else \
 stat=1; \
fi; \
sleep 1; \
rm -rf Modules .zcompdump; \
exit $stat
/opt/src/zsh-2013-07-17/Test/D03procsubst.ztst: starting.
ZTST_getsect: read section name: prep
ZTST_getchunk: read code chunk:
  if grep '#define PATH_DEV_FD' $ZTST_testdir/../config.h > /dev/null 2>&1 ||
     grep '#define HAVE_FIFOS' $ZTST_testdir/../config.h > /dev/null 2>&1; then
    mkdir procsubst.tmp
    cd procsubst.tmp
    print 'First\tSecond\tThird\tFourth' >FILE1
    print 'Erste\tZweite\tDritte\tVierte' >FILE2
  else
    ZTST_unimplemented="process substitution is not supported"
    true
  fi
ZTST_execchunk: status 0
ZTST_getchunk: read code chunk:
  function copycat { cat "$@" }
ZTST_execchunk: status 0
ZTST_getchunk: read code chunk:

ZTST_getsect: read section name: test
ZTST_test: looking for new test
ZTST_test: examining line:
  paste <(cut -f1 FILE1) <(cut -f3 FILE2)
ZTST_getchunk: read code chunk:
  paste <(cut -f1 FILE1) <(cut -f3 FILE2)
ZTST_test: examining line:
>First  Dritte
ZTST_getredir: read redir for '>':
First   Dritte
ZTST_test: examining line:

Running test: <(...) substitution
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.15606, output: /tmp/zsh.ztst.out.15606, error:
/tmp/zsh.ztst.terr.15606
ZTST_execchunk: status 0
ZTST_test: test produced standard output:
First   Dritte
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  { paste <(cut -f2 FILE1) <(cut -f4 FILE2) } > >(sed 's/e/E/g' >OUTFILE)
ZTST_getchunk: read code chunk:
  { paste <(cut -f2 FILE1) <(cut -f4 FILE2) } > >(sed 's/e/E/g' >OUTFILE)
  cat OUTFILE
ZTST_test: examining line:
>SEcond ViErtE
ZTST_getredir: read redir for '>':
SEcond  ViErtE
ZTST_test: examining line:

Running test: >(...) substitution
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.15606, output: /tmp/zsh.ztst.out.15606, error:
/tmp/zsh.ztst.terr.15606
ZTST_execchunk: status 0
ZTST_test: test produced standard output:
SEcond  ViErtE
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  diff =(cat FILE1) =(cat FILE2)
ZTST_getchunk: read code chunk:
  diff =(cat FILE1) =(cat FILE2)
ZTST_test: examining line:
>1c1
ZTST_getredir: read redir for '>':
1c1
< First Second  Third   Fourth
---
> Erste Zweite  Dritte  Vierte
ZTST_test: examining line:

Running test: =(...) substituion
ZTST_test: expecting status: 1
Input: /tmp/zsh.ztst.in.15606, output: /tmp/zsh.ztst.out.15606, error:
/tmp/zsh.ztst.terr.15606
ZTST_execchunk: status 1
ZTST_test: test produced standard output:
1c1
< First Second  Third   Fourth
---
> Erste Zweite  Dritte  Vierte
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  copycat <(print First) <(print Zweite)
ZTST_getchunk: read code chunk:
  copycat <(print First) <(print Zweite)
ZTST_test: examining line:
>First
ZTST_getredir: read redir for '>':
First
Zweite
ZTST_test: examining line:

Running test: FDs remain open for external commands called from functions
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.15606, output: /tmp/zsh.ztst.out.15606, error:
/tmp/zsh.ztst.terr.15606
ZTST_execchunk: status 0
ZTST_test: test produced standard output:
First
Zweite
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  catfield2() {
ZTST_getchunk: read code chunk:
  catfield2() {
    local -a args
    args=(${(s.,.)1})
    print $args[1]
    cat $args[2]
    print $args[3]
  }
  catfield2 up,<(print $'\x64'own),sideways
ZTST_test: examining line:
>up
ZTST_getredir: read redir for '>':
up
down
sideways
ZTST_test: examining line:

Running test: <(...) when embedded within an argument
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.15606, output: /tmp/zsh.ztst.out.15606, error:
/tmp/zsh.ztst.terr.15606
ZTST_execchunk: status 0
ZTST_test: test produced standard output:
up
down
sideways
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  outputfield2() {
ZTST_getchunk: read code chunk:
  outputfield2() {
    local -a args
    args=(${(s.,.)1})
    print $args[1]
    echo 'How sweet the moonlight sits upon the bank' >$args[2]
    print $args[3]
  }
  outputfield2 muddy,>(sed -e s/s/th/g >outputfield2.txt),vesture
  # yuk
  while [[ ! -e outputfield2.txt || ! -s outputfield2.txt ]]; do :; done
  cat outputfield2.txt
ZTST_test: examining line:
>muddy
ZTST_getredir: read redir for '>':
muddy
vesture
How thweet the moonlight thitth upon the bank
ZTST_test: examining line:

Running test: >(...) when embedded within an argument
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.15606, output: /tmp/zsh.ztst.out.15606, error:
/tmp/zsh.ztst.terr.15606


 - Vin

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Wed, 17 Jul 2013 23:50:25 -0400
Vin Shelton <acs@alumni.princeton.edu> wrote:
> "make check" now hangs and cannot be interrupted with ^C.

Ah.  I think it's this:

  outputfield2() {
    local -a args
    args=(${(s.,.)1})
    print $args[1]
    echo 'How sweet the moonlight sits upon the bank' >$args[2]
    print $args[3]
  }
  outputfield2 muddy,>(sed -e s/s/th/g >outputfield2.txt),vesture

The shell is waiting for the sed to finish but it doesn't because its
input is still open, because we now don't close it until the job is
finished.  lsof seems to agree with me.

We're going to have to be quite smart about when we close output file
descriptors for process substitutions, because I think the problem we
saw before applies just as much to output as to input.

Will it be good enough to go through all process substitution output
file descriptors for a job when we enter zwaitjob() and close them?  I
think at that point nothing more can write from the main shell process,
and subprocesses in any case have to manage their own copies of the file
descriptor.

pws

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Thu, 18 Jul 2013 09:57:41 +0100
Peter Stephenson <p.stephenson@samsung.com> wrote:
> Will it be good enough to go through all process substitution output
> file descriptors for a job when we enter zwaitjob() and close them?  I
> think at that point nothing more can write from the main shell process,
> and subprocesses in any case have to manage their own copies of the file
> descriptor.

Thinking about it, it would presumably be safe just to close all
temporary file descriptors at that point, including input, since there's
nothing in the job that can read input now, either.  So there's no need
to distinguish input and output.  We keep the existing close logic as a
backup.  I don't think it's safe to delete temporary files yet, however,
since unlike the file descriptors, which are local to the process, those
can be accessed directly by subprocesses still running.

pws

Re: bug with eval, proc-subst and pipes

[Stephane Chazelas]

2013-07-17 20:17:33 +0100, Peter Stephenson:
[...]
> In that case, this patch does the exact equivalent.  I've made the
> entries in the temporary file list larger rather than adding a separate
> entry to the job since it should use less memory in general.
> 
> It seems to fix the completely reproducible variant.
[...]

Thanks,

for the record, in case it was not clear, it doesn't fix the
issue I reported initially.

Cheers,
Stephane

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Thu, 18 Jul 2013 10:22:27 +0100
Peter Stephenson <p.stephenson@samsung.com> wrote:
> On Thu, 18 Jul 2013 09:57:41 +0100
> Peter Stephenson <p.stephenson@samsung.com> wrote:
> > Will it be good enough to go through all process substitution output
> > file descriptors for a job when we enter zwaitjob() and close them?  I
> > think at that point nothing more can write from the main shell process,
> > and subprocesses in any case have to manage their own copies of the file
> > descriptor.
> 
> Thinking about it, it would presumably be safe just to close all
> temporary file descriptors at that point, including input, since there's
> nothing in the job that can read input now, either.  So there's no need
> to distinguish input and output.  We keep the existing close logic as a
> backup.  I don't think it's safe to delete temporary files yet, however,
> since unlike the file descriptors, which are local to the process, those
> can be accessed directly by subprocesses still running.

Apparently working (except for the fact it doesn't fix Stephane's
original problem, don't know if we're any further forward with that).
However, I'll wait in case Bart can see something I missed.

diff --git a/Src/jobs.c b/Src/jobs.c
index a1955bb..432f0f5 100644
--- a/Src/jobs.c
+++ b/Src/jobs.c
@@ -1368,6 +1368,30 @@ zwaitjob(int job, int wait_cmd)
 	jn->stat |= STAT_LOCKED;
 	if (jn->stat & STAT_CHANGED)
 	    printjob(jn, !!isset(LONGLISTJOBS), 1);
+	if (jn->filelist) {
+	    /*
+	     * The main shell is finished with any file descriptors used
+	     * for process substitution associated with this job: close
+	     * them to indicate to listeners there's no more input.
+	     *
+	     * Note we can't safely delete temporary files yet as these
+	     * are directly visible to other processes.  However,
+	     * we can't deadlock on the fact that those still exist, so
+	     * that's not a problem.
+	     */
+	    LinkNode node = firstnode(jn->filelist);
+	    while (node) {
+		Jobfile jf = (Jobfile)getdata(node);
+		if (jf->is_fd) {
+		    LinkNode next = nextnode(node);
+		    (void)remnode(jn->filelist, node);
+		    zclose(jf->u.fd);
+		    node = next;
+		} else {
+		    incnode(node);
+		}
+	    }
+	}
 	while (!errflag && jn->stat &&
 	       !(jn->stat & STAT_DONE) &&
 	       !(interact && (jn->stat & STAT_STOPPED))) {

pws

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Thu, 18 Jul 2013 12:32:10 +0100
Peter Stephenson <p.stephenson@samsung.com> wrote:
> +		if (jf->is_fd) {
> +		    LinkNode next = nextnode(node);
> +		    (void)remnode(jn->filelist, node);
> +		    zclose(jf->u.fd);
> +		    node = next;
> +		} else {

I distinctly remember writing

		    zfree(jf, sizeof(*jf));

but it's gone.  I'll put it back in when I commit.

Sorry for excess emails.

pws

Re: bug with eval, proc-subst and pipes

[Bart Schaefer]

On Jul 18, 10:44am, Stephane Chazelas wrote:
} Subject: Re: bug with eval, proc-subst and pipes
}
} 2013-07-17 20:17:33 +0100, Peter Stephenson:
} > It seems to fix the completely reproducible variant.
} 
} for the record, in case it was not clear, it doesn't fix the
} issue I reported initially.

Hmm.  Some strace output would be nice, then, because the only way the
/proc/self/fd or whatever file should be not found is if the descriptor
has been closed, and this patch should keep it from being closed until
after paste exits.

Re: bug with eval, proc-subst and pipes

[Bart Schaefer]

On Jul 18, 11:08am, Bart Schaefer wrote:
}
} Hmm.  Some strace output would be nice, then, because the only way the
} /proc/self/fd or whatever file should be not found is if the descriptor
} has been closed

Hrm, or if somehow it wasn't open *yet* when paste was forked, but that
makes even less sense.

Here's a possible experiment:

What do you get if you replace "paste" with "ls -l" ?

I get this on CentOS:

lr-x------  1 schaefer schaefer 64 Jul 18 11:22 /proc/self/fd/13 -> pipe:[815357]
lr-x------  1 schaefer schaefer 64 Jul 18 11:22 /proc/self/fd/14 -> pipe:[815358]
lr-x------  1 schaefer schaefer 64 Jul 18 11:22 /proc/self/fd/15 -> pipe:[815359]
lr-x------  1 schaefer schaefer 64 Jul 18 11:22 /proc/self/fd/16 -> pipe:[815360]

Or on ubuntu:

prw-rw----  0 schaefer  staff  2 Jul 18 11:20 /dev/fd/13
prw-rw----  0 schaefer  staff  2 Jul 18 11:20 /dev/fd/14
prw-rw----  0 schaefer  staff  2 Jul 18 11:20 /dev/fd/15
prw-rw----  0 schaefer  staff  2 Jul 18 11:20 /dev/fd/16

Re: bug with eval, proc-subst and pipes

[Stephane Chazelas]

[-- Attachment #1: Type: text/plain, Size: 2980 bytes --]

2013-07-18 12:32:10 +0100, Peter Stephenson:
[...]
> Apparently working (except for the fact it doesn't fix Stephane's
> original problem, don't know if we're any further forward with that).
> However, I'll wait in case Bart can see something I missed.
> 
> diff --git a/Src/jobs.c b/Src/jobs.c
[...]

Not better with that one (on top of the other one).

~$ eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
ls: cannot access /proc/self/fd/11: No such file or directory
ls: cannot access /proc/self/fd/13: No such file or directory
lr-x------ 1 chazelas chazelas 64 Jul 19 20:52 /proc/self/fd/10 -> pipe:[509333]
lr-x------ 1 chazelas chazelas 64 Jul 19 20:52 /proc/self/fd/12 -> pipe:[509335]

Though I've also seen (with one zsh invocation consistently but no other one):

~$ eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
ls: cannot access /proc/self/fd/11: No such file or directory
ls: cannot access /proc/self/fd/12: No such file or directory
ls: cannot access /proc/self/fd/13: No such file or directory
lr-x------ 1 chazelas chazelas 64 Jul 19 20:51 /proc/self/fd/10 -> pipe:[504591]

Attached is the strace output (for the first case).

See how fd 13 is closed in the process that execs ls and fd 11
in its parent shortly before the fork.

I verified that I can reproduce it from a fresh debootstrap of
debian sid amd64, so it should be easily reproduced from a
Debian-based system like ubuntu or ubuntu live CD.

Procedure:

sudo debootstrap --arch amd64 sid debian http://ftp.uk.debian.org/debian
cd debian
sudo mount --bind /dev dev
sudo mount -t proc p proc
sudo mount -t sysfs s sys
sudo mount -t devpts d dev/pts
chroot . bash
apt-get install zsh
zsh
autoload compinit
compinit
eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat

From the config.status of the debian package build:

ac_cs_config="'--build=x86_64-linux-gnu' '--includedir=\${prefix}/include' '--mandir=\${prefix}/share/man' '--infodir=\${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=\${prefix}/lib/x86_64-linux-gnu' '--libexecdir=\${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--prefix=/usr' '--mandir=/usr/share/man' '--bindir=/bin' 'LDFLAGS=-Wl,--as-needed -g -Wl,-z,relro' '--infodir=/usr/share/info' '--enable-maildir-support' '--enable-max-jobtable-size=256' '--enable-etcdir=/etc/zsh' '--enable-function-subdirs' '--enable-site-fndir=/usr/local/share/zsh/site-functions' '--enable-fndir=/usr/share/zsh/functions' '--with-tcsetpgrp' '--with-term-lib=ncursesw tinfo' '--enable-cap' '--enable-pcre' '--enable-readnullcmd=pager' '--enable-custom-patchlevel=Debian' '--enable-additional-fpath=/usr/share/zsh/vendor-functions,/usr/share/zsh/vendor-completions' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -g' 'CPPFLAGS=-D_FORTIFY_SOURCE=2'"

-- 
Stephane

[-- Attachment #2: strace.out --]
[-- Type: text/plain, Size: 62065 bytes --]

8713  20:20:27.164587 read(10, "\n", 1) = 1
8713  20:20:28.250709 fcntl(0, F_DUPFD, 10) = 11
8713  20:20:28.250958 close(0)          = 0
8713  20:20:28.251208 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:28.251431 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:28.251555 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:28.251665 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:28.251775 write(1, "\33[?1l\33>", 7) = 7
8713  20:20:28.251890 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:28.251940 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:28.252006 dup2(11, 0)       = 0
8713  20:20:28.252045 close(11)         = 0
8713  20:20:28.252090 write(10, "\33[K", 3) = 3
8713  20:20:28.252170 write(10, "\r\n", 2) = 2
8713  20:20:28.252217 alarm(0)          = 0
8713  20:20:28.252345 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
8713  20:20:28.252401 ioctl(10, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo ...}) = 0
8713  20:20:28.252489 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
8713  20:20:28.252545 munmap(0x7fccc6374000, 16384) = 0
8713  20:20:28.252595 ioctl(10, TIOCSPGRP, [8713]) = 0
8713  20:20:28.252643 fstat(0, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
8713  20:20:28.252700 fcntl(0, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
8713  20:20:28.252747 mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fccc6374000
8713  20:20:28.252803 geteuid()         = 1000
8713  20:20:28.252847 capget(0x20080522, 0, NULL) = 0
8713  20:20:28.252894 capget(0x20080522, 0, {0, 0, 0}) = 0
8713  20:20:28.252948 write(10, "\33[1m\33[3m%\33[23m\33[1m\33[0m          "..., 215) = 215
8713  20:20:28.253043 rt_sigaction(SIGINT, {0x473a00, [], SA_RESTORER|SA_INTERRUPT, 0x34e1435260}, NULL, 8) = 0
8713  20:20:28.253105 rt_sigaction(SIGINT, {0x473a00, [], SA_RESTORER|SA_INTERRUPT, 0x34e1435260}, NULL, 8) = 0
8713  20:20:28.253235 geteuid()         = 1000
8713  20:20:28.253313 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
8713  20:20:28.253407 ioctl(10, FIONREAD, [0]) = 0
8713  20:20:28.253468 ioctl(10, TIOCSPGRP, [8713]) = 0
8713  20:20:28.253515 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
8713  20:20:28.253563 ioctl(10, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig -icanon -echo ...}) = 0
8713  20:20:28.253614 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
8713  20:20:28.253662 write(10, "\r\33[0m\33[23m\33[24m\33[J\33[3m1\33[23m~$ ", 31) = 31
8713  20:20:28.253715 write(10, "\33[K\33[181C20:20\33[3m\33[23m\33[186D", 29) = 29
8713  20:20:28.253784 fcntl(0, F_DUPFD, 10) = 11
8713  20:20:28.253832 close(0)          = 0
8713  20:20:28.253893 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:28.253953 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:28.254004 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:28.254055 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:28.254117 write(1, "\33[?1h\33=", 7) = 7
8713  20:20:28.254172 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:28.254223 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:28.254295 dup2(11, 0)       = 0
8713  20:20:28.254343 close(11)         = 0
8713  20:20:28.254391 read(10, "e", 1)  = 1
8713  20:20:32.498719 write(10, "e", 1) = 1
8713  20:20:32.498984 read(10, "v", 1)  = 1
8713  20:20:32.690584 write(10, "\10ev", 3) = 3
8713  20:20:32.690850 read(10, "\33", 1) = 1
8713  20:20:33.226092 read(10, "p", 1)  = 1
8713  20:20:33.226235 write(10, "al 'ls -l <(echo 1) <(echo 2) <("..., 56) = 56
8713  20:20:33.226325 read(10, "\n", 1) = 1
8713  20:20:37.314618 fcntl(0, F_DUPFD, 10) = 11
8713  20:20:37.314875 close(0)          = 0
8713  20:20:37.315248 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:37.315591 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.315819 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.316015 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:37.316227 write(1, "\33[?1l\33>", 7) = 7
8713  20:20:37.316364 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.316457 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.316579 dup2(11, 0)       = 0
8713  20:20:37.316659 close(11)         = 0
8713  20:20:37.316756 write(10, "\33[K", 3) = 3
8713  20:20:37.316856 write(10, "\r\n", 2) = 2
8713  20:20:37.317066 alarm(0)          = 0
8713  20:20:37.317419 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
8713  20:20:37.317651 ioctl(10, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo ...}) = 0
8713  20:20:37.317829 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
8713  20:20:37.317937 symlink("/pid-8713/host-sc", "/home/chazelas/.zsh-history.5.0.2.LOCK") = 0
8713  20:20:37.318063 open("/home/chazelas/.zsh-history.5.0.2", O_WRONLY|O_CREAT|O_NOCTTY|O_APPEND, 0600) = 3
8713  20:20:37.318145 fcntl(3, F_GETFL) = 0x8401 (flags O_WRONLY|O_APPEND|O_LARGEFILE)
8713  20:20:37.318213 fstat(3, {st_mode=S_IFREG|0600, st_size=334450, ...}) = 0
8713  20:20:37.318327 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fccc72c4000
8713  20:20:37.318401 lseek(3, 0, SEEK_CUR) = 0
8713  20:20:37.318468 write(3, "eval 'ls -l <(echo 1) <(echo 2) "..., 59) = 59
8713  20:20:37.318556 fstat(3, {st_mode=S_IFREG|0600, st_size=334509, ...}) = 0
8713  20:20:37.318645 close(3)          = 0
8713  20:20:37.318711 munmap(0x7fccc72c4000, 4096) = 0
8713  20:20:37.318783 unlink("/home/chazelas/.zsh-history.5.0.2.LOCK") = 0
8713  20:20:37.318918 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:37.318987 pipe([3, 4])      = 0
8713  20:20:37.319057 fcntl(3, F_DUPFD, 10) = 11
8713  20:20:37.319109 close(3)          = 0
8713  20:20:37.319160 fcntl(4, F_DUPFD, 10) = 12
8713  20:20:37.319210 close(4)          = 0
8713  20:20:37.319272 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.319336 pipe([3, 4])      = 0
8713  20:20:37.319408 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccc72bf9d0) = 18103
8713  20:20:37.319779 close(4 <unfinished ...>
18103 20:20:37.319833 close(3 <unfinished ...>
8713  20:20:37.319855 <... close resumed> ) = 0
18103 20:20:37.319874 <... close resumed> ) = 0
18103 20:20:37.319926 setpgid(0, 18103 <unfinished ...>
8713  20:20:37.319950 read(3,  <unfinished ...>
18103 20:20:37.319969 <... setpgid resumed> ) = 0
18103 20:20:37.320051 ioctl(10, TIOCSPGRP, [18103]) = 0
18103 20:20:37.320150 rt_sigaction(SIGTTOU, {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, {SIG_IGN, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.320264 rt_sigaction(SIGTTIN, {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260}, {SIG_IGN, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.320370 rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260}, {SIG_IGN, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.320487 rt_sigaction(SIGTERM, {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, {SIG_IGN, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.320603 rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260}, {0x473a00, [], SA_RESTORER|SA_INTERRUPT, 0x34e1435260}, 8) = 0
18103 20:20:37.320719 rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260}, {SIG_IGN, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.320871 getrusage(RUSAGE_CHILDREN, {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18103 20:20:37.320973 close(4)          = 0
8713  20:20:37.321054 <... read resumed> "", 1) = 0
18103 20:20:37.321166 dup2(12, 1 <unfinished ...>
8713  20:20:37.321190 close(3 <unfinished ...>
18103 20:20:37.321210 <... dup2 resumed> ) = 1
18103 20:20:37.321281 close(12 <unfinished ...>
8713  20:20:37.321303 <... close resumed> ) = 0
18103 20:20:37.321324 <... close resumed> ) = 0
18103 20:20:37.321374 close(10 <unfinished ...>
8713  20:20:37.321410 close(12 <unfinished ...>
18103 20:20:37.321429 <... close resumed> ) = 0
18103 20:20:37.321499 close(11 <unfinished ...>
8713  20:20:37.321519 <... close resumed> ) = 0
18103 20:20:37.321538 <... close resumed> ) = 0
18103 20:20:37.321592 close(13 <unfinished ...>
8713  20:20:37.321650 access("/bin/cat", X_OK) = 0
18103 20:20:37.321750 <... close resumed> ) = 0
8713  20:20:37.321775 stat("/bin/cat", {st_mode=S_IFREG|0755, st_size=58800, ...}) = 0
8713  20:20:37.321885 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18103 20:20:37.321932 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
8713  20:20:37.321958 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18103 20:20:37.321983 <... rt_sigprocmask resumed> [CHLD], 8) = 0
8713  20:20:37.322008 pipe( <unfinished ...>
18103 20:20:37.322047 pipe( <unfinished ...>
8713  20:20:37.322068 <... pipe resumed> [3, 4]) = 0
18103 20:20:37.322090 <... pipe resumed> [3, 4]) = 0
8713  20:20:37.322117 clone( <unfinished ...>
18103 20:20:37.322137 fcntl(3, F_DUPFD, 10) = 10
18103 20:20:37.322204 close(3)          = 0
18103 20:20:37.322280 fcntl(4, F_DUPFD, 10) = 11
8713  20:20:37.322350 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccc72bf9d0) = 18104
18103 20:20:37.322382 close(4 <unfinished ...>
8713  20:20:37.322404 close(4 <unfinished ...>
18104 20:20:37.322424 close(3 <unfinished ...>
18103 20:20:37.322442 <... close resumed> ) = 0
18104 20:20:37.322461 <... close resumed> ) = 0
18103 20:20:37.322484 clone( <unfinished ...>
18104 20:20:37.322504 setpgid(0, 18103 <unfinished ...>
8713  20:20:37.322523 <... close resumed> ) = 0
18104 20:20:37.322542 <... setpgid resumed> ) = 0
8713  20:20:37.322563 read(3,  <unfinished ...>
18104 20:20:37.322583 kill(4294949193, SIG_0) = 0
18104 20:20:37.322659 rt_sigaction(SIGTTOU, {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, {SIG_IGN, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.322776 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccc72bf9d0) = 18105
18104 20:20:37.322805 rt_sigaction(SIGTTIN, {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18105 20:20:37.322848 rt_sigaction(SIGINT, {SIG_IGN, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.322886 <... rt_sigaction resumed> {SIG_IGN, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18105 20:20:37.322924 <... rt_sigaction resumed> {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.322964 rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18105 20:20:37.323005 rt_sigaction(SIGQUIT, {SIG_IGN, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.323042 <... rt_sigaction resumed> {SIG_IGN, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18105 20:20:37.323079 <... rt_sigaction resumed> {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.323119 rt_sigaction(SIGTERM, {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18105 20:20:37.323158 ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS <unfinished ...>
18104 20:20:37.323178 <... rt_sigaction resumed> {SIG_IGN, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18105 20:20:37.323215 <... ioctl resumed> , {B38400 opost isig icanon echo ...}) = 0
18104 20:20:37.323246 rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18105 20:20:37.323285 close(0 <unfinished ...>
18104 20:20:37.323303 <... rt_sigaction resumed> {0x473a00, [], SA_RESTORER|SA_INTERRUPT, 0x34e1435260}, 8) = 0
18105 20:20:37.323340 <... close resumed> ) = 0
18104 20:20:37.323362 rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18105 20:20:37.323401 open("/dev/null", O_RDWR|O_NOCTTY <unfinished ...>
18104 20:20:37.323427 <... rt_sigaction resumed> {SIG_IGN, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18105 20:20:37.323476 <... open resumed> ) = 0
18103 20:20:37.323498 close(11 <unfinished ...>
18105 20:20:37.323519 rt_sigaction(SIGTTOU, {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.323558 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18105 20:20:37.323577 <... rt_sigaction resumed> {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.323614 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18105 20:20:37.323659 rt_sigaction(SIGTTIN, {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.323696 close(4 <unfinished ...>
18105 20:20:37.323714 <... rt_sigaction resumed> {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.323752 <... close resumed> ) = 0
18105 20:20:37.323773 rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.323811 dup2(11, 0 <unfinished ...>
18105 20:20:37.323829 <... rt_sigaction resumed> {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.323866 <... dup2 resumed> ) = 0
18105 20:20:37.323888 rt_sigaction(SIGTERM, {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.323925 close(11 <unfinished ...>
18105 20:20:37.323943 <... rt_sigaction resumed> {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.323980 <... close resumed> ) = 0
18103 20:20:37.323999 <... close resumed> ) = 0
18105 20:20:37.324019 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18104 20:20:37.324039 close(10 <unfinished ...>
18105 20:20:37.324057 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18104 20:20:37.324098 <... close resumed> ) = 0
18105 20:20:37.324119 dup2(11, 1 <unfinished ...>
18104 20:20:37.324138 close(13 <unfinished ...>
18105 20:20:37.324156 <... dup2 resumed> ) = 1
18104 20:20:37.324174 <... close resumed> ) = 0
18105 20:20:37.324194 close(11 <unfinished ...>
18103 20:20:37.324213 pipe( <unfinished ...>
18105 20:20:37.324231 <... close resumed> ) = 0
18104 20:20:37.324252 rt_sigprocmask(SIG_UNBLOCK, [CHLD],  <unfinished ...>
18105 20:20:37.324277 close(10 <unfinished ...>
18104 20:20:37.324295 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18105 20:20:37.324318 <... close resumed> ) = 0
18103 20:20:37.324337 <... pipe resumed> [3, 4]) = 0
8713  20:20:37.324359 <... read resumed> "", 1) = 0
18105 20:20:37.324383 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18104 20:20:37.324408 close(13 <unfinished ...>
18105 20:20:37.324426 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18104 20:20:37.324449 <... close resumed> ) = -1 EBADF (Bad file descriptor)
18103 20:20:37.324482 fcntl(3, F_DUPFD, 10 <unfinished ...>
18104 20:20:37.324505 execve("/bin/cat", ["cat"], [/* 39 vars */] <unfinished ...>
18105 20:20:37.324576 write(1, "1\n", 2 <unfinished ...>
18103 20:20:37.324599 <... fcntl resumed> ) = 11
18105 20:20:37.324618 <... write resumed> ) = 2
18103 20:20:37.324639 close(3 <unfinished ...>
18105 20:20:37.324661 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18103 20:20:37.324686 <... close resumed> ) = 0
18105 20:20:37.324705 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18103 20:20:37.324733 fcntl(4, F_DUPFD, 10 <unfinished ...>
18105 20:20:37.324756 rt_sigprocmask(SIG_UNBLOCK, [CHLD],  <unfinished ...>
18103 20:20:37.324780 <... fcntl resumed> ) = 12
18105 20:20:37.324799 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18103 20:20:37.324825 close(4 <unfinished ...>
18105 20:20:37.324845 close(1 <unfinished ...>
18104 20:20:37.324865 <... execve resumed> ) = 0
18105 20:20:37.324883 <... close resumed> ) = 0
18105 20:20:37.324912 exit_group(0)     = ?
18104 20:20:37.324954 brk(0 <unfinished ...>
18103 20:20:37.324976 <... close resumed> ) = 0
18104 20:20:37.324995 <... brk resumed> ) = 0x1547000
18103 20:20:37.325019 clone( <unfinished ...>
8713  20:20:37.325039 close(3 <unfinished ...>
18104 20:20:37.325060 access("/etc/ld.so.nohwcap", F_OK <unfinished ...>
8713  20:20:37.325088 <... close resumed> ) = 0
18104 20:20:37.325118 <... access resumed> ) = -1 ENOENT (No such file or directory)
8713  20:20:37.325143 close(11 <unfinished ...>
18104 20:20:37.325167 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
8713  20:20:37.325189 <... close resumed> ) = 0
18104 20:20:37.325208 <... mmap resumed> ) = 0x7ff9a72c4000
18103 20:20:37.325264 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccc72bf9d0) = 18106
18104 20:20:37.325292 access("/etc/ld.so.preload", R_OK <unfinished ...>
18106 20:20:37.325323 rt_sigaction(SIGINT, {SIG_IGN, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.325362 <... access resumed> ) = -1 ENOENT (No such file or directory)
18106 20:20:37.325383 <... rt_sigaction resumed> {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.325423 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC <unfinished ...>
18106 20:20:37.325451 rt_sigaction(SIGQUIT, {SIG_IGN, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.325488 <... open resumed> ) = 3
18106 20:20:37.325507 <... rt_sigaction resumed> {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.325546 fstat(3,  <unfinished ...>
18106 20:20:37.325566 ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS <unfinished ...>
18104 20:20:37.325586 <... fstat resumed> {st_mode=S_IFREG|0644, st_size=239015, ...}) = 0
18106 20:20:37.325630 <... ioctl resumed> , {B38400 opost isig icanon echo ...}) = 0
18104 20:20:37.325663 mmap(NULL, 239015, PROT_READ, MAP_PRIVATE, 3, 0 <unfinished ...>
18106 20:20:37.325685 close(0 <unfinished ...>
18104 20:20:37.325703 <... mmap resumed> ) = 0x7ff9a7289000
18106 20:20:37.325722 <... close resumed> ) = 0
18104 20:20:37.325741 close(3 <unfinished ...>
18106 20:20:37.325762 open("/dev/null", O_RDWR|O_NOCTTY <unfinished ...>
18104 20:20:37.325787 <... close resumed> ) = 0
18106 20:20:37.325806 <... open resumed> ) = 0
18104 20:20:37.325827 access("/etc/ld.so.nohwcap", F_OK <unfinished ...>
18106 20:20:37.325855 rt_sigaction(SIGTTOU, {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.325891 <... access resumed> ) = -1 ENOENT (No such file or directory)
18106 20:20:37.325912 <... rt_sigaction resumed> {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.325953 open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC <unfinished ...>
18106 20:20:37.325984 rt_sigaction(SIGTTIN, {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.326021 <... open resumed> ) = 3
18106 20:20:37.326040 <... rt_sigaction resumed> {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.326080 read(3,  <unfinished ...>
18106 20:20:37.326102 rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.326138 <... read resumed> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\32B\3414\0\0\0"..., 832) = 832
18106 20:20:37.326173 <... rt_sigaction resumed> {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.326213 fstat(3,  <unfinished ...>
18106 20:20:37.326247 rt_sigaction(SIGTERM, {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.326288 <... fstat resumed> {st_mode=S_IFREG|0755, st_size=1740224, ...}) = 0
18106 20:20:37.326333 <... rt_sigaction resumed> {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.326375 mmap(0x34e1400000, 3849280, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0 <unfinished ...>
18106 20:20:37.326400 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18104 20:20:37.326418 <... mmap resumed> ) = 0x34e1400000
18106 20:20:37.326437 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18104 20:20:37.326482 mprotect(0x34e15a2000, 2097152, PROT_NONE <unfinished ...>
18106 20:20:37.326503 dup2(12, 1 <unfinished ...>
18104 20:20:37.326521 <... mprotect resumed> ) = 0
18106 20:20:37.326540 <... dup2 resumed> ) = 1
18104 20:20:37.326572 mmap(0x34e17a2000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a2000 <unfinished ...>
18106 20:20:37.326596 close(12 <unfinished ...>
18104 20:20:37.326615 <... mmap resumed> ) = 0x34e17a2000
18106 20:20:37.326633 <... close resumed> ) = 0
18104 20:20:37.326656 mmap(0x34e17a8000, 15424, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0 <unfinished ...>
18106 20:20:37.326678 close(10 <unfinished ...>
18104 20:20:37.326696 <... mmap resumed> ) = 0x34e17a8000
18106 20:20:37.326715 <... close resumed> ) = 0
18104 20:20:37.326734 close(3 <unfinished ...>
18106 20:20:37.326754 close(11 <unfinished ...>
18104 20:20:37.326772 <... close resumed> ) = 0
18106 20:20:37.326790 <... close resumed> ) = 0
18103 20:20:37.326810 close(12 <unfinished ...>
18104 20:20:37.326834 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
18106 20:20:37.326858 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18104 20:20:37.326975 <... mmap resumed> ) = 0x7ff9a7288000
18106 20:20:37.326997 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18104 20:20:37.327025 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
18103 20:20:37.327047 <... close resumed> ) = 0
18104 20:20:37.327066 <... mmap resumed> ) = 0x7ff9a7287000
8713  20:20:37.327089 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18106 20:20:37.327115 write(1, "2\n", 2 <unfinished ...>
18104 20:20:37.327141 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
18106 20:20:37.327162 <... write resumed> ) = 2
18104 20:20:37.327180 <... mmap resumed> ) = 0x7ff9a7286000
18106 20:20:37.327202 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18104 20:20:37.327227 arch_prctl(ARCH_SET_FS, 0x7ff9a7287700 <unfinished ...>
18106 20:20:37.327250 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18104 20:20:37.327275 <... arch_prctl resumed> ) = 0
18106 20:20:37.327296 rt_sigprocmask(SIG_UNBLOCK, [CHLD],  <unfinished ...>
18103 20:20:37.327321 pipe( <unfinished ...>
18106 20:20:37.327339 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18104 20:20:37.327365 mprotect(0x60b000, 4096, PROT_READ <unfinished ...>
18106 20:20:37.327385 close(1 <unfinished ...>
18104 20:20:37.327403 <... mprotect resumed> ) = 0
18106 20:20:37.327422 <... close resumed> ) = 0
18104 20:20:37.327442 mprotect(0x34e17a2000, 16384, PROT_READ <unfinished ...>
18106 20:20:37.327462 exit_group(0)     = ?
18104 20:20:37.327503 <... mprotect resumed> ) = 0
18103 20:20:37.327524 <... pipe resumed> [3, 4]) = 0
8713  20:20:37.327548 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18103 20:20:37.327591 fcntl(3, F_DUPFD, 10 <unfinished ...>
18104 20:20:37.327631 mprotect(0x34e1221000, 4096, PROT_READ <unfinished ...>
18103 20:20:37.327654 <... fcntl resumed> ) = 12
18104 20:20:37.327674 <... mprotect resumed> ) = 0
18103 20:20:37.327694 close(3 <unfinished ...>
18104 20:20:37.327714 munmap(0x7ff9a7289000, 239015 <unfinished ...>
18103 20:20:37.327733 <... close resumed> ) = 0
18104 20:20:37.327752 <... munmap resumed> ) = 0
18103 20:20:37.327774 fcntl(4, F_DUPFD, 10 <unfinished ...>
8713  20:20:37.327795 rt_sigsuspend([INT] <unfinished ...>
18103 20:20:37.327821 <... fcntl resumed> ) = 13
18103 20:20:37.327854 close(4)          = 0
18103 20:20:37.327915 clone( <unfinished ...>
18104 20:20:37.327938 brk(0)            = 0x1547000
18104 20:20:37.327997 brk(0x1568000)    = 0x1568000
18104 20:20:37.328075 open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC <unfinished ...>
18103 20:20:37.328143 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccc72bf9d0) = 18107
18104 20:20:37.328175 <... open resumed> ) = 3
18104 20:20:37.328206 fstat(3,  <unfinished ...>
18103 20:20:37.328228 close(13 <unfinished ...>
18107 20:20:37.328250 rt_sigaction(SIGINT, {SIG_IGN, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.328291 <... fstat resumed> {st_mode=S_IFREG|0644, st_size=2117296, ...}) = 0
18107 20:20:37.328348 <... rt_sigaction resumed> {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.328392 mmap(NULL, 2117296, PROT_READ, MAP_PRIVATE, 3, 0 <unfinished ...>
18107 20:20:37.328417 rt_sigaction(SIGQUIT, {SIG_IGN, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.328454 <... mmap resumed> ) = 0x7ff9a7081000
18107 20:20:37.328473 <... rt_sigaction resumed> {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.328512 close(3 <unfinished ...>
18107 20:20:37.328533 ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS <unfinished ...>
18104 20:20:37.328554 <... close resumed> ) = 0
18107 20:20:37.328573 <... ioctl resumed> , {B38400 opost isig icanon echo ...}) = 0
18103 20:20:37.328601 <... close resumed> ) = 0
18107 20:20:37.328621 close(0 <unfinished ...>
18104 20:20:37.328641 fstat(1,  <unfinished ...>
18107 20:20:37.328659 <... close resumed> ) = 0
18104 20:20:37.328678 <... fstat resumed> {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
18107 20:20:37.328725 open("/dev/null", O_RDWR|O_NOCTTY <unfinished ...>
18104 20:20:37.328752 fstat(0,  <unfinished ...>
18107 20:20:37.328771 <... open resumed> ) = 0
18104 20:20:37.328790 <... fstat resumed> {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
18107 20:20:37.328835 rt_sigaction(SIGTTOU, {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.328876 fadvise64(0, 0, 0, POSIX_FADV_SEQUENTIAL <unfinished ...>
18107 20:20:37.328896 <... rt_sigaction resumed> {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18104 20:20:37.328934 <... fadvise64 resumed> ) = -1 ESPIPE (Illegal seek)
18107 20:20:37.328958 rt_sigaction(SIGTTIN, {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18104 20:20:37.328997 read(0,  <unfinished ...>
18107 20:20:37.329016 <... rt_sigaction resumed> {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.329054 pipe( <unfinished ...>
18107 20:20:37.329076 rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18103 20:20:37.329113 <... pipe resumed> [3, 4]) = 0
18107 20:20:37.329135 <... rt_sigaction resumed> {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.329175 fcntl(3, F_DUPFD, 10 <unfinished ...>
18107 20:20:37.329197 rt_sigaction(SIGTERM, {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18103 20:20:37.329235 <... fcntl resumed> ) = 13
18107 20:20:37.329254 <... rt_sigaction resumed> {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.329292 close(3 <unfinished ...>
18107 20:20:37.329329 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18103 20:20:37.329350 <... close resumed> ) = 0
18107 20:20:37.329369 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18103 20:20:37.329414 fcntl(4, F_DUPFD, 10 <unfinished ...>
18107 20:20:37.329435 dup2(13, 1 <unfinished ...>
18103 20:20:37.329453 <... fcntl resumed> ) = 14
18107 20:20:37.329472 <... dup2 resumed> ) = 1
18103 20:20:37.329492 close(4 <unfinished ...>
18107 20:20:37.329512 close(13 <unfinished ...>
18103 20:20:37.329530 <... close resumed> ) = 0
18107 20:20:37.329548 <... close resumed> ) = 0
18103 20:20:37.329571 clone( <unfinished ...>
18107 20:20:37.329590 close(10)         = 0
18107 20:20:37.329653 close(11)         = 0
18107 20:20:37.329719 close(12)         = 0
18103 20:20:37.329785 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccc72bf9d0) = 18108
18107 20:20:37.329820 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18103 20:20:37.329849 close(14 <unfinished ...>
18107 20:20:37.329868 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18108 20:20:37.329895 rt_sigaction(SIGINT, {SIG_IGN, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18103 20:20:37.329933 <... close resumed> ) = 0
18108 20:20:37.329952 <... rt_sigaction resumed> {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18107 20:20:37.330002 write(1, "3\n", 2 <unfinished ...>
18108 20:20:37.330029 rt_sigaction(SIGQUIT, {SIG_IGN, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18107 20:20:37.330066 <... write resumed> ) = 2
18108 20:20:37.330085 <... rt_sigaction resumed> {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18107 20:20:37.330126 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18108 20:20:37.330152 ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS <unfinished ...>
18107 20:20:37.330173 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18108 20:20:37.330197 <... ioctl resumed> , {B38400 opost isig icanon echo ...}) = 0
18107 20:20:37.330252 rt_sigprocmask(SIG_UNBLOCK, [CHLD],  <unfinished ...>
18108 20:20:37.330284 close(0 <unfinished ...>
18107 20:20:37.330303 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18108 20:20:37.330327 <... close resumed> ) = 0
18107 20:20:37.330347 close(1 <unfinished ...>
18108 20:20:37.330369 open("/dev/null", O_RDWR|O_NOCTTY <unfinished ...>
18107 20:20:37.330395 <... close resumed> ) = 0
18108 20:20:37.330414 <... open resumed> ) = 0
18107 20:20:37.330434 exit_group(0)     = ?
18108 20:20:37.330476 rt_sigaction(SIGTTOU, {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18103 20:20:37.330517 close(11)         = 0
18103 20:20:37.330658 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18108 20:20:37.330688 <... rt_sigaction resumed> {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.330727 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18108 20:20:37.330754 rt_sigaction(SIGTTIN, {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18103 20:20:37.330794 ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS <unfinished ...>
18108 20:20:37.330814 <... rt_sigaction resumed> {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.330851 <... ioctl resumed> , 0x7fffbe693020) = -1 ENOTTY (Inappropriate ioctl for device)
18108 20:20:37.330880 rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18103 20:20:37.330921 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18108 20:20:37.330944 <... rt_sigaction resumed> {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.330982 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18108 20:20:37.331011 rt_sigaction(SIGTERM, {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18103 20:20:37.331093 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18108 20:20:37.331120 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18103 20:20:37.331139 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18108 20:20:37.331163 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18103 20:20:37.331206 pipe( <unfinished ...>
18108 20:20:37.331226 dup2(14, 1 <unfinished ...>
18103 20:20:37.331245 <... pipe resumed> [3, 4]) = 0
18108 20:20:37.331267 <... dup2 resumed> ) = 1
18103 20:20:37.331291 clone( <unfinished ...>
18108 20:20:37.331310 close(14)         = 0
18108 20:20:37.331380 close(10)         = 0
18108 20:20:37.331449 close(11 <unfinished ...>
18103 20:20:37.331501 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccc72bf9d0) = 18109
18108 20:20:37.331532 <... close resumed> ) = 0
18103 20:20:37.331553 close(4 <unfinished ...>
18109 20:20:37.331573 close(3 <unfinished ...>
18108 20:20:37.331592 close(12 <unfinished ...>
18109 20:20:37.331610 <... close resumed> ) = 0
18108 20:20:37.331629 <... close resumed> ) = 0
18109 20:20:37.331651 rt_sigaction(SIGTTOU, {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18108 20:20:37.331691 close(13 <unfinished ...>
18109 20:20:37.331710 <... rt_sigaction resumed> {SIG_DFL, [TTOU], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18108 20:20:37.331747 <... close resumed> ) = 0
18109 20:20:37.331769 rt_sigaction(SIGTTIN, {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18108 20:20:37.331808 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18109 20:20:37.331842 <... rt_sigaction resumed> {SIG_DFL, [TTIN], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18108 20:20:37.331881 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18109 20:20:37.331908 rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18103 20:20:37.331945 <... close resumed> ) = 0
18109 20:20:37.331964 <... rt_sigaction resumed> {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18108 20:20:37.332005 write(1, "4\n", 2 <unfinished ...>
18109 20:20:37.332030 rt_sigaction(SIGTERM, {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18108 20:20:37.332066 <... write resumed> ) = 2
18109 20:20:37.332085 <... rt_sigaction resumed> {SIG_DFL, [TERM], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18108 20:20:37.332125 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18109 20:20:37.332152 rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18108 20:20:37.332188 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18109 20:20:37.332212 <... rt_sigaction resumed> {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18108 20:20:37.332252 rt_sigprocmask(SIG_UNBLOCK, [CHLD],  <unfinished ...>
18109 20:20:37.332278 rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260},  <unfinished ...>
18108 20:20:37.332315 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18109 20:20:37.332338 <... rt_sigaction resumed> {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x34e1435260}, 8) = 0
18108 20:20:37.332377 close(1 <unfinished ...>
18103 20:20:37.332398 read(3,  <unfinished ...>
18109 20:20:37.332418 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18108 20:20:37.332436 <... close resumed> ) = 0
18109 20:20:37.332455 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18108 20:20:37.332498 exit_group(0)     = ?
18109 20:20:37.332539 close(4)          = 0
18103 20:20:37.332599 <... read resumed> "", 1) = 0
18103 20:20:37.332646 close(3 <unfinished ...>
18109 20:20:37.332671 rt_sigprocmask(SIG_UNBLOCK, [CHLD],  <unfinished ...>
18103 20:20:37.332698 <... close resumed> ) = 0
18109 20:20:37.332717 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18103 20:20:37.332757 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
18109 20:20:37.332821 close(13 <unfinished ...>
18103 20:20:37.332843 rt_sigsuspend([INT] <unfinished ...>
18109 20:20:37.332866 <... close resumed> ) = 0
18103 20:20:37.332886 <... rt_sigsuspend resumed> ) = ? ERESTARTNOHAND (To be restarted)
18109 20:20:37.332907 execve("/bin/ls", ["ls", "-l", "/proc/self/fd/10", "/proc/self/fd/11", "/proc/self/fd/12", "/proc/self/fd/13"], [/* 39 vars */] <unfinished ...>
18103 20:20:37.332998 --- SIGCHLD (Child exited) @ 0 (0) ---
18103 20:20:37.333039 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [INT CHLD], 8) = 0
18103 20:20:37.333127 rt_sigprocmask(SIG_SETMASK, [INT CHLD], ~[KILL STOP RTMIN RT_1], 8) = 0
18103 20:20:37.333219 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED|WCONTINUED, {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 18105
18109 20:20:37.333311 <... execve resumed> ) = 0
18103 20:20:37.333333 getrusage(RUSAGE_CHILDREN, {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18109 20:20:37.333408 brk(0 <unfinished ...>
18103 20:20:37.333431 wait4(-1,  <unfinished ...>
18109 20:20:37.333449 <... brk resumed> ) = 0x17fd000
18103 20:20:37.333469 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED|WCONTINUED, {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 18106
18109 20:20:37.333517 access("/etc/ld.so.nohwcap", F_OK <unfinished ...>
18103 20:20:37.333545 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18109 20:20:37.333564 <... access resumed> ) = -1 ENOENT (No such file or directory)
18103 20:20:37.333585 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18109 20:20:37.333632 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
18103 20:20:37.333657 wait4(-1,  <unfinished ...>
18109 20:20:37.333675 <... mmap resumed> ) = 0x7f25818cc000
18103 20:20:37.333707 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED|WCONTINUED, {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 18107
18109 20:20:37.333757 access("/etc/ld.so.preload", R_OK <unfinished ...>
18103 20:20:37.333786 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18109 20:20:37.333804 <... access resumed> ) = -1 ENOENT (No such file or directory)
18103 20:20:37.333825 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18109 20:20:37.333869 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC <unfinished ...>
18103 20:20:37.333897 wait4(-1,  <unfinished ...>
18109 20:20:37.333916 <... open resumed> ) = 3
18103 20:20:37.333935 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED|WCONTINUED, {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 18108
18109 20:20:37.333984 fstat(3,  <unfinished ...>
18103 20:20:37.334004 getrusage(RUSAGE_CHILDREN,  <unfinished ...>
18109 20:20:37.334023 <... fstat resumed> {st_mode=S_IFREG|0644, st_size=239015, ...}) = 0
18103 20:20:37.334068 <... getrusage resumed> {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 0
18109 20:20:37.334114 mmap(NULL, 239015, PROT_READ, MAP_PRIVATE, 3, 0 <unfinished ...>
18103 20:20:37.334137 wait4(-1,  <unfinished ...>
18109 20:20:37.334156 <... mmap resumed> ) = 0x7f2581891000
18103 20:20:37.334175 <... wait4 resumed> 0x7fffbe69267c, WNOHANG|WSTOPPED|WCONTINUED, 0x7fffbe6926a0) = 0
18109 20:20:37.334197 close(3 <unfinished ...>
18103 20:20:37.334217 rt_sigreturn(0xffffffff <unfinished ...>
18109 20:20:37.334261 <... close resumed> ) = 0
18103 20:20:37.334284 <... rt_sigreturn resumed> ) = -1 EINTR (Interrupted system call)
18109 20:20:37.334307 access("/etc/ld.so.nohwcap", F_OK <unfinished ...>
18103 20:20:37.334334 kill(18109, SIGCONT <unfinished ...>
18109 20:20:37.334352 <... access resumed> ) = -1 ENOENT (No such file or directory)
18103 20:20:37.334367 <... kill resumed> ) = 0
18109 20:20:37.334376 --- SIGCONT (Continued) @ 0 (0) ---
18103 20:20:37.334383 rt_sigprocmask(SIG_BLOCK, [CHLD],  <unfinished ...>
18109 20:20:37.334411 open("/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC <unfinished ...>
18103 20:20:37.334427 <... rt_sigprocmask resumed> [CHLD], 8) = 0
18109 20:20:37.334439 <... open resumed> ) = 3
18103 20:20:37.334449 rt_sigsuspend([INT] <unfinished ...>
18109 20:20:37.334461 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240a\300\3464\0\0\0"..., 832) = 832
18109 20:20:37.334518 fstat(3, {st_mode=S_IFREG|0644, st_size=137024, ...}) = 0
18109 20:20:37.334576 mmap(0x34e6c00000, 2234432, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e6c00000
18109 20:20:37.334628 mprotect(0x34e6c1f000, 2097152, PROT_NONE) = 0
18109 20:20:37.334679 mmap(0x34e6e1f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1f000) = 0x34e6e1f000
18109 20:20:37.334734 mmap(0x34e6e21000, 2112, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x34e6e21000
18109 20:20:37.334785 close(3)          = 0
18109 20:20:37.334831 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
18109 20:20:37.334894 open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.334954 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20#\200\3424\0\0\0"..., 832) = 832
18109 20:20:37.335012 fstat(3, {st_mode=S_IFREG|0644, st_size=34720, ...}) = 0
18109 20:20:37.335081 mmap(0x34e2800000, 2128888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e2800000
18109 20:20:37.335139 mprotect(0x34e2807000, 2093056, PROT_NONE) = 0
18109 20:20:37.335196 mmap(0x34e2a06000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x34e2a06000
18109 20:20:37.335259 close(3)          = 0
18109 20:20:37.335317 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
18109 20:20:37.335382 open("/lib/x86_64-linux-gnu/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.335442 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360!\300\3664\0\0\0"..., 832) = 832
18109 20:20:37.335504 fstat(3, {st_mode=S_IFREG|0644, st_size=37816, ...}) = 0
18109 20:20:37.335575 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2581890000
18109 20:20:37.335634 mmap(0x34f6c00000, 2130600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34f6c00000
18109 20:20:37.335692 mprotect(0x34f6c08000, 2093056, PROT_NONE) = 0
18109 20:20:37.335748 mmap(0x34f6e07000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x34f6e07000
18109 20:20:37.335811 close(3)          = 0
18109 20:20:37.335867 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
18109 20:20:37.335928 open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.335987 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\32B\3414\0\0\0"..., 832) = 832
18109 20:20:37.336047 fstat(3, {st_mode=S_IFREG|0755, st_size=1740224, ...}) = 0
18109 20:20:37.336109 mmap(0x34e1400000, 3849280, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e1400000
18109 20:20:37.336167 mprotect(0x34e15a2000, 2097152, PROT_NONE) = 0
18109 20:20:37.336225 mmap(0x34e17a2000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a2000) = 0x34e17a2000
18109 20:20:37.336291 mmap(0x34e17a8000, 15424, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x34e17a8000
18109 20:20:37.336344 close(3)          = 0
18109 20:20:37.336393 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
18109 20:20:37.336451 open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.336506 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\32\300\3424\0\0\0"..., 832) = 832
18109 20:20:37.336560 fstat(3, {st_mode=S_IFREG|0644, st_size=256584, ...}) = 0
18109 20:20:37.336617 mmap(0x34e2c00000, 2349248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e2c00000
18109 20:20:37.336667 mprotect(0x34e2c3e000, 2093056, PROT_NONE) = 0
18109 20:20:37.336717 mmap(0x34e2e3d000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3d000) = 0x34e2e3d000
18109 20:20:37.336771 close(3)          = 0
18109 20:20:37.336817 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
18109 20:20:37.336869 open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.336922 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@l\0\3424\0\0\0"..., 832) = 832
18109 20:20:37.336976 fstat(3, {st_mode=S_IFREG|0755, st_size=134067, ...}) = 0
18109 20:20:37.337028 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f258188f000
18109 20:20:37.337079 mmap(0x34e2000000, 2208688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e2000000
18109 20:20:37.337129 mprotect(0x34e2017000, 2093056, PROT_NONE) = 0
18109 20:20:37.337179 mmap(0x34e2216000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x34e2216000
18109 20:20:37.337234 mmap(0x34e2218000, 13232, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x34e2218000
18109 20:20:37.337283 close(3)          = 0
18109 20:20:37.337329 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
18109 20:20:37.337383 open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.337437 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\200\3414\0\0\0"..., 832) = 832
18109 20:20:37.337492 fstat(3, {st_mode=S_IFREG|0644, st_size=17400, ...}) = 0
18109 20:20:37.337544 mmap(0x34e1800000, 2109720, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e1800000
18109 20:20:37.337593 mprotect(0x34e1803000, 2093056, PROT_NONE) = 0
18109 20:20:37.337643 mmap(0x34e1a02000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x34e1a02000
18109 20:20:37.337697 close(3)          = 0
18109 20:20:37.337744 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
18109 20:20:37.337796 open("/lib/x86_64-linux-gnu/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.337850 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\24@\3604\0\0\0"..., 832) = 832
18109 20:20:37.337907 fstat(3, {st_mode=S_IFREG|0644, st_size=21112, ...}) = 0
18109 20:20:37.337964 mmap(0x34f0400000, 2113896, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34f0400000
18109 20:20:37.338013 mprotect(0x34f0404000, 2093056, PROT_NONE) = 0
18109 20:20:37.338063 mmap(0x34f0603000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x34f0603000
18109 20:20:37.338116 close(3)          = 0
18109 20:20:37.338159 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f258188e000
18109 20:20:37.338218 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f258188d000
18109 20:20:37.338265 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f258188b000
18109 20:20:37.338304 arch_prctl(ARCH_SET_FS, 0x7f258188b800) = 0
18109 20:20:37.338371 mprotect(0x61a000, 4096, PROT_READ) = 0
18109 20:20:37.338406 mprotect(0x34e6e1f000, 4096, PROT_READ) = 0
18109 20:20:37.338439 mprotect(0x34e2a06000, 4096, PROT_READ) = 0
18109 20:20:37.338472 mprotect(0x34f6e07000, 4096, PROT_READ) = 0
18109 20:20:37.338505 mprotect(0x34e17a2000, 16384, PROT_READ) = 0
18109 20:20:37.338539 mprotect(0x34e2216000, 4096, PROT_READ) = 0
18109 20:20:37.338572 mprotect(0x34e1a02000, 4096, PROT_READ) = 0
18109 20:20:37.338605 mprotect(0x34e1221000, 4096, PROT_READ) = 0
18109 20:20:37.338637 mprotect(0x34f0603000, 4096, PROT_READ) = 0
18109 20:20:37.338670 munmap(0x7f2581891000, 239015) = 0
18109 20:20:37.338723 set_tid_address(0x7f258188bad0) = 18109
18109 20:20:37.338767 set_robust_list(0x7f258188bae0, 0x18) = 0
18109 20:20:37.338813 rt_sigaction(SIGRTMIN, {0x34e20067a0, [], SA_RESTORER|SA_SIGINFO, 0x34e200f210}, NULL, 8) = 0
18109 20:20:37.338867 rt_sigaction(SIGRT_1, {0x34e2006820, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x34e200f210}, NULL, 8) = 0
18109 20:20:37.338920 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
18109 20:20:37.338967 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
18109 20:20:37.339042 statfs("/sys/fs/selinux", 0x7fffff3d8890) = -1 ENOENT (No such file or directory)
18109 20:20:37.339101 statfs("/selinux", 0x7fffff3d8890) = -1 ENOENT (No such file or directory)
18109 20:20:37.339197 brk(0)            = 0x17fd000
18109 20:20:37.339245 brk(0x181e000)    = 0x181e000
18109 20:20:37.339297 open("/proc/filesystems", O_RDONLY) = 3
18109 20:20:37.339362 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
18109 20:20:37.339419 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f25818cb000
18109 20:20:37.339468 read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 1024) = 378
18109 20:20:37.339537 read(3, "", 1024) = 0
18109 20:20:37.339581 close(3)          = 0
18109 20:20:37.339630 munmap(0x7f25818cb000, 4096) = 0
18109 20:20:37.339699 open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.339755 fstat(3, {st_mode=S_IFREG|0644, st_size=2117296, ...}) = 0
18109 20:20:37.339808 mmap(NULL, 2117296, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f2581686000
18109 20:20:37.339859 close(3)          = 0
18109 20:20:37.339918 ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffff3d83f0) = -1 ENOTTY (Inappropriate ioctl for device)
18109 20:20:37.339967 ioctl(1, TIOCGWINSZ, 0x7fffff3d84d0) = -1 ENOTTY (Inappropriate ioctl for device)
18109 20:20:37.340029 open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.340086 fstat(3, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
18109 20:20:37.340139 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f25818cb000
18109 20:20:37.340187 read(3, "# Locale name alias data base.\n#"..., 4096) = 2502
18109 20:20:37.340254 read(3, "", 4096) = 0
18109 20:20:37.340298 close(3)          = 0
18109 20:20:37.340346 munmap(0x7f25818cb000, 4096) = 0
18109 20:20:37.340413 open("/usr/share/locale/en_GB.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.340469 open("/usr/share/locale/en_GB.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.340525 open("/usr/share/locale/en_GB/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.340579 open("/usr/share/locale/en.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.340632 open("/usr/share/locale/en.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.340685 open("/usr/share/locale/en/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.340748 open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3
18109 20:20:37.340805 fstat(3, {st_mode=S_IFREG|0644, st_size=26258, ...}) = 0
18109 20:20:37.340858 mmap(NULL, 26258, PROT_READ, MAP_SHARED, 3, 0) = 0x7f25818c5000
18109 20:20:37.340918 close(3)          = 0
18109 20:20:37.340992 futex(0x34e17a7918, FUTEX_WAKE_PRIVATE, 2147483647) = 0
18109 20:20:37.341123 lstat("/proc/self/fd/10", {st_mode=S_IFLNK|0500, st_size=64, ...}) = 0
18109 20:20:37.341227 lgetxattr("/proc/self/fd/10", "security.selinux", 0x1804100, 255) = -1 EOPNOTSUPP (Operation not supported)
18109 20:20:37.341292 readlink("/proc/self/fd/10", "pipe:[454132]", 65) = 13
18109 20:20:37.341360 socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
18109 20:20:37.341415 connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
18109 20:20:37.341519 sendto(3, "\2\0\0\0\v\0\0\0\7\0\0\0passwd\0", 19, MSG_NOSIGNAL, NULL, 0) = 19
18109 20:20:37.341603 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 ([{fd=3, revents=POLLIN|POLLHUP}])
18109 20:20:37.341653 recvmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"passwd\0", 7}, {"\310O\3\0\0\0\0\0", 8}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {4}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 15
18109 20:20:37.341720 mmap(NULL, 217032, PROT_READ, MAP_SHARED, 4, 0) = 0x7f2581651000
18109 20:20:37.341772 close(4)          = 0
18109 20:20:37.341816 close(3)          = 0
18109 20:20:37.341870 socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
18109 20:20:37.341920 connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
18109 20:20:37.342019 sendto(3, "\2\0\0\0\f\0\0\0\6\0\0\0group\0", 18, MSG_NOSIGNAL, NULL, 0) = 18
18109 20:20:37.342075 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 ([{fd=3, revents=POLLIN|POLLHUP}])
18109 20:20:37.342127 recvmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"group\0", 6}, {"\310O\3\0\0\0\0\0", 8}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {4}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 14
18109 20:20:37.342184 mmap(NULL, 217032, PROT_READ, MAP_SHARED, 4, 0) = 0x7f258161c000
18109 20:20:37.342233 close(4)          = 0
18109 20:20:37.342273 close(3)          = 0
18109 20:20:37.342315 lstat("/proc/self/fd/11", 0x17ff6c0) = -1 ENOENT (No such file or directory)
18109 20:20:37.342383 open("/usr/share/locale/en_GB.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.342430 open("/usr/share/locale/en_GB.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.342475 open("/usr/share/locale/en_GB/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.342519 open("/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.342562 open("/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.342606 open("/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.342663 write(2, "ls: ", 4) = 4
18109 20:20:37.342752 write(2, "cannot access /proc/self/fd/11", 30) = 30
18109 20:20:37.342809 open("/usr/share/locale/en_GB.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.342971 open("/usr/share/locale/en_GB.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.343025 open("/usr/share/locale/en_GB/LC_MESSAGES/libc.mo", O_RDONLY) = 3
18109 20:20:37.343064 fstat(3, {st_mode=S_IFREG|0644, st_size=1474, ...}) = 0
18109 20:20:37.343106 mmap(NULL, 1474, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f25818c4000
18109 20:20:37.343140 close(3)          = 0
18109 20:20:37.343175 open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.343214 open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.343248 open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
18109 20:20:37.343290 write(2, ": No such file or directory", 27) = 27
18109 20:20:37.343329 write(2, "\n", 1) = 1
18109 20:20:37.343377 lstat("/proc/self/fd/12", {st_mode=S_IFLNK|0500, st_size=64, ...}) = 0
18109 20:20:37.343428 lgetxattr("/proc/self/fd/12", "security.selinux", 0x18053f0, 255) = -1 EOPNOTSUPP (Operation not supported)
18109 20:20:37.343472 readlink("/proc/self/fd/12", "pipe:[459548]", 65) = 13
18109 20:20:37.343518 lstat("/proc/self/fd/13", 0x17ff780) = -1 ENOENT (No such file or directory)
18109 20:20:37.343559 write(2, "ls: ", 4) = 4
18109 20:20:37.343616 write(2, "cannot access /proc/self/fd/13", 30) = 30
18109 20:20:37.343734 write(2, ": No such file or directory", 27) = 27
18109 20:20:37.343863 write(2, "\n", 1) = 1
18109 20:20:37.343954 fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
18109 20:20:37.344020 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f25818c3000
18109 20:20:37.344070 open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
18109 20:20:37.344116 fstat(3, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
18109 20:20:37.344164 fstat(3, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
18109 20:20:37.344212 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f25818c2000
18109 20:20:37.344252 read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\7\0\0\0\0"..., 4096) = 3661
18109 20:20:37.344300 lseek(3, -2338, SEEK_CUR) = 1323
18109 20:20:37.344337 read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 2338
18109 20:20:37.344383 close(3)          = 0
18109 20:20:37.344420 munmap(0x7f25818c2000, 4096) = 0
18109 20:20:37.344477 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
18109 20:20:37.344533 write(1, "lr-x------ 1 chazelas chazelas 6"..., 162) = 162
18104 20:20:37.344575 <... read resumed> "lr-x------ 1 chazelas chazelas 6"..., 32768) = 162
18109 20:20:37.344593 close(1 <unfinished ...>
18104 20:20:37.344604 write(1, "lr-x------ 1 chazelas chazelas 6"..., 162 <unfinished ...>
18109 20:20:37.344616 <... close resumed> ) = 0
18104 20:20:37.344626 <... write resumed> ) = 162
18109 20:20:37.344636 munmap(0x7f25818c3000, 4096 <unfinished ...>
18104 20:20:37.344646 read(0,  <unfinished ...>
18109 20:20:37.344655 <... munmap resumed> ) = 0
18109 20:20:37.344676 close(2)          = 0
18109 20:20:37.344731 exit_group(2)     = ?
18103 20:20:37.344873 <... rt_sigsuspend resumed> ) = ? ERESTARTNOHAND (To be restarted)
18103 20:20:37.344901 --- SIGCHLD (Child exited) @ 0 (0) ---
18103 20:20:37.344922 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [INT CHLD], 8) = 0
18103 20:20:37.344964 rt_sigprocmask(SIG_SETMASK, [INT CHLD], ~[KILL STOP RTMIN RT_1], 8) = 0
18103 20:20:37.345003 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 2}], WNOHANG|WSTOPPED|WCONTINUED, {ru_utime={0, 0}, ru_stime={0, 4000}, ...}) = 18109
18103 20:20:37.345067 wait4(-1, 0x7fffbe69267c, WNOHANG|WSTOPPED|WCONTINUED, 0x7fffbe6926a0) = -1 ECHILD (No child processes)
18103 20:20:37.345104 rt_sigreturn(0xffffffff) = -1 EINTR (Interrupted system call)
18103 20:20:37.345143 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
18103 20:20:37.345183 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
18103 20:20:37.345223 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
18103 20:20:37.345258 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
18103 20:20:37.345305 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
18103 20:20:37.345340 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
18103 20:20:37.345401 close(10)         = 0
18103 20:20:37.345442 close(12)         = 0
18103 20:20:37.345479 close(13)         = 0
18103 20:20:37.345517 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
18103 20:20:37.345551 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
18103 20:20:37.345591 close(0)          = 0
18103 20:20:37.345620 close(1 <unfinished ...>
18104 20:20:37.345673 <... read resumed> "", 32768) = 0
18103 20:20:37.345696 <... close resumed> ) = 0
18104 20:20:37.345716 close(0 <unfinished ...>
18103 20:20:37.345732 close(2 <unfinished ...>
18104 20:20:37.345741 <... close resumed> ) = 0
18103 20:20:37.345750 <... close resumed> ) = 0
18104 20:20:37.345768 close(1 <unfinished ...>
18103 20:20:37.345779 exit_group(2)     = ?
18104 20:20:37.345800 <... close resumed> ) = 0
18104 20:20:37.345823 close(2)          = 0
18104 20:20:37.345874 exit_group(0)     = ?
8713  20:20:37.345923 <... rt_sigsuspend resumed> ) = ? ERESTARTNOHAND (To be restarted)
8713  20:20:37.345948 --- SIGCHLD (Child exited) @ 0 (0) ---
8713  20:20:37.345970 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [INT CHLD], 8) = 0
8713  20:20:37.346012 rt_sigprocmask(SIG_SETMASK, [INT CHLD], ~[KILL STOP RTMIN RT_1], 8) = 0
8713  20:20:37.346051 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 2}], WNOHANG|WSTOPPED|WCONTINUED, {ru_utime={0, 0}, ru_stime={0, 4000}, ...}) = 18103
8713  20:20:37.346102 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED|WCONTINUED, {ru_utime={0, 0}, ru_stime={0, 0}, ...}) = 18104
8713  20:20:37.346150 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
8713  20:20:37.346189 ioctl(10, TIOCGPGRP, [18103]) = 0
8713  20:20:37.346223 ioctl(10, TIOCSPGRP, [8713]) = 0
8713  20:20:37.346291 ioctl(10, TIOCGWINSZ, {ws_row=52, ws_col=191, ws_xpixel=1920, ws_ypixel=1055}) = 0
8713  20:20:37.346345 wait4(-1, 0x7fffbe693e7c, WNOHANG|WSTOPPED|WCONTINUED, 0x7fffbe693ea0) = 0
8713  20:20:37.346392 rt_sigreturn(0xffffffff) = -1 EINTR (Interrupted system call)
8713  20:20:37.346438 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.346485 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.346535 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:37.346584 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.346635 munmap(0x7fccc6374000, 16384) = 0
8713  20:20:37.346692 ioctl(10, TIOCSPGRP, [8713]) = 0
8713  20:20:37.346738 fstat(0, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
8713  20:20:37.346795 fcntl(0, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
8713  20:20:37.346843 mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fccc6374000
8713  20:20:37.346903 geteuid()         = 1000
8713  20:20:37.346952 capget(0x20080522, 0, NULL) = 0
8713  20:20:37.346998 capget(0x20080522, 0, {0, 0, 0}) = 0
8713  20:20:37.347052 write(10, "\33[1m\33[3m%\33[23m\33[1m\33[0m          "..., 215) = 215
8713  20:20:37.347132 rt_sigaction(SIGINT, {0x473a00, [], SA_RESTORER|SA_INTERRUPT, 0x34e1435260}, NULL, 8) = 0
8713  20:20:37.347191 rt_sigaction(SIGINT, {0x473a00, [], SA_RESTORER|SA_INTERRUPT, 0x34e1435260}, NULL, 8) = 0
8713  20:20:37.347269 geteuid()         = 1000
8713  20:20:37.347329 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
8713  20:20:37.347405 ioctl(10, FIONREAD, [0]) = 0
8713  20:20:37.347452 ioctl(10, TIOCSPGRP, [8713]) = 0
8713  20:20:37.347498 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
8713  20:20:37.347546 ioctl(10, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig -icanon -echo ...}) = 0
8713  20:20:37.347604 ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
8713  20:20:37.347660 write(10, "\r\33[0m\33[23m\33[24m\33[J\33[3m1\33[23m~$ ", 31) = 31
8713  20:20:37.347727 write(10, "\33[K\33[181C20:20\33[3m\33[23m\33[186D", 29) = 29
8713  20:20:37.347786 fcntl(0, F_DUPFD, 10) = 11
8713  20:20:37.347837 close(0)          = 0
8713  20:20:37.347896 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:37.347955 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.348004 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.348053 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
8713  20:20:37.348113 write(1, "\33[?1h\33=", 7) = 7
8713  20:20:37.348168 rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.348218 rt_sigprocmask(SIG_UNBLOCK, [CHLD], [CHLD], 8) = 0
8713  20:20:37.348278 dup2(11, 0)       = 0
8713  20:20:37.348321 close(11)         = 0

Re: bug with eval, proc-subst and pipes

[Stephane Chazelas]

2013-07-19 21:01:59 +0100, Stephane Chazelas:
[...]
> See how fd 13 is closed in the process that execs ls and fd 11
> in its parent shortly before the fork.
[...]

According to gdb, fd 11 is closed by zclose(subsh_close) in
execcmd()

and fd 13 by closedumps() (the missing link to compinit?).

(gdb) p *dumps
$2 = {next = 0x0, dev = 64768, ino = 66808, fd = 13, map = 0x7f0d4e833000, addr = 0x7f0d4e833000, len = 125304, count = 1,
  filename = 0x13ae650 "/usr/share/zsh/functions/Completion/Base.zwc"}

It sounds like those fd are closed too many times. I don't have
much time to investigate any further.

Cheers,
Stephane

Re: bug with eval, proc-subst and pipes

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

On Friday, July 19, 2013, Stephane Chazelas wrote:

> 2013-07-18 12:32:10 +0100, Peter Stephenson:
> [...]
> > Apparently working (except for the fact it doesn't fix Stephane's
> > original problem, don't know if we're any further forward with that).
> > However, I'll wait in case Bart can see something I missed.


For some reason I never received that message from Peter.  Did it perhaps
go privately to Stephen?

~$ eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
> ls: cannot access /proc/self/fd/11: No such file or directory
> ls: cannot access /proc/self/fd/13: No such file or directory
> lr-x------ 1 chazelas chazelas 64 Jul 19 20:52 /proc/self/fd/10 ->
> pipe:[509333]
> lr-x------ 1 chazelas chazelas 64 Jul 19 20:52 /proc/self/fd/12 ->
> pipe:[509335]


Ok, this is kinda strange, because on my Ubuntu system there is no
/proc/self/fd ... The "ls" shows /dev/fd files.  Why would that be
different?

And it's further odd that some of the descriptors are closed but not all of
them.


> [In strace]
>
> See how fd 13 is closed in the process that execs ls and fd 11
> in its parent shortly before the fork.


That's also odd.  It would almost seem to imply that those descriptors got
the close-on-exec flag set somehow.

Re: bug with eval, proc-subst and pipes

[Stephane Chazelas]

2013-07-19 14:36:07 -0700, Bart Schaefer:
> On Friday, July 19, 2013, Stephane Chazelas wrote:
> 
> > 2013-07-18 12:32:10 +0100, Peter Stephenson:
> > [...]
> > > Apparently working (except for the fact it doesn't fix Stephane's
> > > original problem, don't know if we're any further forward with that).
> > > However, I'll wait in case Bart can see something I missed.
> 
> 
> For some reason I never received that message from Peter.  Did it perhaps
> go privately to Stephen?

http://thread.gmane.org/gmane.comp.shells.zsh.devel/26797/focus=26823

> ~$ eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
> > ls: cannot access /proc/self/fd/11: No such file or directory
> > ls: cannot access /proc/self/fd/13: No such file or directory
> > lr-x------ 1 chazelas chazelas 64 Jul 19 20:52 /proc/self/fd/10 ->
> > pipe:[509333]
> > lr-x------ 1 chazelas chazelas 64 Jul 19 20:52 /proc/self/fd/12 ->
> > pipe:[509335]
> 
> 
> Ok, this is kinda strange, because on my Ubuntu system there is no
> /proc/self/fd ... The "ls" shows /dev/fd files.  Why would that be
> different?

>From config.log:

configure:10381: checking for /dev/fd filesystem
configure:10394: result: /proc/self/fd

> 
> And it's further odd that some of the descriptors are closed but not all of
> them.
> 
> 
> > [In strace]
> >
> > See how fd 13 is closed in the process that execs ls and fd 11
> > in its parent shortly before the fork.
> 
> 
> That's also odd.  It would almost seem to imply that those descriptors got
> the close-on-exec flag set somehow.

If you look at my other email
(http://thread.gmane.org/gmane.comp.shells.zsh.devel/26797/focus=26823).
Those fds are closed because they are taken for something else
(I suppose because they had been closed before already and
reassigned).

-- 
Stephane

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Fri, 19 Jul 2013 21:58:53 +0100
Stephane Chazelas <stephane.chazelas@gmail.com> wrote:
> and fd 13 by closedumps() (the missing link to compinit?).

That's distinctly suspicious, if 13 was opened for the process
substitution.

Does the following fix it, or is there yet more?

If it does we still have an error for older systems, but I suspect
FD_CLOEXEC (and O_CLOEXEC) are widely enough supported that may be more
a theoretical then an actual problem.

diff --git a/Src/exec.c b/Src/exec.c
index d462d97..f9efb3a 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -448,7 +448,9 @@ zexecve(char *pth, char **argv, char **newenvp)
     else
 	sprintf(buf + 2, "%s/%s", pwd, pth);
     zputenv(buf);
+#ifndef FD_CLOEXEC
     closedumps();
+#endif
 
     if (newenvp == NULL)
 	    newenvp = environ;
diff --git a/Src/parse.c b/Src/parse.c
index 753080d..b670925 100644
--- a/Src/parse.c
+++ b/Src/parse.c
@@ -3171,6 +3171,9 @@ load_dump_file(char *dump, struct stat *sbuf, int other, int len)
     d->dev = sbuf->st_dev;
     d->ino = sbuf->st_ino;
     d->fd = fd;
+#ifdef FD_CLOEXEC
+    fcntl(fd, F_SETFD, FD_CLOEXEC);
+#endif
     d->map = addr + (other ? (len - off) / sizeof(wordcode) : 0);
     d->addr = addr;
     d->len = len;
@@ -3439,6 +3442,7 @@ decrdumpcount(FuncDump f)
     }
 }
 
+#ifndef FD_CLOEXEC
 /**/
 mod_export void
 closedumps(void)
@@ -3448,6 +3452,7 @@ closedumps(void)
     for (p = dumps; p; p = p->next)
 	zclose(p->fd);
 }
+#endif
 
 #else
 
@@ -3461,11 +3466,13 @@ decrdumpcount(FuncDump f)
 {
 }
 
+#ifndef FD_CLOEXEC
 /**/
 mod_export void
 closedumps(void)
 {
 }
+#endif
 
 #endif
 
-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: bug with eval, proc-subst and pipes

[Stephane Chazelas]

2013-07-20 18:11:44 +0100, Peter Stephenson:
> On Fri, 19 Jul 2013 21:58:53 +0100
> Stephane Chazelas <stephane.chazelas@gmail.com> wrote:
> > and fd 13 by closedumps() (the missing link to compinit?).
> 
> That's distinctly suspicious, if 13 was opened for the process
> substitution.
> 
> Does the following fix it, or is there yet more?
> 
> If it does we still have an error for older systems, but I suspect
> FD_CLOEXEC (and O_CLOEXEC) are widely enough supported that may be more
> a theoretical then an actual problem.
[...]

It's better:

$ eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
ls: cannot access /proc/self/fd/11: No such file or directory
lr-x------ 1 chazelas chazelas 64 Jul 20 21:23 /proc/self/fd/10 -> pipe:[1541173]
lr-x------ 1 chazelas chazelas 64 Jul 20 21:23 /proc/self/fd/12 -> pipe:[1541175]
lr-x------ 1 chazelas chazelas 64 Jul 20 21:23 /proc/self/fd/13 -> pipe:[1541176]

Only one now instead of two.

-- 
Stephane

Re: bug with eval, proc-subst and pipes

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 2539 bytes --]

Note, there are at least three messages in this thread that I've never
received ... they don't seem to be in my spam folder, either ...

On Saturday, July 20, 2013, Peter Stephenson wrote:

> On Fri, 19 Jul 2013 21:58:53 +0100
> Stephane Chazelas <stephane.chazelas@gmail.com <javascript:;>> wrote:
> > and fd 13 by closedumps() (the missing link to compinit?).
>
> That's distinctly suspicious, if 13 was opened for the process
> substitution.
>
> Does the following fix it, or is there yet more?


Something doesn't look right about this patch, but I've only looked at the
patch itself, not at the context (I'm away from my usual computers at the
moment).  The #ifndef lines seem to put both a functional and a no-op
version of closedumps() in the same not-FD_CLOEXEC context?  And if
closedumps() was a no-op with FD_CLOEXEC before (?) then would it change
anything to remove it at the point of call?

(Pardon the excessive quoting below.)


If it does we still have an error for older systems, but I suspect
> FD_CLOEXEC (and O_CLOEXEC) are widely enough supported that may be more
> a theoretical then an actual problem.
>
> diff --git a/Src/exec.c b/Src/exec.c
> index d462d97..f9efb3a 100644
> --- a/Src/exec.c
> +++ b/Src/exec.c
> @@ -448,7 +448,9 @@ zexecve(char *pth, char **argv, char **newenvp)
>      else
>         sprintf(buf + 2, "%s/%s", pwd, pth);
>      zputenv(buf);
> +#ifndef FD_CLOEXEC
>      closedumps();
> +#endif
>
>      if (newenvp == NULL)
>             newenvp = environ;
> diff --git a/Src/parse.c b/Src/parse.c
> index 753080d..b670925 100644
> --- a/Src/parse.c
> +++ b/Src/parse.c
> @@ -3171,6 +3171,9 @@ load_dump_file(char *dump, struct stat *sbuf, int
> other, int len)
>      d->dev = sbuf->st_dev;
>      d->ino = sbuf->st_ino;
>      d->fd = fd;
> +#ifdef FD_CLOEXEC
> +    fcntl(fd, F_SETFD, FD_CLOEXEC);
> +#endif
>      d->map = addr + (other ? (len - off) / sizeof(wordcode) : 0);
>      d->addr = addr;
>      d->len = len;
> @@ -3439,6 +3442,7 @@ decrdumpcount(FuncDump f)
>      }
>  }
>
> +#ifndef FD_CLOEXEC
>  /**/
>  mod_export void
>  closedumps(void)
> @@ -3448,6 +3452,7 @@ closedumps(void)
>      for (p = dumps; p; p = p->next)
>         zclose(p->fd);
>  }
> +#endif
>
>  #else
>
> @@ -3461,11 +3466,13 @@ decrdumpcount(FuncDump f)
>  {
>  }
>
> +#ifndef FD_CLOEXEC
>  /**/
>  mod_export void
>  closedumps(void)
>  {
>  }
> +#endif
>
>  #endif
>
> --
> Peter Stephenson <p.w.stephenson@ntlworld.com <javascript:;>>
> Web page now at http://homepage.ntlworld.com/p.w.stephenson/
>

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

Bart Schaefer wrote:
> Something doesn't look right about this patch, but I've only looked at the
> patch itself, not at the context (I'm away from my usual computers at the
> moment).  The #ifndef lines seem to put both a functional and a no-op
> version of closedumps() in the same not-FD_CLOEXEC context?  And if
> closedumps() was a no-op with FD_CLOEXEC before (?) then would it change
> anything to remove it at the point of call?

It's a no-op in the not-USE_MMAP case for dumps.  In that case there was
no file descriptor in the first place, so closedumps() wasn't needed.
It would have been more logical not to define it and mark its use as

#if defined(USE_MMAP) && !defined(FD_CLOEXEC)

but I had no reason to change the way the not-USE_MMAP case was
implemented, particularly since it's not the case that gets compiled on
my or most modern systems.

So it's a minor inconsistency with no effect.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Sat, 20 Jul 2013 21:25:31 +0100
Stephane Chazelas <stephane.chazelas@gmail.com> wrote:
> It's better:
> 
> $ eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
> ls: cannot access /proc/self/fd/11: No such file or directory
> lr-x------ 1 chazelas chazelas 64 Jul 20 21:23 /proc/self/fd/10 -> pipe:[1541173]
> lr-x------ 1 chazelas chazelas 64 Jul 20 21:23 /proc/self/fd/12 -> pipe:[1541175]
> lr-x------ 1 chazelas chazelas 64 Jul 20 21:23 /proc/self/fd/13 -> pipe:[1541176]
> 
> Only one now instead of two.

OK, getting somewhere...

In that case, your previous mail suggests zclose(subsh_close) is the
other thing that needs looking at.

It seems to me that subsh_close is liable to the same problem as I
(eventually) fixed in the example Bart came up with.  Any chance the
following patch resolves the issue?  (I checked and there shouldn't be
any work for Vincent this time round...)

One of the lines I removed,

-    if (subsh_close >= 0 && fdtable[subsh_close] == FDT_UNUSED)
-	subsh_close = -1;

suggests we might be on the right track here... at some point, someone
was obviously worried that subsh_close might already have been closed,
and if that's the case it's entirely possible that it was also reopened
for a different purpose, which would be what you're seeing.

Paranoid note: the traditional problem with associating new things with
jobs, as addfilelist() does, is you find out there isn't a job because
it's been optimised out.  However, I believe that by the time we've got
as far as execpline2() the case is sufficiently complicated it must have
a job.  (Ultra paranoid note: in which case we *must* ensure the job is
cleared up, right?  There's no possibility of a file descriptor leak?)
(If you understood that I will endorse your Unix experience on
LinkedIn...)

diff --git a/Src/exec.c b/Src/exec.c
index f9efb3a..1c44565 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -1653,8 +1653,6 @@ execpline(Estate state, wordcode slcode, int how, int last1)
     return lastval;
 }
 
-static int subsh_close = -1;
-
 /* execute pipeline.  This function assumes the `pline' is not NULL. */
 
 /**/
@@ -1731,7 +1729,7 @@ execpline2(Estate state, wordcode pcode,
 	    }
 	} else {
 	    /* otherwise just do the pipeline normally. */
-	    subsh_close = pipes[0];
+	    addfilelist(NULL, pipes[0]);
 	    execcmd(state, input, pipes[1], how, 0);
 	}
 	zclose(pipes[1]);
@@ -1744,8 +1742,6 @@ execpline2(Estate state, wordcode pcode,
 	execpline2(state, *state->pc++, how, pipes[0], output, last1);
 	list_pipe = old_list_pipe;
 	cmdpop();
-	zclose(pipes[0]);
-	subsh_close = -1;
     }
 }
 
@@ -2162,8 +2158,6 @@ addfd(int forked, int *save, struct multio **mfds, int fd1, int fd2, int rflag,
 	    mfds[fd1]->fds[mfds[fd1]->ct++] = fdN;
 	}
     }
-    if (subsh_close >= 0 && fdtable[subsh_close] == FDT_UNUSED)
-	subsh_close = -1;
 }
 
 /**/
@@ -3245,11 +3239,6 @@ execcmd(Estate state, int input, int output, int how, int last1)
 
 	    if (is_shfunc) {
 		/* It's a shell function */
-
-		if (subsh_close >= 0)
-		    zclose(subsh_close);
-		subsh_close = -1;
-
 		execshfunc((Shfunc) hn, args);
 	    } else {
 		/* It's a builtin */
@@ -3325,9 +3314,6 @@ execcmd(Estate state, int input, int output, int how, int last1)
 		DPUTS(varspc,
 		      "BUG: assignment before complex command");
 		list_pipe = 0;
-		if (subsh_close >= 0)
-		    zclose(subsh_close);
-                subsh_close = -1;
 		/* If we're forked (and we should be), no need to return */
 		DPUTS(last1 != 1 && !forked, "BUG: not exiting?");
 		DPUTS(type != WC_SUBSH, "Not sure what we're doing.");
@@ -5069,7 +5055,6 @@ execsave(void)
     es->trapisfunc = trapisfunc;
     es->traplocallevel = traplocallevel;
     es->noerrs = noerrs;
-    es->subsh_close = subsh_close;
     es->underscore = ztrdup(zunderscore);
     es->next = exstack;
     exstack = es;
@@ -5100,7 +5085,6 @@ execrestore(void)
     trapisfunc = exstack->trapisfunc;
     traplocallevel = exstack->traplocallevel;
     noerrs = exstack->noerrs;
-    subsh_close = exstack->subsh_close;
     setunderscore(exstack->underscore);
     zsfree(exstack->underscore);
     en = exstack->next;
diff --git a/Src/zsh.h b/Src/zsh.h
index ebd3cb7..d7b130c 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -977,7 +977,6 @@ struct execstack {
     int trapisfunc;
     int traplocallevel;
     int noerrs;
-    int subsh_close;
     char *underscore;
 };
 
-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: bug with eval, proc-subst and pipes

[Stephane Chazelas]

2013-07-20 23:55:01 +0100, Peter Stephenson:
[...]
> Any chance the following patch resolves the issue?  (I checked
> and there shouldn't be any work for Vincent this time
> round...)
[...]

Well, something I failed to mention as I had overlooked it
myself is that "ls" is a function from my .zshrc

ls() {
  if [[ -t 1 ]]; then
    command ls -F --color "$@"
  else
    command ls "$@"
  fi
}

Now, and it was already the case before that last patch,
I can't reproduce it with "paste" or "/bin/ls", only with that
"ls" function, so it's a different case from the initial issue I
was originally reporting (which seems to be now fixed with the
patch before this one).

To be clear, to reproduce this new one:

zsh -f
ls() command ls "$@"
eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat

(no need for compinit here)

And I can reproduce that one with Ubuntu 10.04 and 12.04 as well.

-- 
Stephane

Re: bug with eval, proc-subst and pipes

[Stephane Chazelas]

2013-07-21 10:45:25 +0100, Stephane Chazelas:
[...]
> To be clear, to reproduce this new one:
> 
> zsh -f
> ls() command ls "$@"
> eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
> 
> (no need for compinit here)
[...]

Sorry, I'm confused now. I can't reproduce it so I must have
screwed up somewhere. I'll start from scratch applying one patch
at a time to clarify things later, but for now, it seems
everything works fine with all the patches applied.


Sorry again for the confusion
Stephane

Re: bug with eval, proc-subst and pipes

[Stephane Chazelas]

2013-07-21 13:23:43 +0100, Stephane Chazelas:
> 2013-07-21 10:45:25 +0100, Stephane Chazelas:
> [...]
> > To be clear, to reproduce this new one:
> > 
> > zsh -f
> > ls() command ls "$@"
> > eval 'ls -l <(echo 1) <(echo 2) <(echo 3) <(echo 4)' | cat
> > 
> > (no need for compinit here)
> [...]
> 
> Sorry, I'm confused now. I can't reproduce it so I must have
> screwed up somewhere. I'll start from scratch applying one patch
> at a time to clarify things later, but for now, it seems
> everything works fine with all the patches applied.
[...]

OK, so if we number Peter's patches on that p1 to p4, my initial
issue (compinit;eval 'paste <(:) <(:) <:) <(:)'|cat) is fixed by
p1+p2+p3, while (x()ls $@; eval 'x <(:) <(:)' | cat) is fixed by
p1+p2+p3+p4.

-- 
Stephane

Re: bug with eval, proc-subst and pipes

[Peter Stephenson]

On Sat, 20 Jul 2013 18:11:44 +0100
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On Fri, 19 Jul 2013 21:58:53 +0100
> Stephane Chazelas <stephane.chazelas@gmail.com> wrote:
> > and fd 13 by closedumps() (the missing link to compinit?).
> 
> That's distinctly suspicious, if 13 was opened for the process
> substitution.
> 
> Does the following fix it, or is there yet more?

It did, apart from the other bit.

> If it does we still have an error for older systems, but I suspect
> FD_CLOEXEC (and O_CLOEXEC) are widely enough supported that may be more
> a theoretical then an actual problem.

This removes the side effects of the error in this case by properly
removing the dump record instead of just trimming it inconsistently, so
no one will try to use the closed file descriptor in the parent shell.

However, it presumably implies there are cases where the dump is being
removed where it doesn't need to be, which is the only way that could
trigger reuse of the file descriptor in the dump record.  The only
obvious way that could happen is a failed exec without a fork.  That's a
slightly strange to thing to happen in your main shell: it would mean
you're trying to replace it with something else but gave up and shrugged
your shoulders and carried on using it.  So there's presumably still
some aspect I'm missing.

diff --git a/Src/parse.c b/Src/parse.c
index b670925..0c2a458 100644
--- a/Src/parse.c
+++ b/Src/parse.c
@@ -3418,6 +3418,16 @@ incrdumpcount(FuncDump f)
     f->count++;
 }
 
+/**/
+static void
+freedump(FuncDump f)
+{
+    munmap((void *) f->addr, f->len);
+    zclose(f->fd);
+    zsfree(f->filename);
+    zfree(f, sizeof(*f));
+}
+
 /* Decrement the reference counter for a dump file. If zero, unmap the file. */
 
 /**/
@@ -3434,10 +3444,7 @@ decrdumpcount(FuncDump f)
 		q->next = p->next;
 	    else
 		dumps = p->next;
-	    munmap((void *) f->addr, f->len);
-	    zclose(f->fd);
-	    zsfree(f->filename);
-	    zfree(f, sizeof(*f));
+	    freedump(f);
 	}
     }
 }
@@ -3447,10 +3454,11 @@ decrdumpcount(FuncDump f)
 mod_export void
 closedumps(void)
 {
-    FuncDump p;
-
-    for (p = dumps; p; p = p->next)
-	zclose(p->fd);
+    while (dumps) {
+	FuncDump p = dumps->next;
+	freedump(dumps);
+	dumps = p;
+    }
 }
 #endif
 

SUGGESTION: Simplify auto-completion, a little

[mail]

Hi zsh folks,

kindly note my suggestion. Please smash or value it:

Now:    git co<tab>m<tab>
Future: git co<tab>m
Result: git commit

git co<tab>
commit         -- record changes to repository
commit-tree    -- create new commit object
config         -- get and set repository or global options
count-objects  -- count unpacked objects and display their disk
consumption

I hit ‘m<tab>’ to complete the expression to `git commit` (alternative
is <tab><tab>).
It would be sufficient just hitting the symbol ‘m’ to auto-complete to
`git commit`.
Why? The ‘m’ is in this instance the symbol which identifies ‘commit’
(excludes the other). There is no requirement for another hit on the
<tab> key because it already has been hit once.
This can be done with expressions of any length.

Regards

Benjamin

Re: SUGGESTION: Simplify auto-completion, a little

[Bart Schaefer]

On Feb 15,  9:45pm, mail@bwe.im wrote:
}
} Now:    git co<tab>m<tab>
} Future: git co<tab>m
} Result: git commit
} 
} It would be sufficient just hitting the symbol `m' to auto-complete to
} `git commit`.

Ordinary keybindings don't work that way; the "m" key is bound to the
self-insert widget and does not re-enter completion.  You should seek
out one of the various options for enabling "predictive typing", or
you should try setting up menu-selection (so that the first tab would
take you into a different keymap where "m" would select from among
the possible choices).

Re: SUGGESTION: Simplify auto-completion, a little

[Peter Stephenson]

Bart Schaefer wrote:
> On Feb 15,  9:45pm, mail@bwe.im wrote:
> }
> } Now:    git co<tab>m<tab>
> } Future: git co<tab>m
> } Result: git commit
> } 
> } It would be sufficient just hitting the symbol `m' to auto-complete to
> } `git commit`.
> 
> Ordinary keybindings don't work that way; the "m" key is bound to the
> self-insert widget and does not re-enter completion.  You should seek
> out one of the various options for enabling "predictive typing", or
> you should try setting up menu-selection (so that the first tab would
> take you into a different keymap where "m" would select from among
> the possible choices).

Hmm... I'm not sure how easy it is to do that at the moment.  There is a
keymap for menu-selection, but I think it's only got a limited set of
functions to do with the menu --- I don't think you can make a key
execute a widget that leaves completion with the command line as it is
*and* does normal (non-completion) ZLE stuff like self-insert and
re-enter completion, can you?

This implies a level of tweaking stuff within the completion function
environment it's not currently desiged to do.  It probably wouldn't be
beyond the wit of humankind to add an internal widget like
narrow-completion that could retrieve the list of completions and narrow
it down, but I bet it's still not trivial.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: SUGGESTION: Simplify auto-completion, a little

[Bart Schaefer]

On Feb 15, 11:29pm, Peter Stephenson wrote:
} Subject: Re: SUGGESTION: Simplify auto-completion, a little
}
} > you should try setting up menu-selection (so that the first tab would
} > take you into a different keymap where "m" would select from among
} > the possible choices).
} 
} Hmm... I'm not sure how easy it is to do that at the moment.  There is a
} keymap for menu-selection, but I think it's only got a limited set of
} functions to do with the menu

You're right.  I was thinking of something that incremental-complete-word
does, and attributing it to menu selection.

But even i-c-w's not quite as automatic as was requested.  Even if there
is only one completion, you have to confirm that it's correct by hitting
an extra tab.  Completion tends to assume that there's always a chance
you may want to enter something that's not on its list of choices.

I.e., that it's better to hit one extra tab than to have to backspace
over half an unwanted word.

However, it might be interesting to have incremental-complete-word accept
the current completion on space as well as tab (optionally, I suppose).

[PATCH] Fix loading of multi-line history entires from disk

[Augie Fackler]

Fixes the issue I posted about yesterday with
history-incremental-search-backward. I have no idea as to the
correctness of the fix, but my history file is written out in a way
that matches 4.3.x with this patch applied, and the behavior matches
again.

(Not subscribed to zsh-workers, so if you want me to see replies
promptly please keep me on the CC line. Thanks!)

commit 4b2e7d1047419648291701d80268629f63ad6208
Author: Augie Fackler <raf@durin42.com>
Date:   Wed Jul 16 09:35:22 2014 -0400

    hist.c: fix regression around parsing multi-line history entries

    History files touched by an afflicted zsh with EXTENDED_HISTORY set
    will be slightly corrupt: when zsh rewrites the mis-parsed history
    with the extended metadata at the start of each line, since it
    (wrongly) thought each line was a separate command.

diff --git a/Src/hist.c b/Src/hist.c
index 64f88f5..359e89d 100644
--- a/Src/hist.c
+++ b/Src/hist.c
@@ -2308,8 +2308,7 @@ readhistline(int start, char **bufp, int *bufsiz, FILE *in)
        }
        else {
            buf[len - 1] = '\0';
-           if (len > 1 && buf[len - 2] == '\\' &&
-               (len < 3 || buf[len - 3] != '\\')) {
+           if (len > 1 && buf[len - 2] == '\\') {
                buf[--len - 1] = '\n';
                if (!feof(in))
                    return readhistline(len, bufp, bufsiz, in);

Re: [PATCH] Fix loading of multi-line history entires from disk

[Bart Schaefer]

On Jul 17,  8:58am, Augie Fackler wrote:
} Subject: [PATCH] Fix loading of multi-line history entires from disk
}
} Fixes the issue I posted about yesterday with
} history-incremental-search-backward. I have no idea as to the
} correctness of the fix, but my history file is written out in a way
} that matches 4.3.x with this patch applied, and the behavior matches
} again.

We've got a hairy problem here.  See workers/30432 -- if you write a
line that ends in two backslashes, it is NOT rewritten on output, but
if you write a line than ends in one backslash then that IS doubled
on output ...

So either the first case is broken (with Augie's patch and/or prior to
workers/30443, it treats that line as continued when it should not) or
the second case is broken (it treats continuations as multiple lines).

The complication is that a line ending in THREE backslashes (or any
odd number of them) needs to go back to being treated as continuation.
In that instance there should be a newline embedded in the internal
history entry.

To sum up:

(1) All history files containing any history entries that intentionally
end with a double backslash, are broken, for as long as zsh has been
writing history files, because those lines are impossible to distinguish
from a continuation line where zsh doubled the backslash at output.

(2) Zsh versions 5.0.0 through 5.0.5 will incorrectly reload history that
contains continuation lines, because of an attempt to fix (1).  Versions
prior to 5.0.0 will incorrectly load files that have intentional double
backslashes, because of (1).

(3) Incorrectly reloaded history files will also have been incorrectly
written back by versions 5.0.0 through 5.0.5, in any circumstance that
saves in extended history format.

(4) The only reasonable solution is to fix (1) by changing the way the
entries are written, which means old history files will remain broken.
Attempting to fix it when loading the files was misguided (my error).

So, check out the following.  This includes Augie's patch (backs out
30433) and adds an attempt at changing the output (appends a space after
an even number of backslashes appearing at the end of an entry).

An alternative would be to append a semicolon there.  Opinions?  Is this
approach also misguided, and if so, what do we do?


diff --git a/Src/hist.c b/Src/hist.c
index 64f88f5..770d559 100644
--- a/Src/hist.c
+++ b/Src/hist.c
@@ -2308,8 +2308,7 @@ readhistline(int start, char **bufp, int *bufsiz, FILE *in)
 	}
 	else {
 	    buf[len - 1] = '\0';
-	    if (len > 1 && buf[len - 2] == '\\' &&
-		(len < 3 || buf[len - 3] != '\\')) {
+	    if (len > 1 && buf[len - 2] == '\\') {
 		buf[--len - 1] = '\n';
 		if (!feof(in))
 		    return readhistline(len, bufp, bufsiz, in);
@@ -2618,6 +2617,8 @@ savehistfile(char *fn, int err, int writeflags)
 
 	ret = 0;
 	for (; he && he->histnum <= xcurhist; he = down_histent(he)) {
+	    int count_backslashes = 0;
+
 	    if ((writeflags & HFILE_SKIPDUPS && he->node.flags & HIST_DUP)
 	     || (writeflags & HFILE_SKIPFOREIGN && he->node.flags & HIST_FOREIGN)
 	     || he->node.flags & HIST_TMPSTORE)
@@ -2649,9 +2650,18 @@ savehistfile(char *fn, int err, int writeflags)
 		if (*t == '\n')
 		    if ((ret = fputc('\\', out)) < 0)
 			break;
+		if (*t == '\\')
+		    count_backslashes++;
+		else
+		    count_backslashes = 0;
 		if ((ret = fputc(*t, out)) < 0)
 		    break;
 	    }
+	    if (ret < 0)
+	    	break;
+	    if (count_backslashes && (count_backslashes % 2 == 0))
+		if ((ret = fputc(' ', out)) < 0)
+		    break;
 	    if (ret < 0 || (ret = fputc('\n', out)) < 0)
 		break;
 	}

Re: [PATCH] Fix loading of multi-line history entires from disk

[Peter Stephenson]

On Thu, 17 Jul 2014 08:58:41 -0400
Augie Fackler <raf@durin42.com> wrote:
> Fixes the issue I posted about yesterday with
> history-incremental-search-backward. I have no idea as to the
> correctness of the fix, but my history file is written out in a way
> that matches 4.3.x with this patch applied, and the behavior matches
> again.

You're right, it does work in both cases.

The change that introduced this was to fix lines ending in two
backslashes.  That presumably needs something more sophisticated.  But
I'm not sure what --- the traditional doubling of backslashes to
indicate a continuation line isn't syntactically unique, so I don't see
how you can tell, short of checking for an extendedhistory introduction
on the next line if that option's set, and obviously there's no
guarantee it is.

Specifically, for a continuation line

% echo foo \
> bar \
> rod

I get

: 1405625973:0;echo foo \\
bar \\
rod

and for three separate lines

% echo foo \\
% echo bar \\
% echo rod \\

I get

: 1405626360:0;echo foo \\
: 1405626364:0;echo bar \\
: 1405626453:0;echo rod \\

So I think we're stuck without breaking backward compatibility.

Doubling all backslashes and looking for an odd number at the end would
have worked better.

pws

Re: [PATCH] Fix loading of multi-line history entires from disk

[Peter Stephenson]

Bart Schaefer wrote:
> So, check out the following.  This includes Augie's patch (backs out
> 30433) and adds an attempt at changing the output (appends a space after
> an even number of backslashes appearing at the end of an entry).

That seems as minimal and compatible as we're going to get.

Is it worth appending something syntactically null but a little more
complicated that we can strip on input with little likelihood of
stepping on anyone's toes?  " ; "?   Unless we do attempt that, appending
a simple space is as good as anything.

pws

Re: [PATCH] Fix loading of multi-line history entires from disk

[Augie Fackler]

On Thu, Jul 17, 2014 at 08:58:45PM +0100, Peter Stephenson wrote:
> Bart Schaefer wrote:
> > So, check out the following.  This includes Augie's patch (backs out
> > 30433) and adds an attempt at changing the output (appends a space after
> > an even number of backslashes appearing at the end of an entry).
>
> That seems as minimal and compatible as we're going to get.
>
> Is it worth appending something syntactically null but a little more
> complicated that we can strip on input with little likelihood of
> stepping on anyone's toes?  " ; "?   Unless we do attempt that, appending
> a simple space is as good as anything.

Makes sense. Thanks to both for the thorough explanation of the
problem. A space sounds reasonable, as does " ; ". Thanks for the
quick responses!

>
> pws

Re: [PATCH] Fix loading of multi-line history entires from disk

[Bart Schaefer]

On Jul 17,  8:58pm, Peter Stephenson wrote:
} Subject: Re: [PATCH] Fix loading of multi-line history entires from disk
}
} Bart Schaefer wrote:
} > So, check out the following.  This includes Augie's patch (backs out
} > 30433) and adds an attempt at changing the output (appends a space after
} > an even number of backslashes appearing at the end of an entry).
} 
} That seems as minimal and compatible as we're going to get.

OK, I'm going to go ahead and push 32882 while we discuss alternate
end-of-entry markers.

} Is it worth appending something syntactically null but a little more
} complicated that we can strip on input with little likelihood of
} stepping on anyone's toes?  " ; "?   Unless we do attempt that, appending
} a simple space is as good as anything.

We could follow the examples of extended-history and _complete_debug to
use something that begins with ";:".

E.g. we could do something like ";: ENDS_WITH_BACKSLASH" but that's not
really "syntactically null" though it is semantically null.

Re: [PATCH] Fix loading of multi-line history entires from disk

[Bart Schaefer]

On Jul 17,  8:08pm, Bart Schaefer wrote:
} Subject: Re: [PATCH] Fix loading of multi-line history entires from disk
}
} On Jul 17,  8:58pm, Peter Stephenson wrote:
} } Subject: Re: [PATCH] Fix loading of multi-line history entires from disk
} }
} } Bart Schaefer wrote:
} } > So, check out the following.  This includes Augie's patch (backs out
} } > 30433) and adds an attempt at changing the output (appends a space after
} } > an even number of backslashes appearing at the end of an entry).
} } 
} } That seems as minimal and compatible as we're going to get.
} 
} OK, I'm going to go ahead and push 32882 while we discuss alternate
} end-of-entry markers.

It occurs to me that perhaps there's no point in distinguishing an even
number of backslashes -- if the entire history event ends with a
backslash, it doesn't matter whether there are an even number of them.

On the other hand, following an odd number of backslashes with a space
or semicolon wouldn't have the right effect either.

Is it even possible for a history event to end with an odd number of
backslashes?

Re: [PATCH] Fix loading of multi-line history entires from disk

[Peter Stephenson]

On Thu, 17 Jul 2014 23:01:09 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> It occurs to me that perhaps there's no point in distinguishing an even
> number of backslashes -- if the entire history event ends with a
> backslash, it doesn't matter whether there are an even number of them.
> 
> On the other hand, following an odd number of backslashes with a space
> or semicolon wouldn't have the right effect either.
> 
> Is it even possible for a history event to end with an odd number of
> backslashes?

I can't think of one that forms a complete line.  You could get it by
writing something explicitly of that form to the history.  Given zsh's
flexibility, there's no law saying people can't save partial lines to
the history for subsequent recall and editing.  This would be a very
special case, though.

pws

Re: [PATCH] Fix loading of multi-line history entires from disk

[Peter Stephenson]

On Thu, 17 Jul 2014 20:08:24 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> } Is it worth appending something syntactically null but a little more
> } complicated that we can strip on input with little likelihood of
> } stepping on anyone's toes?  " ; "?   Unless we do attempt that, appending
> } a simple space is as good as anything.
> 
> We could follow the examples of extended-history and _complete_debug to
> use something that begins with ";:".
> 
> E.g. we could do something like ";: ENDS_WITH_BACKSLASH" but that's not
> really "syntactically null" though it is semantically null.

I think we need something that works without extended-history on,
though, and even though we can still read it in specially if it's like
that it kind of violates the spirit.

Possible bug to RPROMPT

[Felipe G. Silveira]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

Hello,

I've been tinkering with zsh for the past couple weeks, and I'm loving it!

But I think I hit a bug regarding the RPROMPT definition.

Since I setup my PROMPT so the information comes in one line and my cursor
below it, I was trying to align the RPROMPT to always appear 1 line before
where it generally is printed.

This is a bit confuse, but hopefully everything will clear up with an
example. So, lets suppose I have:

PROMPT='a lot of cosmetic and information stuff here.
{colors} > {reset colors}'

So I have a 2 line definition. this works fine.

so I tried

RPROMPT='< [current time]'

this works, BUT it's one line below form where I want it.

so after some tinkering, I decided to try some escape commands:

PROMPT=$'\e[s'$'\e[1A''< [clock] ''$'\e[1B'$'\e[u'

Where:
$'\e[s' - Save current cursor position
$'\e[1A' - go back one line up
<<render stuff between them >>
$'\e[1B' - go one line down
$'\e[u' - Restore cursor position

It worked! But... The cursor ends up on the far left side of the screen,
before any characters that are already printed.  (so, in my example, it
would be before the > printed by PROMPT).

So.. is this a real (possible?) bug, or did I "abuse" too much from the
prompt parameters and broke it ?

Hopefully it's something that can be fixed!

Thank you in advance!

-Felipe

Re: Possible bug to RPROMPT

[Bart Schaefer]

On Aug 1,  5:20am, Felipe G. Silveira wrote:
}
} Since I setup my PROMPT so the information comes in one line and my cursor
} below it, I was trying to align the RPROMPT to always appear 1 line before
} where it generally is printed.

If I understand correctly, what you want is:

First_line_of_PROMPT_here...................................RPROMPT_here
Second_line.cursor

where I have used dots in place of whitespace in case of line wrap.

} so after some tinkering, I decided to try some escape commands:
} 
} PROMPT=$'\e[s'$'\e[1A''< [clock] ''$'\e[1B'$'\e[u'

You have the right general idea (though there are too many quotes in
that line) but have overlooked the %{ and %} prompt formats, which help
ZLE to compute the start and end positions of the prompt text.  If you
move the cursor with terminal escapes while inside the prompt, you must
wrap the "zero width" parts in %{...%} so ZLE won't count them when it
calculates the number of columns occupied by the prompt.

In the case of your specific desire, you can probably get away with
using only vertical motions (no save/restore of cursor) like so:

PROMPT="First_line_of_PROMPT_here"$'\n'"Second_line "
RPROMPT=$'%{\e[1A%}'"RPROMPT_here"$'%{\e[1B%}'

This just says that the right prompt begins with a zero-width motion,
which happens to be "up one line", and ends with another zero-width
motion, "down one line".  Because ZLE only cares about the horizontal
position for its calculations, as long as you make sure the cursor
starts and ends on the same line it all works out.

Note, though, that if "First_line_of_PROMPT_here" is wide enough to
overlap with "RPROMPT_here", the RPROMPT will write over it.  Also,
the TRANSIENT_RPROMPT option will have no effect because the text is
one line above where ZLE expects it to be.

For a more complicated example of a prompt that looks like what you
want but does not use RPROMPT to accomplish it, read the "Prompt Themes"
section of "man zshcontrib" and then try "prompt -h bart".

Re: Possible bug to RPROMPT

[Peter Stephenson]

Bart Schaefer wrote:
> For a more complicated example of a prompt that looks like what you
> want but does not use RPROMPT to accomplish it, read the "Prompt Themes"
> section of "man zshcontrib" and then try "prompt -h bart".

I can't help feeling it should be easier than that...

We could do with something that works a bit like \fil so that you can
divide up the remaining width in proportion.  That's probably not so
difficult with a prompt escape.

\bye

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Possible bug to RPROMPT

[Bart Schaefer]

On Aug 4,  8:50pm, Peter Stephenson wrote:
} Subject: Re: Possible bug to RPROMPT
}
} Bart Schaefer wrote:
} > For a more complicated example of a prompt that looks like what you
} > want but does not use RPROMPT to accomplish it, read the "Prompt Themes"
} > section of "man zshcontrib" and then try "prompt -h bart".
} 
} I can't help feeling it should be easier than that...

The really complicated bits are removing the upper right prompt as the
upper left prompt gets longer, and moving the %~ expansion to a third
line if things are still "too long".

} We could do with something that works a bit like \fil so that you can
} divide up the remaining width in proportion.

That'd be interesting, but what's really needed is for %<...< and %>...>
to have a truncation length that's relative to the number of columns
remaining in the line.  Hmm, perhaps %12<...< means truncate at 12
characters, and %-12<...< means to truncate at the remaining length of
the current line minus twelve characters?

Re: Possible bug to RPROMPT

[Bart Schaefer]

On Aug 4,  1:50pm, Bart Schaefer wrote:
}
} That'd be interesting, but what's really needed is for %<...< and %>...>
} to have a truncation length that's relative to the number of columns
} remaining in the line.  Hmm, perhaps %12<...< means truncate at 12
} characters, and %-12<...< means to truncate at the remaining length of
} the current line minus twelve characters?

So, this works (patch below).  You can do for example

PS1="%-40<<$PS1"

and then you always have 40 characters in which to type.  It belatedly
occurs to me that you can *already* get this with

setopt promptsubst
PS1='%$((COLUMNS-40))<<'"$PS1"

but this seems a worthwhile shorthand and doesn't require promptsubst.

On the other hand, if $COLUMNS is less than 40 then the patch below
changes the behavior of the promptsubst variant for very narrow windows.

It's still tricky to get the leftmost part of the prompt to shrink from
the midde, because "remaining width" is always calculated left-to-right.

In the patch, I hav enot applied this to the older %[<...] syntax.  The
second hunks is for something I believe to be a bug, if anyone would
care to confirm or deny.

diff --git a/Src/prompt.c b/Src/prompt.c
index c16d781..c6624cc 100644
--- a/Src/prompt.c
+++ b/Src/prompt.c
@@ -560,6 +560,12 @@ putpromptchar(int doprint, int endchar, unsigned int *txtchangep)
 		break;
 	    case '<':
 	    case '>':
+		/* Test (minus) here so -0 means "at the right margin" */
+		if (minus) {
+		    *bv->bp = '\0';
+		    countprompt(bv->bufline, &t0, 0, 0);
+		    arg = zterm_columns - t0 + arg;
+		}
 		if (!prompttrunc(arg, *bv->fm, doprint, endchar, txtchangep))
 		    return *bv->fm;
 		break;
@@ -1475,7 +1481,7 @@ prompttrunc(int arg, int truncchar, int doprint, int endchar,
 	/* Now we have to trick it into matching endchar again */
 	bv->fm--;
     } else {
-	if (*bv->fm != ']')
+	if (*bv->fm != endchar)
 	    bv->fm++;
 	while(*bv->fm && *bv->fm != truncchar) {
 	    if (*bv->fm == '\\' && bv->fm[1])

-- 
Barton E. Schaefer

Re: Possible bug to RPROMPT

[Bart Schaefer]

On Aug 5, 12:54am, Bart Schaefer wrote:
} Subject: Re: Possible bug to RPROMPT
}
} PS1="%-40<<$PS1"
} 
} and then you always have 40 characters in which to type.  It belatedly
} occurs to me that you can *already* get this with
} 
} setopt promptsubst
} PS1='%$((COLUMNS-40))<<'"$PS1"

I should have pointed out that these aren't fully equivalent unless they
appear at the "left end" of the prompt, because of course the first one
truncates more of the prompt the farther to the right it appears, whereas
the second always truncates the same.

However, there's some odd behavior when the computed result is actually
zero or less; it's then treated as %<< (i.e., as if with no numeric
argument), and no truncation occurs at all.  That's very unintuitive,
but it stems from being able to use %<< as an end marker for a preceding
truncation.

Merely documenting this doesn't seem sufficient.  On the other hand, a
similar odd thing happens with positive truncation because of implicit
line wrapping; try %90>=> followed by 100+ characters on an 80-column
terminal to see what I mean.

[PATCH] Better prompt width calculations (Re: Possible bug to RPROMPT)

[Bart Schaefer]

On Aug 6, 12:57am, Bart Schaefer wrote:
}
} However, there's some odd behavior when the computed result is actually
} zero or less; it's then treated as %<< (i.e., as if with no numeric
} argument), and no truncation occurs at all.  That's very unintuitive,
} but it stems from being able to use %<< as an end marker for a preceding
} truncation.
} 
} Merely documenting this doesn't seem sufficient.  On the other hand, a
} similar odd thing happens with positive truncation because of implicit
} line wrapping; try %90>=> followed by 100+ characters on an 80-column
} terminal to see what I mean.

Here is a revised patch, including documentation, which I think makes
this fairly useful and more importantly comprehensible.  I've made two
changes:

First, a negative argument with the 'l' conditional counts the number
of characters remaining instead of the number of characters printed:

	%40(l.MORE THAN.LESS THAN) FORTY characters printed
	%-40(l.MORE THAN.LESS THAN) FORTY characters remain

Previously the absolute value of the argument was always used here
(and still is, for all other ternary conditionals).

Second, as in the previous patch a negative argument with truncation
leaves at least N characters unused to the right of the truncated
portion.  However, I've now made the minimum width 1 character, so
that we don't run into the special meaning of zero (or less).

Finally, I also fixed a couple of comments that got messed up by the
global-text-replace of "buf" with "bv->buf" several years ago.


diff --git a/Doc/Zsh/prompt.yo b/Doc/Zsh/prompt.yo
index 36f351b..963207f 100644
--- a/Doc/Zsh/prompt.yo
+++ b/Doc/Zsh/prompt.yo
@@ -302,7 +302,8 @@ sitem(tt(g))(True if the effective gid of the current process is var(n).)
 sitem(tt(j))(True if the number of jobs is at least var(n).)
 sitem(tt(L))(True if the tt(SHLVL) parameter is at least var(n).)
 sitem(tt(l))(True if at least var(n) characters have already been
-printed on the current line.)
+printed on the current line.  When var(n) is negative, true if at least
+var(n) characters remain before the right margin.)
 sitem(tt(S))(True if the tt(SECONDS) parameter is at least var(n).)
 sitem(tt(T))(True if the time in hours is equal to var(n).)
 sitem(tt(t))(True if the time in minutes is equal to var(n).)
@@ -318,13 +319,21 @@ item(tt(%[)var(xstring)tt(]))(
 Specifies truncation behaviour for the remainder of the prompt string.
 The third, deprecated, form is equivalent to `tt(%)var(xstringx)',
 i.e. var(x) may be `tt(<)' or `tt(>)'.
-The numeric argument, which in the third form may appear immediately
-after the `tt([)', specifies the maximum permitted length of
-the various strings that can be displayed in the prompt.
 The var(string) will be displayed in
 place of the truncated portion of any string; note this does not
 undergo prompt expansion.
 
+The numeric argument, which in the third form may appear immediately
+after the `tt([)', specifies the maximum permitted length of
+the various strings that can be displayed in the prompt.
+In the first two forms, this numeric argument may be negative, in which
+case the truncation length is determined by subtracting the absolute
+value of the numeric argument from the number of character positions
+remaining on the current prompt line.  If this results in a zero or
+negative length, a length of 1 is used.  In other words, a negative
+argument arranges that after truncation at least var(n) characters
+remain before the right margin.
+
 The forms with `tt(<)' truncate at the left of the string,
 and the forms with `tt(>)' truncate at the right of the string.
 For example, if the current directory is `tt(/home/pike)',
@@ -344,11 +353,19 @@ string, or to the end of the next enclosing group of the `tt(%LPAR())'
 construct, or to the next truncation encountered at the same grouping
 level (i.e. truncations inside a `tt(%LPAR())' are separate), which
 ever comes first.  In particular, a truncation with argument zero
-(e.g. `tt(%<<)') marks the end of the range of the string to be
+(e.g., `tt(%<<)') marks the end of the range of the string to be
 truncated while turning off truncation from there on. For example, the
 prompt '%10<...<%~%<<%# ' will print a truncated representation of the
 current directory, followed by a `tt(%)' or `tt(#)', followed by a
 space.  Without the `tt(%<<)', those two characters would be included
-in the string to be truncated.
+in the string to be truncated.  Note that `tt(%-0<<)' is a distinct
+
+Truncation applies only within each individual line of the prompt, as
+delimited by embedded newlines (if any).  If the total length of any line
+of the prompt after truncation is greater than the terminal width, or if
+the part to be truncated contains embedded newlines, truncation behavior
+is undefined and may change in a future version of the shell.  Use
+`tt(%-var(n)LPAR()l.var(true-text).var(false-text)RPAR())' to remove parts
+of the prompt when the available space is less than var(n).
 )
 enditem()
diff --git a/Src/prompt.c b/Src/prompt.c
index c16d781..328ae3c 100644
--- a/Src/prompt.c
+++ b/Src/prompt.c
@@ -367,6 +367,8 @@ putpromptchar(int doprint, int endchar, unsigned int *txtchangep)
 		case 'l':
 		    *bv->bp = '\0';
 		    countprompt(bv->bufline, &t0, 0, 0);
+		    if (minus)
+			t0 = zterm_columns - t0;
 		    if (t0 >= arg)
 			test = 1;
 		    break;
@@ -560,6 +562,14 @@ putpromptchar(int doprint, int endchar, unsigned int *txtchangep)
 		break;
 	    case '<':
 	    case '>':
+		/* Test (minus) here so -0 means "at the right margin" */
+		if (minus) {
+		    *bv->bp = '\0';
+		    countprompt(bv->bufline, &t0, 0, 0);
+		    arg = zterm_columns - t0 + arg;
+		    if (arg <= 0)
+			arg = 1;
+		}
 		if (!prompttrunc(arg, *bv->fm, doprint, endchar, txtchangep))
 		    return *bv->fm;
 		break;
@@ -1174,7 +1184,7 @@ prompttrunc(int arg, int truncchar, int doprint, int endchar,
 	    addbufspc(1);
 	    *bv->bp++ = '<';
 	}
-	ptr = bv->buf + w;		/* addbv->bufspc() may have realloc()'d bv->buf */
+	ptr = bv->buf + w;	/* addbufspc() may have realloc()'d bv->buf */
 	/*
 	 * Now:
 	 *   bv->buf is the start of the output prompt buffer
@@ -1189,7 +1199,7 @@ prompttrunc(int arg, int truncchar, int doprint, int endchar,
 	bv->trunccount = bv->dontcount;
 	putpromptchar(doprint, endchar, txtchangep);
 	bv->trunccount = 0;
-	ptr = bv->buf + w;		/* putpromptchar() may have realloc()'d */
+	ptr = bv->buf + w;	/* putpromptchar() may have realloc()'d */
 	*bv->bp = '\0';
 	/*
 	 * Now:
@@ -1475,7 +1485,7 @@ prompttrunc(int arg, int truncchar, int doprint, int endchar,
 	/* Now we have to trick it into matching endchar again */
 	bv->fm--;
     } else {
-	if (*bv->fm != ']')
+	if (*bv->fm != endchar)
 	    bv->fm++;
 	while(*bv->fm && *bv->fm != truncchar) {
 	    if (*bv->fm == '\\' && bv->fm[1])

Re: [PATCH] Better prompt width calculations (Re: Possible bug to RPROMPT)

[Jun T.]

Sorry for replying to a half-year old patch.

2014/08/07 17:36, Bart Schaefer <schaefer@brasslantern.com> wrote:

> diff --git a/Doc/Zsh/prompt.yo b/Doc/Zsh/prompt.yo
> index 36f351b..963207f 100644
> --- a/Doc/Zsh/prompt.yo
> +++ b/Doc/Zsh/prompt.yo
...
> @@ -344,11 +353,19 @@ string, or to the end of the next enclosing group of the `tt(%LPAR())'
> construct, or to the next truncation encountered at the same grouping
...
> 
> current directory, followed by a `tt(%)' or `tt(#)', followed by a
> space.  Without the `tt(%<<)', those two characters would be included
> -in the string to be truncated.
> +in the string to be truncated.  Note that `tt(%-0<<)' is a distinct
> +
> +Truncation applies only within each individual line of the prompt, as

The statement
  ".... is a distinct"
seems to be "truncated". Maybe something like the following?
  "... is not equivalent to `tt(%<<)' but specifies that the prompt is
  truncated at the right margin."

Re: [PATCH] Better prompt width calculations (Re: Possible bug to RPROMPT)

[Bart Schaefer]

On Apr 3, 12:46am, Jun T. wrote:
}
} > -in the string to be truncated.
} > +in the string to be truncated.  Note that `tt(%-0<<)' is a distinct
} > +
} 
} The statement
}   ".... is a distinct"
} seems to be "truncated". Maybe something like the following?
}   "... is not equivalent to `tt(%<<)' but specifies that the prompt is
}   truncated at the right margin."

Yes, that must have been where I was going with that.  Feel free to make
the edit.  Thanks.

Possibly excessive WARN_CREATE_GLOBAL

[Bart Schaefer]

It appears that none of the currently-existing modules create non-special
autoloaded parameters, so the setsparam() and setaparam() calls in module.c
do not presently cause any issues.

However, do we want to be complaining about REPLY, reply, match, mbegin,
mend, MBEGIN, MEND, and BASH_REMATCH?

Re: Possibly excessive WARN_CREATE_GLOBAL

[Peter Stephenson]

On Sun, 22 Nov 2015 09:52:12 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> However, do we want to be complaining about REPLY, reply, match, mbegin,
> mend, MBEGIN, MEND, and BASH_REMATCH?
 
Yes, those are exactly the ones where it's important we do --- in fact,
match etc. were in my mind when I introduced the option, although at the
time the warning didn't actually cover the usual cases of that.  With
most variables in functions, leaking isn't that important; you just get
a randomly named variable in the outer scope.  Those variables, on the
other hand, are the ones most likely to be used in nested functions and
in multiple ways within the same higher level function.  For example,
you match a pattern (setting "match" etc.), call another function that
also uses the feature, then try to use $match in the outer function ---
which is now misset.

(Unfortunately, in this example we don't actually get a warning for the
inner scope if the outer function had already localised the variable
(and that outer use is the only one that would be reported if it didn't
do so); however, if the inner function got used in another context, we
would get the warning there.  But I don't think that's relevant to the
basic point.  You could argue for a warning "variable created in outer
scope", I suppose.)

REPLY/reply are complicated by the fact you need to localise them only in
an outer scope, but I'd argue that it's therefore even easier to forget so
a warning is even more useful.

If you're not worried about *those* variables polluting outer scope,
therefore, it seems to me unlikely you're interested in the option at
all.

If we need to make exceptions, I think they're more likely to be connected
to specific instances of setting variables internally.

pws

Re: Possibly excessive WARN_CREATE_GLOBAL

[Bart Schaefer]

On Nov 22,  6:27pm, Peter Stephenson wrote:
} Subject: Re: Possibly excessive WARN_CREATE_GLOBAL
}
} On Sun, 22 Nov 2015 09:52:12 -0800
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > However, do we want to be complaining about REPLY, reply, match, mbegin,
} > mend, MBEGIN, MEND, and BASH_REMATCH?
}  
} Yes, those are exactly the ones where it's important we do --- in fact,
} match etc. were in my mind when I introduced the option [...]
} 
} (Unfortunately, in this example we don't actually get a warning for the
} inner scope if the outer function had already localised the variable
} [...].  You could argue for a warning "variable created in outer
} scope", I suppose.)

Hmm.

No time to fix it now, but declaring something "private" suppresses the
warncreateglobal warning without allowing the variable to become set in
the inner scope.  (What SHOULD happen here?  An error?)

torch% zmodload zsh/param/private
torch% setopt extendedglob warncreateglobal
torch% () {
 local -P match
 () {
  [[ abc = (#b)*(?)* ]]
  typeset -p match mbegin mend
 }
 print ${(t)match}
}
(anon):1: array parameter mbegin created globally in function (anon)
(anon):1: array parameter mend created globally in function (anon)
(anon):typeset:2: no such variable: match
typeset -a mbegin
mbegin=( 3 )
typeset -a mend
mend=( 3 )
scalar-local-hide-hideval-special

Re: Possibly excessive WARN_CREATE_GLOBAL

[Peter Stephenson]

Bart Schaefer wrote:
> No time to fix it now, but declaring something "private" suppresses the
> warncreateglobal warning without allowing the variable to become set in
> the inner scope.  (What SHOULD happen here?  An error?)

Do you mean, it gets created in the inner scope as if it were global but
gets wiped out in the higher scope?  That would be functionally OK in
the particular case but dodgy as far as the definition of the inner
function is concerned.  It might be more weight for an extended warning
"created without localisation in some inner scope even though it doesn't
actually propagate back to the top level", except shorter.

Or do you mean it doesn't actually get set *at all*?  That should surely
trigger an error immediately in the inner scope, shouldn't it?

pws

Re: Possibly excessive WARN_CREATE_GLOBAL

[Bart Schaefer]

On Nov 22,  7:15pm, Peter Stephenson wrote:
}
} Or do you mean it doesn't actually get set *at all*?  That should surely
} trigger an error immediately in the inner scope, shouldn't it?

It doesn't get set at all.

We come into createparam() and discover at line 878 that the parameter
already exists, which is what we want, but in spite of it being flagged
PM_SPECIAL|PM_REMOVABLE all createparam() does is toggle off the PM_UNSET
flag, which seems dubious.

(And *then* it tests PM_RESTRICTED, which it seems to me ought to come
first?  That's not related to this thread, but I wouldn't think you
should be able to change the unset-ness of a restricted parameter.)

Then we enter fetchvalue() at line 2926, which correctly returns that the
parameter does not have a value, and assignaparam() silently surrenders at
line 2929.  Maybe there should always be an error at that point, I don't
know how else we might get there.

Re: Possibly excessive WARN_CREATE_GLOBAL

[Peter Stephenson]

On Sun, 22 Nov 2015 12:30:28 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Nov 22,  7:15pm, Peter Stephenson wrote:
> }
> } Or do you mean it doesn't actually get set *at all*?  That should surely
> } trigger an error immediately in the inner scope, shouldn't it?
> 
> It doesn't get set at all.
> 
> We come into createparam() and discover at line 878 that the parameter
> already exists, which is what we want, but in spite of it being flagged
> PM_SPECIAL|PM_REMOVABLE all createparam() does is toggle off the PM_UNSET
> flag, which seems dubious.

Being special and removable shouldn't stop you re-enabling the parameter
--- this test has nothing to do with scope here, it's just recreating
the parameter as you've asked it to do.  (It's not checking PM_HIDE here,
or you might think that was useful.)

Perhaps what it should be doing is creating a new one in the inner
scope that hides the private one as the least bad effect.

Or, possibly in this case, it should be an error that tells you need to
use "typeset" to override a private variable in a local scope.  So this
is a stronger and more special version of the option, but depending on
the fact you've already opted in to new syntax by loading the private
module.  That might be good enough, but for top marks we'd probably need
to be surer that's what's really happened.

> (And *then* it tests PM_RESTRICTED, which it seems to me ought to come
> first?  That's not related to this thread, but I wouldn't think you
> should be able to change the unset-ness of a restricted parameter.)

Quite possibly.

> Then we enter fetchvalue() at line 2926, which correctly returns that the
> parameter does not have a value, and assignaparam() silently surrenders at
> line 2929.  Maybe there should always be an error at that point, I don't
> know how else we might get there.

Certainly you've asked it to assign a value and it hasn't, which seems
like an error.  The assumption may previously have been there was
already an error.

pws

Re: Possibly excessive WARN_CREATE_GLOBAL

[Bart Schaefer]

On Nov 23,  3:49pm, Peter Stephenson wrote:
} Subject: Re: Possibly excessive WARN_CREATE_GLOBAL
}
} On Sun, 22 Nov 2015 12:30:28 -0800
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > We come into createparam() and discover at line 878 that the parameter
} > already exists, which is what we want, but in spite of it being flagged
} > PM_SPECIAL|PM_REMOVABLE all createparam() does is toggle off the PM_UNSET
} > flag, which seems dubious.
} 
} Being special and removable shouldn't stop you re-enabling the parameter
} --- this test has nothing to do with scope here, it's just recreating
} the parameter as you've asked it to do.  (It's not checking PM_HIDE here,
} or you might think that was useful.)

It took me a while to figure out how to explain why I think this is
dubious, but finally decided it comes down to this:  Toggling PM_UNSET
off here without calling the GSU setfn implicitly provides a default
value for the parameter, which is sort of OK for normal parameters but
wrong for PM_SPECIAL.  If nothing else, it means the getfn has to be
prepared to return something even if the setfn has never been called.

(Of course theoretically the setfn was called with NULL before the
PM_UNSET flag was set in the first place, but also theoretically the
setfn could make use of the state of PM_UNSET at the time of call.)

Unfortunately none of the standard setfn do anything with PM_UNSET;
they all rely on it having been cleared by createparam(), so a change
there could have far-reaching effects.  Not something to do while a
release is imminent.

} Perhaps what it should be doing is creating a new one in the inner
} scope that hides the private one as the least bad effect.

What should happen is that it creates a new one at global scope, and
trips warncreateglobal if necessary.  Which DOES happen if the one at
global scope was previously assigned and/or explicitly declared.  The
difficulty only arises when the private parameter is at the outermost
scope where any parameter of that name can be found.

} Or, possibly in this case, it should be an error that tells you need to
} use "typeset" to override a private variable in a local scope.

It might be possible to pull this off, but probably not until it's too
late for the PM_UNSET flag to have been (in)correctly handled.
 

Re: Possibly excessive WARN_CREATE_GLOBAL

[Peter Stephenson]

On Mon, 23 Nov 2015 20:43:51 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:

 On Nov 23,  3:49pm, Peter Stephenson wrote:
> } Subject: Re: Possibly excessive WARN_CREATE_GLOBAL
> } Being special and removable shouldn't stop you re-enabling the parameter
> } --- this test has nothing to do with scope here, it's just recreating
> } the parameter as you've asked it to do.  (It's not checking PM_HIDE here,
> } or you might think that was useful.)
> 
> It took me a while to figure out how to explain why I think this is
> dubious, but finally decided it comes down to this:  Toggling PM_UNSET
> off here without calling the GSU setfn implicitly provides a default
> value for the parameter, which is sort of OK for normal parameters but
> wrong for PM_SPECIAL.  If nothing else, it means the getfn has to be
> prepared to return something even if the setfn has never been called.

I think we're over a barrel here: on the one hand, whether a parameter
is set or unset is a generic property that naturally gets managed at
this point, while for a special preparing a value to return is a highly
specific matter.  I don't think we could do this much better without
upgrading the interfaces somewhere around these lower levels.

The best we could do, perhaps, is enforce a rule that a special that's
set is required to return something sensible (not necessarily useful)
even before a value has been passed in.  A lot of specials probably
already respect this --- we fixed problems in the distant past with
nulls appearing where we didn't want them, I seem to remember.

> } Perhaps what it should be doing is creating a new one in the inner
> } scope that hides the private one as the least bad effect.
> 
> What should happen is that it creates a new one at global scope, and
> trips warncreateglobal if necessary.  Which DOES happen if the one at
> global scope was previously assigned and/or explicitly declared.  The
> difficulty only arises when the private parameter is at the outermost
> scope where any parameter of that name can be found.

OK, I'm missing the fact that usually the private stuff keeps well out
of the way of whatever else is going on, hence its name, except for some
slightly suspect interactions with a previous parameter at this point.

pws