Adding tests for zle? The missing X series tests

Adding tests for zle? The missing X series tests

[Felix Rosencrantz]

I was thinking about refactoring doisearch(), so it is easier to
read/understand/change, so the base function might look like below.
Though before attempting this I would like to have the safety net of
tests, to assure the changes don't break the incremental search
widgets. There are no zle tests, though the letter X was reserved for
zle tests.  I was wondering if anyone had thought about how this might
be done.  I looked at the tests for the completion, and tried to copy
that, though got mired down a little, since I'm not  familiar with the
terminal codes.  The completion code added xml like tags with zstyle
formatting controls (list-colors, message format, etc) and then parses
that output, though the incremental search widgets don't have that
kind of control.

-FR

How the base function might be refactored:

/*
 * doisearch:
 *   args: (char**) args to search widget
 *      Currently only first arg is used, given as input to ungetbytes, for
 *      initial search string.
 *   dir: (int) direction (-1=reverse, 1= forward)
 *   is_regex: (int) is search string a regex? (0= No, 1=Yes)
 */
static int
doisearch(char **args, int dir, int is_regex)
{
    int action, ret;
    struct isearch_globals *isg;
    isg = doisearch_init(args, dir);
    if (isg == NULL) return 1;
    for(;;) {
      doisearch_search(isg, is_regex);
      doisearch_highlighting(isg);
      action = doisearch_handleinput(isg);
      if (act == SEARCH_DONE) break;
    }
    ret = doisearch_cleanup(isg);
    return ret;
}

Re: Adding tests for zle? The missing X series tests

[Peter Stephenson]

On Tue, 20 Dec 2011 08:20:07 -0800
Felix Rosencrantz <f.rosencrantz@gmail.com> wrote:
> I was thinking about refactoring doisearch(), so it is easier to
> read/understand/change, so the base function might look like below.
> Though before attempting this I would like to have the safety net of
> tests, to assure the changes don't break the incremental search
> widgets. There are no zle tests, though the letter X was reserved for
> zle tests.  I was wondering if anyone had thought about how this might
> be done.  I looked at the tests for the completion, and tried to copy
> that, though got mired down a little, since I'm not  familiar with the
> terminal codes.  The completion code added xml like tags with zstyle
> formatting controls (list-colors, message format, etc) and then parses
> that output, though the incremental search widgets don't have that
> kind of control.

This is definitely at the expensive end of the market... at the level
you're talking about, simply verifying that the right widgets get called
isn't good enough, you need to check what's happening to the command
line.  I don't see any real alternative to checking something coming
out of the shell that is or corresponds to the terminal sequences.

We could probably arrange to rig up our own pseudotermcap entry that
outputs something along the lines that the completion code does, but
this sounds like a lot of work --- we're at least two generations of
terminal library along from termcap, so ensuring every test set-up uses
our substitute terminal definitions, and that we've caught all relevant
codes, seems like hard work.

Probably easier is to intercept the terminal output at a low-level
inside the shell by having an option that directly outputs values from
an associative array with termcap codes as keys (this doesn't need to be
particularly user-friendly, and as zsh internally still uses termcap
codes this looks like the easiest way to test it) instead of using the
termcap output functions.  That shouldn't be too much work --- termcap
is vastly simpler than e.g. curses --- and makes the job no harder (!)
than testing completion.  There are also potential uses for this in
debugging --- it should provide a simpler method of finding out what has
caused the cursor to end up where it did.

Simpler still is just to output some marked up version of the termcap
code itself.  You could have a format string stored in a shell variable
with placeholders to insert the code and a numeric argument within it to
get the markup.  The extra flexibility for an associative array isn't
hard, since all the framework for it is already there, but maybe we
don't need it.

I'm on holiday from tomorrow for two weeks and while I'll be able to
read email I'll be in Androidland so am unlikely to be very
communicative.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: Adding tests for zle? The missing X series tests

[Bart Schaefer]

On Dec 20,  8:20am, Felix Rosencrantz wrote:
}
} There are no zle tests, though the letter X was reserved for
} zle tests.  I was wondering if anyone had thought about how this might
} be done.

I have thought a bit about this in the past and I think there may be a
sort of brute-force way to do it.  It wouldn't work for everything, but
it should work for most simple ZLE edits and for incremental search in
particular.

First you'd need to set up a zpty session like the completion tests do.
Make sure to give it adequate dimensions so ZLE scrolling does not kick
in (unless that's what you're testing, I guess).

Next you set PS1 to a recognizable string and clear RPS1 etc.  For some
tests you'd might want a recognizable PS2 as well, but for incremental
search it wouldn't be necessary.

Finally you arrange that clear-screen is bound to a well-known keystroke
(the defaut of ctrl-L is probably fine).

Now you're ready to test.  Start sending keystrokes to the zpty and
read back (and probably discard) the zpty output as you do so.  At
the point where you want to check the result, send clear-screen, which
causes the whole screen to be redrawn (even during incremental search)
including the PS1 prompt.  This repaint should result in pretty much a
linear dump of the editor buffers with few or no cursor movements after
the initial screen clear.

Consequently your next zpty read after the clear-screen can scan for
the PS1 string you set previously, in order to skip over the terminal
control output, and then slurp up everything else that was printed and
compare to what you expected to find.

(A quick check using the script command to capture output indicates that
there is a clear-to-end-of-line emitted at the end of each line of the
output, but that should be relatively easy to ignore when parsing.)

This won't uncover terminal-handling bugs like unexpected line wraps at
the last line/column, but it should be able to verify e.g. the search is
finding the right thing at each incremental step.

Re: Adding tests for zle? The missing X series tests

[Peter Stephenson]

On Wed, 21 Dec 2011 11:39:47 +0000
Peter Stephenson <Peter.Stephenson@csr.com> wrote:
> Probably easier is to intercept the terminal output at a low-level
> inside the shell by having an option that directly outputs values from
> an associative array with termcap codes as keys (this doesn't need to be
> particularly user-friendly, and as zsh internally still uses termcap
> codes this looks like the easiest way to test it) instead of using the
> termcap output functions.  That shouldn't be too much work --- termcap
> is vastly simpler than e.g. curses --- and makes the job no harder (!)
> than testing completion.  There are also potential uses for this in
> debugging --- it should provide a simpler method of finding out what has
> caused the cursor to end up where it did.

A function both makes this flexible and allows you to handle all termcap
entries uniformly.

I've done this by adding a generic command for zle transformations, but
I don't know that we'll ever use this for anything else.

Try

tcfunc() {
  if [[ -n $2 ]]; then
    REPLY="<tc=$1,arg=$2>"
  else
    REPLY="<tc=$1>"
fi
}  
zle -T tc tcfunc

and

zle -Tr tc

to turn it off (your typing had better be accurate).

There's a completely irrelevant zsfree(argzero) snuck in because I was
too lazy to separate it.  This stops a valgrind warning in some cases
(it's a very minor leak since it only happens once at the start).

Index: Doc/Zsh/zle.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/zle.yo,v
retrieving revision 1.95
diff -p -u -r1.95 zle.yo
--- Doc/Zsh/zle.yo	4 Nov 2011 14:14:27 -0000	1.95
+++ Doc/Zsh/zle.yo	4 Jan 2012 17:32:13 -0000
@@ -370,6 +370,7 @@ xitem(tt(zle) tt(-U) var(string))
 xitem(tt(zle) tt(-K) var(keymap))
 xitem(tt(zle) tt(-F) [ tt(-L) ] [ var(fd) [ var(handler) ] ])
 xitem(tt(zle) tt(-I))
+xitem(tt(zle) tt(-T) [ tt(tc) var(function) | tt(-r) tt(tc) | tt(-L) ] )
 item(tt(zle) var(widget) tt([ -n) var(num) tt(]) tt([ -Nw ] [ -K) var(keymap) tt(]) var(args) ...)(
 The tt(zle) builtin performs a number of different actions concerning
 ZLE.
@@ -572,6 +573,33 @@ this may have been by a previous call to
 notification.  To test if a zle widget may be called at this point, execute
 tt(zle) with no arguments and examine the return status.
 )
+item(tt(-T))(
+This is used to add, list or remove internal transformations on the
+processing performed by the line editor.  It is typically used only for
+debugging or testing and is therefore of little interest to the general
+user.
+
+`tt(zle -T) var(transformation) var(func)' specifies that the
+given var(transformation) (see below) is effected by shell function
+var(func).
+
+`tt(zle -Tr) var(transformation)' removes the given var(transformation)
+if it was present (it is not an error if none was).
+
+`tt(zle -TL)' can be used to list all transformations currently in
+operation.
+
+Currently the only transformation is tt(tc).  This is used instead
+of outputting termcap codes to the terminal.  When the transformation is
+in operation the shell function is passed the termcap code that would be
+output as its first argument; if the operation required a numeric
+argument, that is passed as a second argument.  The function should set
+the shell variable tt(REPLY) to the transformed termcap code.  Typically
+this is used to produce some simply formatted version of the code and
+optional argument for debugging or testing.  Note that this
+transformation is not applied to other non-printing characters such as
+carriage returns and newlines.
+)
 item(var(widget) tt([ -n) var(num) tt(]) tt([ -Nw ] [ -K) var(keymap) tt(]) var(args) ...)(
 Invoke the specified widget.  This can only be done when ZLE is
 active; normally this will be within a user-defined widget.
Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.120
diff -p -u -r1.120 init.c
--- Src/init.c	9 May 2011 09:49:09 -0000	1.120
+++ Src/init.c	4 Jan 2012 17:32:14 -0000
@@ -558,6 +558,19 @@ static char *tccapnams[TC_COUNT] = {
     "ku", "kd", "kl", "kr", "sc", "rc", "bc", "AF", "AB"
 };
 
+/**/
+mod_export char *
+tccap_get_name(int cap)
+{
+    if (cap >= TC_COUNT) {
+#ifdef DEBUG
+	dputs("name of invalid capability %d requested", cap);
+#endif
+	return "";
+    }
+    return tccapnams[cap];
+}
+
 /* Initialise termcap */
 
 /**/
@@ -978,6 +991,7 @@ setupshin(char *runscript)
 	    exit(127);
 	}
 	scriptfilename = sfname;
+	zsfree(argzero); /* ztrdup'd in parseargs */
 	argzero = runscript;
     }
     /*
Index: Src/Zle/zle_main.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_main.c,v
retrieving revision 1.131
diff -p -u -r1.131 zle_main.c
--- Src/Zle/zle_main.c	2 Jan 2012 19:31:16 -0000	1.131
+++ Src/Zle/zle_main.c	4 Jan 2012 17:32:14 -0000
@@ -1927,7 +1927,7 @@ zle_main_entry(int cmd, va_list ap)
 static struct builtin bintab[] = {
     BUILTIN("bindkey", 0, bin_bindkey, 0, -1, 0, "evaM:ldDANmrsLRp", NULL),
     BUILTIN("vared",   0, bin_vared,   1,  1, 0, "aAcehM:m:p:r:t:", NULL),
-    BUILTIN("zle",     0, bin_zle,     0, -1, 0, "aAcCDFgGIKlLmMNRU", NULL),
+    BUILTIN("zle",     0, bin_zle,     0, -1, 0, "aAcCDFgGIKlLmMNrRTU", NULL),
 };
 
 /* The order of the entries in this table has to match the *HOOK
Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.88
diff -p -u -r1.88 zle_refresh.c
--- Src/Zle/zle_refresh.c	17 Aug 2011 10:15:50 -0000	1.88
+++ Src/Zle/zle_refresh.c	4 Jan 2012 17:32:14 -0000
@@ -233,6 +233,12 @@ int n_region_highlights;
 /**/
 int region_active;
 
+/*
+ * Name of function to use to output termcap values, if defined.
+ */
+/**/
+char *tcout_func_name;
+
 #ifdef HAVE_SELECT
 /* cost of last update */
 /**/
@@ -2271,11 +2277,78 @@ tc_downcurs(int ct)
     return ret;
 }
 
+/*
+ * Output a termcap value using a function defined by "zle -T tc".
+ * Loosely inspired by subst_string_by_func().
+ *
+ * cap is the internal index for the capability; it will be looked up
+ * in the table and the string passed to the function.
+ *
+ * arg is eithr an argument to the capability or -1 if there is none;
+ * if it is not -1 it will be passed as an additional argument to the
+ * function.
+ *
+ * outc is the output function; currently this is always putshout
+ * but in principle it may be used to output to a string.
+ */
+
+/**/
+static void
+tcout_via_func(int cap, int arg, int (*outc)(int))
+{
+    Shfunc tcout_func;
+    int osc, osm, old_incompfunc;
+
+    osc = sfcontext;
+    osm = stopmsg;
+    old_incompfunc = incompfunc;
+
+    sfcontext = SFC_SUBST;
+    incompfunc = 0;
+
+    if ((tcout_func = getshfunc(tcout_func_name))) {
+	LinkList l = newlinklist();
+	char buf[DIGBUFSIZE], *str;
+
+	addlinknode(l, tcout_func_name);
+	addlinknode(l, tccap_get_name(cap));
+
+	if (arg != -1) {
+	    sprintf(buf, "%d", arg);
+	    addlinknode(l, buf);
+	}
+
+	(void)doshfunc(tcout_func, l, 1);
+
+	str = getsparam("REPLY");
+	if (str) {
+	    while (*str) {
+		int chr;
+		if (*str == Meta) {
+		    chr = str[1] ^ 32;
+		    str += 2;
+		} else {
+		    chr = *str++;
+		}
+		(void)outc(chr);
+	    }
+	}
+    }
+
+    sfcontext = osc;
+    stopmsg = osm;
+    incompfunc = old_incompfunc;
+}
+
 /**/
 mod_export void
 tcout(int cap)
 {
-    tputs(tcstr[cap], 1, putshout);
+    if (tcout_func_name) {
+	tcout_via_func(cap, -1, putshout);
+    } else {
+	tputs(tcstr[cap], 1, putshout);
+    }
     SELECT_ADD_COST(tclen[cap]);
 }
 
@@ -2286,7 +2359,11 @@ tcoutarg(int cap, int arg)
     char *result;
 
     result = tgoto(tcstr[cap], arg, arg);
-    tputs(result, 1, putshout);
+    if (tcout_func_name) {
+	tcout_via_func(cap, arg, putshout);
+    } else {
+	tputs(result, 1, putshout);
+    }
     SELECT_ADD_COST(strlen(result));
 }
 
Index: Src/Zle/zle_thingy.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_thingy.c,v
retrieving revision 1.35
diff -p -u -r1.35 zle_thingy.c
--- Src/Zle/zle_thingy.c	4 Nov 2011 14:14:27 -0000	1.35
+++ Src/Zle/zle_thingy.c	4 Jan 2012 17:32:14 -0000
@@ -353,6 +353,7 @@ bin_zle(char *name, char **args, Options
 	{ 'K', bin_zle_keymap, 1, 1 },
 	{ 'I', bin_zle_invalidate, 0, 0 },
 	{ 'F', bin_zle_fd, 0, 2 },
+	{ 'T', bin_zle_transform, 0, 2},
 	{ 0,   bin_zle_call, 0, -1 },
     };
     struct opn const *op, *opp;
@@ -856,6 +857,69 @@ bin_zle_fd(char *name, char **args, Opti
     return 0;
 }
 
+/**/
+static int
+bin_zle_transform(char *name, char **args, Options ops, UNUSED(char func))
+{
+    /*
+     * -1: too few arguments
+     * 0: just right
+     * 1: too many arguments
+     * 2: first argument not recognised
+     */
+    int badargs = 0;
+
+    if (OPT_ISSET(ops,'L')) {
+	if (args[0]) {
+	    if (args[1]) {
+		badargs = 1;
+	    } else if (strcmp(args[0], "tc")) {
+		badargs = 2;
+	    }
+	}
+	if (!badargs && tcout_func_name) {
+	    fputs("zle -T tc ", stdout);
+	    quotedzputs(tcout_func_name, stdout);
+	    putchar('\n');
+	}
+    } else if (OPT_ISSET(ops,'r')) {
+	if (!args[0]) {
+	    badargs = -1;
+	} else if (args[1]) {
+	    badargs = 1;
+	} else if (tcout_func_name) {
+	    zsfree(tcout_func_name);
+	    tcout_func_name = NULL;
+	}
+    } else {
+	if (!args[0] || !args[1]) {
+	    badargs = -1;
+	    /* we've already checked args <= 2 */
+	} else {
+	    if (!strcmp(args[0], "tc")) {
+		if (tcout_func_name) {
+		    zsfree(tcout_func_name);
+		}
+		tcout_func_name = ztrdup(args[1]);
+	    } else {
+		badargs = 2;
+	    }
+	}
+    }
+
+    if (badargs) {
+	if (badargs == 2) {
+	    zwarnnam(name, "-T: no such transformation '%s'", args[0]);
+	} else {
+	    char *way = (badargs > 0) ? "many" : "few";
+	    zwarnnam(name, "too %s arguments for option -T", way);
+	}
+	return 1;
+    }
+
+    return 0;
+}
+
 /*******************/
 /* initialiasation */
 /*******************/

Re: Adding tests for zle? The missing X series tests

[Felix Rosencrantz]

[-- Attachment #1: Type: text/plain, Size: 12432 bytes --]

I've attached draft versions of the incremental search tests, based on
the suggestions of Bart and Peter. I just added on Peter's latest
change, previously I was using a hard coded escape sequence.  The
zletest file is based on comptest, and includes debugging code.  The
test is flaky, so there might be some buffering issues, or I'm not
properly reseting the state between tests.  (Or just confused on what
to expect...)

If you are interested in seeing what it does, you can add these under
the Test directory and run: make TESTNUM=X check

I would be interest in any feedback.  These are still draft condition,
but might be of interest to anyone following this thread.

-FR


On Wed, Jan 4, 2012 at 9:38 AM, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
> On Wed, 21 Dec 2011 11:39:47 +0000
> Peter Stephenson <Peter.Stephenson@csr.com> wrote:
>> Probably easier is to intercept the terminal output at a low-level
>> inside the shell by having an option that directly outputs values from
>> an associative array with termcap codes as keys (this doesn't need to be
>> particularly user-friendly, and as zsh internally still uses termcap
>> codes this looks like the easiest way to test it) instead of using the
>> termcap output functions.  That shouldn't be too much work --- termcap
>> is vastly simpler than e.g. curses --- and makes the job no harder (!)
>> than testing completion.  There are also potential uses for this in
>> debugging --- it should provide a simpler method of finding out what has
>> caused the cursor to end up where it did.
>
> A function both makes this flexible and allows you to handle all termcap
> entries uniformly.
>
> I've done this by adding a generic command for zle transformations, but
> I don't know that we'll ever use this for anything else.
>
> Try
>
> tcfunc() {
>  if [[ -n $2 ]]; then
>    REPLY="<tc=$1,arg=$2>"
>  else
>    REPLY="<tc=$1>"
> fi
> }
> zle -T tc tcfunc
>
> and
>
> zle -Tr tc
>
> to turn it off (your typing had better be accurate).
>
> There's a completely irrelevant zsfree(argzero) snuck in because I was
> too lazy to separate it.  This stops a valgrind warning in some cases
> (it's a very minor leak since it only happens once at the start).
>
> Index: Doc/Zsh/zle.yo
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Doc/Zsh/zle.yo,v
> retrieving revision 1.95
> diff -p -u -r1.95 zle.yo
> --- Doc/Zsh/zle.yo      4 Nov 2011 14:14:27 -0000       1.95
> +++ Doc/Zsh/zle.yo      4 Jan 2012 17:32:13 -0000
> @@ -370,6 +370,7 @@ xitem(tt(zle) tt(-U) var(string))
>  xitem(tt(zle) tt(-K) var(keymap))
>  xitem(tt(zle) tt(-F) [ tt(-L) ] [ var(fd) [ var(handler) ] ])
>  xitem(tt(zle) tt(-I))
> +xitem(tt(zle) tt(-T) [ tt(tc) var(function) | tt(-r) tt(tc) | tt(-L) ] )
>  item(tt(zle) var(widget) tt([ -n) var(num) tt(]) tt([ -Nw ] [ -K) var(keymap) tt(]) var(args) ...)(
>  The tt(zle) builtin performs a number of different actions concerning
>  ZLE.
> @@ -572,6 +573,33 @@ this may have been by a previous call to
>  notification.  To test if a zle widget may be called at this point, execute
>  tt(zle) with no arguments and examine the return status.
>  )
> +item(tt(-T))(
> +This is used to add, list or remove internal transformations on the
> +processing performed by the line editor.  It is typically used only for
> +debugging or testing and is therefore of little interest to the general
> +user.
> +
> +`tt(zle -T) var(transformation) var(func)' specifies that the
> +given var(transformation) (see below) is effected by shell function
> +var(func).
> +
> +`tt(zle -Tr) var(transformation)' removes the given var(transformation)
> +if it was present (it is not an error if none was).
> +
> +`tt(zle -TL)' can be used to list all transformations currently in
> +operation.
> +
> +Currently the only transformation is tt(tc).  This is used instead
> +of outputting termcap codes to the terminal.  When the transformation is
> +in operation the shell function is passed the termcap code that would be
> +output as its first argument; if the operation required a numeric
> +argument, that is passed as a second argument.  The function should set
> +the shell variable tt(REPLY) to the transformed termcap code.  Typically
> +this is used to produce some simply formatted version of the code and
> +optional argument for debugging or testing.  Note that this
> +transformation is not applied to other non-printing characters such as
> +carriage returns and newlines.
> +)
>  item(var(widget) tt([ -n) var(num) tt(]) tt([ -Nw ] [ -K) var(keymap) tt(]) var(args) ...)(
>  Invoke the specified widget.  This can only be done when ZLE is
>  active; normally this will be within a user-defined widget.
> Index: Src/init.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/init.c,v
> retrieving revision 1.120
> diff -p -u -r1.120 init.c
> --- Src/init.c  9 May 2011 09:49:09 -0000       1.120
> +++ Src/init.c  4 Jan 2012 17:32:14 -0000
> @@ -558,6 +558,19 @@ static char *tccapnams[TC_COUNT] = {
>     "ku", "kd", "kl", "kr", "sc", "rc", "bc", "AF", "AB"
>  };
>
> +/**/
> +mod_export char *
> +tccap_get_name(int cap)
> +{
> +    if (cap >= TC_COUNT) {
> +#ifdef DEBUG
> +       dputs("name of invalid capability %d requested", cap);
> +#endif
> +       return "";
> +    }
> +    return tccapnams[cap];
> +}
> +
>  /* Initialise termcap */
>
>  /**/
> @@ -978,6 +991,7 @@ setupshin(char *runscript)
>            exit(127);
>        }
>        scriptfilename = sfname;
> +       zsfree(argzero); /* ztrdup'd in parseargs */
>        argzero = runscript;
>     }
>     /*
> Index: Src/Zle/zle_main.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_main.c,v
> retrieving revision 1.131
> diff -p -u -r1.131 zle_main.c
> --- Src/Zle/zle_main.c  2 Jan 2012 19:31:16 -0000       1.131
> +++ Src/Zle/zle_main.c  4 Jan 2012 17:32:14 -0000
> @@ -1927,7 +1927,7 @@ zle_main_entry(int cmd, va_list ap)
>  static struct builtin bintab[] = {
>     BUILTIN("bindkey", 0, bin_bindkey, 0, -1, 0, "evaM:ldDANmrsLRp", NULL),
>     BUILTIN("vared",   0, bin_vared,   1,  1, 0, "aAcehM:m:p:r:t:", NULL),
> -    BUILTIN("zle",     0, bin_zle,     0, -1, 0, "aAcCDFgGIKlLmMNRU", NULL),
> +    BUILTIN("zle",     0, bin_zle,     0, -1, 0, "aAcCDFgGIKlLmMNrRTU", NULL),
>  };
>
>  /* The order of the entries in this table has to match the *HOOK
> Index: Src/Zle/zle_refresh.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
> retrieving revision 1.88
> diff -p -u -r1.88 zle_refresh.c
> --- Src/Zle/zle_refresh.c       17 Aug 2011 10:15:50 -0000      1.88
> +++ Src/Zle/zle_refresh.c       4 Jan 2012 17:32:14 -0000
> @@ -233,6 +233,12 @@ int n_region_highlights;
>  /**/
>  int region_active;
>
> +/*
> + * Name of function to use to output termcap values, if defined.
> + */
> +/**/
> +char *tcout_func_name;
> +
>  #ifdef HAVE_SELECT
>  /* cost of last update */
>  /**/
> @@ -2271,11 +2277,78 @@ tc_downcurs(int ct)
>     return ret;
>  }
>
> +/*
> + * Output a termcap value using a function defined by "zle -T tc".
> + * Loosely inspired by subst_string_by_func().
> + *
> + * cap is the internal index for the capability; it will be looked up
> + * in the table and the string passed to the function.
> + *
> + * arg is eithr an argument to the capability or -1 if there is none;
> + * if it is not -1 it will be passed as an additional argument to the
> + * function.
> + *
> + * outc is the output function; currently this is always putshout
> + * but in principle it may be used to output to a string.
> + */
> +
> +/**/
> +static void
> +tcout_via_func(int cap, int arg, int (*outc)(int))
> +{
> +    Shfunc tcout_func;
> +    int osc, osm, old_incompfunc;
> +
> +    osc = sfcontext;
> +    osm = stopmsg;
> +    old_incompfunc = incompfunc;
> +
> +    sfcontext = SFC_SUBST;
> +    incompfunc = 0;
> +
> +    if ((tcout_func = getshfunc(tcout_func_name))) {
> +       LinkList l = newlinklist();
> +       char buf[DIGBUFSIZE], *str;
> +
> +       addlinknode(l, tcout_func_name);
> +       addlinknode(l, tccap_get_name(cap));
> +
> +       if (arg != -1) {
> +           sprintf(buf, "%d", arg);
> +           addlinknode(l, buf);
> +       }
> +
> +       (void)doshfunc(tcout_func, l, 1);
> +
> +       str = getsparam("REPLY");
> +       if (str) {
> +           while (*str) {
> +               int chr;
> +               if (*str == Meta) {
> +                   chr = str[1] ^ 32;
> +                   str += 2;
> +               } else {
> +                   chr = *str++;
> +               }
> +               (void)outc(chr);
> +           }
> +       }
> +    }
> +
> +    sfcontext = osc;
> +    stopmsg = osm;
> +    incompfunc = old_incompfunc;
> +}
> +
>  /**/
>  mod_export void
>  tcout(int cap)
>  {
> -    tputs(tcstr[cap], 1, putshout);
> +    if (tcout_func_name) {
> +       tcout_via_func(cap, -1, putshout);
> +    } else {
> +       tputs(tcstr[cap], 1, putshout);
> +    }
>     SELECT_ADD_COST(tclen[cap]);
>  }
>
> @@ -2286,7 +2359,11 @@ tcoutarg(int cap, int arg)
>     char *result;
>
>     result = tgoto(tcstr[cap], arg, arg);
> -    tputs(result, 1, putshout);
> +    if (tcout_func_name) {
> +       tcout_via_func(cap, arg, putshout);
> +    } else {
> +       tputs(result, 1, putshout);
> +    }
>     SELECT_ADD_COST(strlen(result));
>  }
>
> Index: Src/Zle/zle_thingy.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_thingy.c,v
> retrieving revision 1.35
> diff -p -u -r1.35 zle_thingy.c
> --- Src/Zle/zle_thingy.c        4 Nov 2011 14:14:27 -0000       1.35
> +++ Src/Zle/zle_thingy.c        4 Jan 2012 17:32:14 -0000
> @@ -353,6 +353,7 @@ bin_zle(char *name, char **args, Options
>        { 'K', bin_zle_keymap, 1, 1 },
>        { 'I', bin_zle_invalidate, 0, 0 },
>        { 'F', bin_zle_fd, 0, 2 },
> +       { 'T', bin_zle_transform, 0, 2},
>        { 0,   bin_zle_call, 0, -1 },
>     };
>     struct opn const *op, *opp;
> @@ -856,6 +857,69 @@ bin_zle_fd(char *name, char **args, Opti
>     return 0;
>  }
>
> +/**/
> +static int
> +bin_zle_transform(char *name, char **args, Options ops, UNUSED(char func))
> +{
> +    /*
> +     * -1: too few arguments
> +     * 0: just right
> +     * 1: too many arguments
> +     * 2: first argument not recognised
> +     */
> +    int badargs = 0;
> +
> +    if (OPT_ISSET(ops,'L')) {
> +       if (args[0]) {
> +           if (args[1]) {
> +               badargs = 1;
> +           } else if (strcmp(args[0], "tc")) {
> +               badargs = 2;
> +           }
> +       }
> +       if (!badargs && tcout_func_name) {
> +           fputs("zle -T tc ", stdout);
> +           quotedzputs(tcout_func_name, stdout);
> +           putchar('\n');
> +       }
> +    } else if (OPT_ISSET(ops,'r')) {
> +       if (!args[0]) {
> +           badargs = -1;
> +       } else if (args[1]) {
> +           badargs = 1;
> +       } else if (tcout_func_name) {
> +           zsfree(tcout_func_name);
> +           tcout_func_name = NULL;
> +       }
> +    } else {
> +       if (!args[0] || !args[1]) {
> +           badargs = -1;
> +           /* we've already checked args <= 2 */
> +       } else {
> +           if (!strcmp(args[0], "tc")) {
> +               if (tcout_func_name) {
> +                   zsfree(tcout_func_name);
> +               }
> +               tcout_func_name = ztrdup(args[1]);
> +           } else {
> +               badargs = 2;
> +           }
> +       }
> +    }
> +
> +    if (badargs) {
> +       if (badargs == 2) {
> +           zwarnnam(name, "-T: no such transformation '%s'", args[0]);
> +       } else {
> +           char *way = (badargs > 0) ? "many" : "few";
> +           zwarnnam(name, "too %s arguments for option -T", way);
> +       }
> +       return 1;
> +    }
> +
> +    return 0;
> +}
> +
>  /*******************/
>  /* initialiasation */
>  /*******************/

[-- Attachment #2: X01isearch.ztst --]
[-- Type: application/octet-stream, Size: 4820 bytes --]

%prep
  if ( zmodload -i zsh/zpty ) >/dev/null 2>&1; then
    . $ZTST_srcdir/zletest
    mkdir zle.tmp
    cd zle.tmp
    zletestinit -z $ZTST_testdir/../Src/zsh
  else
    ZTST_unimplemented="the zsh/zpty module is not available"
  fi

%test
# Tests to add:
#  case-insesitivity
#  ignoring duplicate lines
#  special keys

# Doesn't work, final prompt is empty, since command ran:
# zletesteval 'fc -R historyX01'
# zletest $'fc -l -3\n'
#0q:Verify incremental search

 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabc'
0q:Verify incremental search first match
>echo <FG 2><BG 1>abc<FG 9><BG 9>
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabcd\C-H'
0q:Verify incremental search first match via backspace
>echo <FG 2><BG 1>abc<FG 9><BG 9>
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\n\C-Rabc\C-R'
0q:Verify incremental search second match
>echo abc <FG 2><BG 1>abc<FG 9><BG 9>def
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabc\C-R\C-R'
0q:Verify incremental search third match
>echo <FG 2><BG 1>abc<FG 9><BG 9> abcdef
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabc\C-R\C-R\C-R'
0q:Verify incremental search fourth match
>echo 123 <FG 2><BG 1>abc<FG 9><BG 9>
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabc\C-R\C-R\C-R\C-R'
0q:Verify incremental search failed fifth match
>echo 123 abc
>failing bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabc\C-R\C-R\C-Rd'
0q:Verify incremental search pass something 
>echo 123 abc
>failing bck-i-search: abcd_

 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabc\C-R\C-R\C-Rd\C-H\C-H\C-Hd'
0q:Verify incremental search pass, backup, and find again.
>echo abc <FG 2><BG 1>abcd<FG 9><BG 9>ef
>bck-i-search: abcd_

 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabcd\C-T'
0q:Verify incremental search pass, then use search forward to find.
>echo abc <FG 2><BG 1>abcd<FG 9><BG 9>ef
>fwd-i-search: abcd_

 zletesteval 'fc -R historyX01'
 zletest $'\C-R^date'
0q:Verify incremental search bol pattern
><FG 2><BG 1>date<FG 9><BG 9>
>bck-i-search: ^date_

 zletesteval 'fc -R historyX01'
 zletest $'\C-R\\'
0q:Verify incremental search backslash
>echo '<FG 2><BG 1>\<FG 9><BG 9>n'
>bck-i-search: \_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XR\\*'
0q:Verify incremental search backslash star
>echo "*WHAT?<FG 2><BG 1>*<FG 9><BG 9>"
>bck-i-search: \*_

############# incremental pattern search

 zletesteval 'fc -R historyX01'
 zletest $'\C-XRabc'
0q:Verify incremental pattern search first match
>echo <FG 2><BG 1>abc<FG 9><BG 9>
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XRabcd\C-H'
0q:Verify incremental pattern search first match via backspace
>echo <FG 2><BG 1>abc<FG 9><BG 9>
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\n\C-XRabc\C-XR'
0q:Verify incremental pattern search second match
>echo abc <FG 2><BG 1>abc<FG 9><BG 9>def
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XRabc\C-XR\C-XR'
0q:Verify incremental pattern search third match
>echo <FG 2><BG 1>abc<FG 9><BG 9> abcdef
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XRabc\C-XR\C-XR\C-XR'
0q:Verify incremental pattern search fourth match
>echo 123 <FG 2><BG 1>abc<FG 9><BG 9>
>bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XRabc\C-XR\C-XR\C-XR\C-XR'
0q:Verify incremental pattern search failed fifth match
>echo 123 abc
>failing bck-i-search: abc_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XRabc\C-XR\C-XR\C-XRd'
0q:Verify incremental pattern search pass something 
>echo 123 abc
>failing bck-i-search: abcd_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XRabc\C-XR\C-XR\C-XRd\C-H\C-H\C-Hd'
0q:Verify incremental pattern search pass, backup, and find again.
>echo abc <FG 2><BG 1>abcd<FG 9><BG 9>ef
>bck-i-search: abcd_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XRabcd\C-XT'
0q:Verify incremental pattern search pass, then use search forward to find.
>echo abc <FG 2><BG 1>abcd<FG 9><BG 9>ef
>fwd-i-search: abcd_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XR^date'
0q:Verify incremental pattern search bol pattern
###><FG 2><BG 1>date<FG 9><BG 9>
###>bck-i-search: ^date_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XR\\'
0q:Verify incremental pattern search backslash
>echo '<FG 2><BG 1>\<FG 9><BG 9>n'
>bck-i-search: \_

 zletesteval 'fc -R historyX01'
 zletest $'\C-XR\\*'
0q:Verify incremental pattern search backslash star
>echo "*WHAT?<FG 2><BG 1>*<FG 9><BG 9>"
>bck-i-search: \*_

%clean
# This optional section cleans up after the test, if necessary,
# e.g. killing processes etc.  This is in addition to the removal of *.tmp
# subdirectories.  This is essentially like %prep, except that status
# return values are ignored.
zmodload -ui zsh/zpty

[-- Attachment #3: zletest --]
[-- Type: application/octet-stream, Size: 2707 bytes --]

zletestinit () {
  setopt extendedglob
  [[ -d $ZTST_testdir/Modules/zsh ]] && module_path=( $ZTST_testdir/Modules )
  fpath=( $ZTST_srcdir/../Functions/*~*/CVS(/)
          $ZTST_srcdir/../Completion
          $ZTST_srcdir/../Completion/*/*~*/CVS(/) )

  zmodload -i zsh/zpty || return $?

  zletest_zsh=${ZSH:-zsh}

  while getopts z: opt; do
    case $opt in
      z) zletest_zsh="$OPTARG";;
    esac
  done
  (( OPTIND > 1 )) && shift $(( OPTIND - 1 ))

  export PS1="<PROMPT>" PS2="" RPS1="" RPS2=""
  zpty zsh "$zletest_zsh -f +Z"

  zpty -r zsh log1 "*<PROMPT>*" || { 
    print "first prompt hasn't appeared."
    return 1
  }

  cat > historyX01 <<EOF
date
echo xyz three
echo xyz two
echo 123 abc
echo abc abcdef
echo abc
echo xyz
echo date
echo '*OH NO*'                                                                                                                                  
echo '\n'
echo "*WHAT?*"
EOF

  zletesteval \
"export LC_ALL=C" \
"emulate -R zsh" \
"export ZDOTDIR=$ZTST_testdir" \
"module_path=( $module_path )" \
"fpath=( $fpath )" \
"zle_highlight=(bg_start_code:'\<BG ' bg_end_code:'\>' fg_start_code:'\<FG ' fg_end_code:'\>' isearch:bg=1,fg=2)" \
'LISTMAX=10000000
TERM=vt100
stty columns 256 rows 240
setopt zle
bindkey -e
tcfunc() {
 if [[ -n $2 ]]; then
   REPLY="<tc=$1,arg=$2>"
 else
   REPLY="<tc=$1>"
fi
}
zle -T tc tcfunc
finish () {
  zle kill-whole-line
  zle kill-whole-line
  zle clear-screen
  print "<finish>"
  zle -R
}
zle -N finish
bindkey "^Z" finish
bindkey "^T" history-incremental-search-forward
bindkey "^XR" history-incremental-pattern-search-backward
bindkey "^XT" history-incremental-pattern-search-forward
'
}

zletesteval () {
  local tmp=/tmp/zletest.$$

  print -lr - "$@" > $tmp
  zpty -w zsh ". $tmp"
  zpty -r -m zsh log_eval "*<PROMPT>*" || {
    print "prompt hasn't appeared."
    return 1
  }
  rm $tmp
}

zletest () {
  input="$*"
  # zpty flags: 
  #  -w send input to pty
  #  -n don't add newline
  #  -m read input until pattern
  zpty -n -w zsh "$input"
  # clear screen, to clear out control sequences
  zpty -n -w zsh $'\C-L'
  # send trailing output to scrape off.
  zpty -n -w zsh $'\C-Z\C-U'
  zpty -r -m zsh log "*<finish>*<PROMPT>" || {
    print "failed to invoke finish widget."
    return 1
  }
  (echo log: $log ; echo DONE) >! /tmp/x01-last.out   # DEBUG
  # there might be multiple <PROMPT>'s in log, grab the last one
  # generated after the clear screen.
  log=${log%%<finish>*}
  logs=(${(s:<PROMPT>:)log})
  log2=$logs[-1]
  # Split output into lines, strip trailing ^[[K to eol
  lines=(${(f)log2})
  for line in "$lines[@]" ; do 
    #line=${line%%$'\e[K'*}
    line=${line%%<tc=c?>*}
    echo $line
  done
}

Re: Adding tests for zle? The missing X series tests

[Peter Stephenson]

On Sun, 8 Jan 2012 16:20:20 -0800
Felix Rosencrantz <f.rosencrantz@gmail.com> wrote:
> I've attached draft versions of the incremental search tests, based on
> the suggestions of Bart and Peter. I just added on Peter's latest
> change, previously I was using a hard coded escape sequence.  The
> zletest file is based on comptest, and includes debugging code.  The
> test is flaky, so there might be some buffering issues, or I'm not
> properly reseting the state between tests.  (Or just confused on what
> to expect...)
> 
> If you are interested in seeing what it does, you can add these under
> the Test directory and run: make TESTNUM=X check
> 
> I would be interest in any feedback.  These are still draft condition,
> but might be of interest to anyone following this thread.

Thanks, I haven't looked at these yet but it occurred to me that even
with the ability to look at termcap sequences we're still restricted by
what capabilities the current terminal type offers, so the result will
vary according to the TERM variable --- I haven't defined a way of
intercepting tccan().  I suppose for full control that needs a zle -T
function too (it just needs to test the return value).

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: Adding tests for zle? The missing X series tests

[Felix Rosencrantz]

On Mon, Jan 9, 2012 at 2:02 AM, Peter Stephenson
<Peter.Stephenson@csr.com> wrote:
> On Sun, 8 Jan 2012 16:20:20 -0800
> Felix Rosencrantz <f.rosencrantz@gmail.com> wrote:
>> I've attached draft versions of the incremental search tests, based on
>> the suggestions of Bart and Peter. I just added on Peter's latest
>> change, previously I was using a hard coded escape sequence.  The
>> zletest file is based on comptest, and includes debugging code.  The
>> test is flaky, so there might be some buffering issues, or I'm not
>> properly reseting the state between tests.  (Or just confused on what
>> to expect...)
>>
>> If you are interested in seeing what it does, you can add these under
>> the Test directory and run: make TESTNUM=X check
>>
>> I would be interest in any feedback.  These are still draft condition,
>> but might be of interest to anyone following this thread.
>
> Thanks, I haven't looked at these yet but it occurred to me that even
> with the ability to look at termcap sequences we're still restricted by
> what capabilities the current terminal type offers, so the result will
> vary according to the TERM variable --- I haven't defined a way of
> intercepting tccan().  I suppose for full control that needs a zle -T
> function too (it just needs to test the return value).

The completion tests set TERM to vt100, as does the zle test script I sent.
Not sure if the completion tests rely on that.  The zle draft test
sent, does rely on Vt100 to scrape off the control sequences.

-FR.

>
> --
> Peter Stephenson <pws@csr.com>            Software Engineer
> Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
> Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK
>
>
> Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
> More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: Adding tests for zle? The missing X series tests

[Peter Stephenson]

On Sun, 8 Jan 2012 16:20:20 -0800
Felix Rosencrantz <f.rosencrantz@gmail.com> wrote:
> I've attached draft versions of the incremental search tests, based on
> the suggestions of Bart and Peter.

Finally got around to looking at this; it's certainly a great start.  I
imagine we're going to be looking for oddities for a while yet.  For me
it's failing on the test below -- which means it's run five
successfully, which is already better than I might have hoped.

I made a start on further changes to protect against termcap
variability (by intercepting tests for capabilities); the complication
is the need to intercept tclen, the length of the sequence, too, and I
haven't worked around that yet.  If VT100 turns out to be standard
enough --- or we have enough ways of forcing a termcap/terminfo
definition, though I have severe doubts about that given the
multiplicity of terminal libraries ---  maybe we don't need any more.

./X01isearch.ztst: starting.
*** /tmp/zsh.ztst.out.8659	Sun Jan 29 18:38:10 2012
--- /tmp/zsh.ztst.tout.8659	Sun Jan 29 18:38:10 2012
***************
*** 1,2 ****
! echo 123 abc
! failing bck-i-search: abc_
--- 1,2 ----
! echo <FG 2><BG 1>abc<FG 9><BG 9>
! bck-i-search: abc_
Test ./X01isearch.ztst failed: output differs from expected as shown above for:
 zletesteval 'fc -R historyX01'
 zletest $'\C-Rabc\C-R\C-R\C-R\C-R'
Was testing: Verify incremental search failed fifth match
./X01isearch.ztst: test failed.
**************************************
0 successful test scripts, 1 failure, 0 skipped
**************************************
make: *** [check] Error 1

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/