zsh-workers
 help / color / mirror / code / Atom feed
* PATCH: command line highlighting
@ 2008-04-02 19:39 Peter Stephenson
  2008-04-03 10:30 ` Oliver Kiddle
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Peter Stephenson @ 2008-04-02 19:39 UTC (permalink / raw)
  To: Zsh hackers list

Now 4.3.6 is out of the way, here's something I've been working on.  It
highlights the normal command line (not a completion list) in three
ways:

- the region:  controlled by an entry in the (normal) array
  zle_highlight.  For this I needed some notion of the region being
  "active", as in Emacs; it's active when you set it or exchange
  point and mark until an insertion takes place or the line is accepted.
- unprintable characters ('nice' rendering):  ditto
- arbitrary chunks of the line:  controlled by the (zle special) array
  region_highlight.  This is limited to static uses at the moment;
  highlighting doesn't track movement.

The current defaults are for inverse video rendering of the region and
unprintable characters, while region_highlight is only used to
embolden the prompt for read-from-minibuffer.  Note you can turn off the
first two effects, plus any similar ones we come up with, with the
single assignment zle_highlight=(none).

I'm expecting there to be subtle bugs with refreshing the display.  I
have managed to tickle a few but none reproducibly.  I'm pretty sure
there are some bugs in terminal emulators, too: they aren't well-tested
with this sort of dynamic use of character highlighting.  Any
reproducible problems on multiple terminal emulators are therefore of
particular interest.

There are all sorts of possible modifications and enhancements.  I will
wait for initial comments before committing it.

Index: Doc/Zsh/zle.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/zle.yo,v
retrieving revision 1.58
diff -u -r1.58 zle.yo
--- Doc/Zsh/zle.yo	29 Jan 2008 17:51:02 -0000	1.58
+++ Doc/Zsh/zle.yo	2 Apr 2008 19:31:57 -0000
@@ -29,10 +29,22 @@
 ifzman(See em(Parameters Used By The Shell) in zmanref(zshparam))\
 ifnzman(noderef(Parameters Used By The Shell)).
 
+The parameter tt(zle_highlight) is also used by the line editor;
+ifzman(see em(Character Highlighting) below)\
+ifnzman(noderef(Character Highlighting)).  Highlighting
+of special characters and the region between the cursor and the
+mark (as set with tt(set-mark-command) in Emacs mode) is enabled
+by default; consult this reference for more information.  Irrascible
+conservatives will wish to know that all highlighting may be disabled by
+the following setting:
+
+example(zle_highlight=(none))
+
 startmenu()
 menu(Keymaps)
 menu(Zle Builtins)
 menu(Zle Widgets)
+menu(Character Highlighting)
 endmenu()
 
 texinode(Keymaps)(Zle Builtins)()(Zsh Line Editor)
@@ -578,7 +590,7 @@
 )
 enditem()
 
-texinode(Zle Widgets)()(Zle Builtins)(Zsh Line Editor)
+texinode(Zle Widgets)(Character Highlighting)(Zle Builtins)(Zsh Line Editor)
 sect(Widgets)
 cindex(widgets)
 All actions in the editor are performed by `widgets'.  A widget's job is
@@ -762,6 +774,45 @@
 If it is assigned to, only that part of the buffer is replaced, and the
 cursor remains between the old tt($LBUFFER) and the new tt($RBUFFER).
 )
+vindex(region_highlight)
+item(tt(region_highlight) (array))(
+Each element of this array may be set to a string that describes
+highlighting for an arbitrary region of the command line that will
+take effect the next time the command line is redisplayed.  Each
+string consists of the following parts:
+
+startlist()
+list(Optionally, a `tt(P)' to signify that the start and end offset that
+follow include any string set by the tt(PREDISPLAY) special parameter;
+this is needed if the predisplay string itself is to be highlighted.
+Whitespace may follow the `tt(P)'.)
+list(A start offset in the same units as tt(CURSOR), terminated by
+whitespace.)
+list(An end offset in the same units as tt(CURSOR), terminated by
+whitespace.)
+list(A highlight specification in the same format as
+used for the contexts tt(region) or tt(special)
+in the parameter tt(zle_highlight),
+ifnzman(noderef(Character Highlighting))\
+ifzman(see Character Highlighting below).
+Hence this should be a comma-separated list of any of the
+words tt(bold), tt(standout) or tt(underline).
+endlist()
+
+For example, 
+
+example(region_highlight=("P0 20 bold"))
+
+specifies that the first twenty characters of the text including
+any predisplay string should be highlighted in bold.
+
+Note that the effect of tt(region_highlight) is not saved and disappears
+as soon as the line is accepted.  The line editor makes no attempt to
+keep the highlighting effect synchronised with the line as it is edited;
+hence region highlighting is best limited to static effects within
+user widgets.
+)
+)
 vindex(WIDGET)
 item(tt(WIDGET) (scalar))(
 The name of the widget currently being executed; read-only.
@@ -1720,7 +1771,11 @@
 )
 tindex(exchange-point-and-mark)
 item(tt(exchange-point-and-mark) (^X^X) (unbound) (unbound))(
-Exchange the cursor position with the position of the mark.
+Exchange the cursor position (point) with the position of the mark.
+Unless a negative prefix argument is given, the region between
+point and mark is activated so that it can be highlighted.
+If a zero prefix argument is given, the region is activated but
+point and mark are not swapped.
 )
 tindex(execute-named-cmd)
 item(tt(execute-named-cmd) (ESC-x) (unbound) (unbound))(
@@ -1914,7 +1969,10 @@
 )
 tindex(set-mark-command)
 item(tt(set-mark-command) (^@) (unbound) (unbound))(
-Set the mark at the cursor position.
+Set the mark at the cursor position.  If called with a negative
+prefix argument, do not set the mark but deactivate the region so that
+it is no longer highlighted (it is still usable for other purposes).
+Otherwise the region is marked as active.
 )
 tindex(spell-word)
 item(tt(spell-word) (ESC-$ ESC-S ESC-s) (unbound) (unbound))(
@@ -1961,3 +2019,98 @@
 continue the argument.  Otherwise, execute vi-beginning-of-line.
 )
 enditem()
+
+texinode(Character Highlighting)()(Zle Widgets)(Zsh Line Editor)
+sect(Character Highlighting)
+
+The line editor has the ability to highlight characters or regions
+of the line that have a particular significance.  This is controlled
+by the array parameter tt(zsh_highlight), if it has been set by the user.
+
+If the parameter contains the single entry tt(none) all highlighting
+is turned off.  Note the parameter is still expected to be an array.
+
+Otherwise each entry of the array should consist of a word indicating a
+context for highlighting, then a colon, then a comma-separated list of
+the types of highlighting to apply in that context.
+
+The contexts available for highlighting are the following:
+
+startitem()
+cindex(region, highlighting)
+cindex(highlighting, region)
+item(tt(region))(
+The region between the cursor (point) and the mark as set with
+tt(set-mark-command).  The region is only highlighted if it is active,
+which is the case if tt(set-mark-command) or tt(exchange-point-and-mark)
+has been called and the line has not been subsequently modified.  The
+region can be deactivated by calling tt(set-mark-command) with a
+negative prefix argument, or reactivated by calling
+tt(exchange-point-and-mark) with a zero prefix argument.  Note
+that whether or not the region is active has no effect on its
+use within widgets, it simply determines whether it is highlighted.
+)
+cindex(special characters, highlighting)
+cindex(highlighting, special characters)
+item(tt(special))(
+Individual characters that have no direct printable
+representation but are shown in a special manner by the line editor.
+These characters are described below.)
+enditem()
+
+The available types of highlighting are the following.  Note that
+not all types of highlighting are available on all terminals:
+
+startitem()
+item(tt(none))(
+No highlighting is applied to the given context.  It is not useful for
+this to appear with other types of highlighting; it is used to override
+a default.
+)
+item(tt(bold))(
+The characters in the given context are shown in a bold font.
+)
+item(tt(standout))(
+The characters in the given context are shown in the terminal's standout
+mode.  The actual effect is specific to the terminal; on many terminals it
+is inverse video.  On some such terminals, where the cursor does not blink
+it appears with standout mode negated, making it less than clear where
+the cursor actually is.  On such terminals one of the other effects
+may be preferable for highlighting the region.
+)
+item(tt(underline))(
+The characters in the given context are shown underlined.  Some
+terminals show the foreground in a different colour instead; in this
+case whitespace will not be highlighted.
+)
+enditem()
+
+The characters described above as `special' are as follows.  The
+formatting described here is used irrespective of whether the characters
+are highlighted:
+
+startitem()
+item(ASCII control characters)(
+Control characters in the ASCII range are shown as
+`tt(^)' followed by the base character.)
+item(Unprintable multibyte characters)(
+If the tt(MULTIBYTE) option is in effect,
+multibyte characters not in the ASCII character set that are reported as
+having zero width are shown as a hexadecimal number between
+angle brackets.  The number is the code point of the character in
+the wide character set; this may or may not be Unicode, depending
+on the operating system.)
+enditem()
+
+If tt(zle_highlight) is not set or no value applies to a particular
+context, the defaults applied are equivalent to
+
+example(zle_highlight=LPAR()region:standout special:standout+RPAR())
+
+i.e. both the region and special characters are shown in standout mode.
+
+Within widgets, arbitrary regions may be highlighted by setting the
+special array parameter tt(region_highglight); see
+ifnzman(nodreref(Zle Widgets))\
+ifzman(above).
+
Index: Functions/Zle/read-from-minibuffer
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/read-from-minibuffer,v
retrieving revision 1.5
diff -u -r1.5 read-from-minibuffer
--- Functions/Zle/read-from-minibuffer	21 Mar 2006 19:19:18 -0000	1.5
+++ Functions/Zle/read-from-minibuffer	2 Apr 2008 19:31:57 -0000
@@ -19,14 +19,14 @@
 done
 (( OPTIND > 1 )) && shift $(( OPTIND - 1 ))
 
-local savelbuffer=$LBUFFER saverbuffer=$RBUFFER
-local savepredisplay=$PREDISPLAY savepostdisplay=$POSTDISPLAY
-
-LBUFFER="$2"
-RBUFFER="$3"
-PREDISPLAY="$PREDISPLAY$savelbuffer$saverbuffer$POSTDISPLAY
-${1:-? }"
-POSTDISPLAY=
+  local pretext="$PREDISPLAY$LBUFFER$RBUFFER$POSTDISPLAY
+"
+local LBUFFER="$2"
+local RBUFFER="$3"
+local PREDISPLAY="$pretext${1:-? }"
+local POSTDISPLAY=
+local -a region_highlight
+region_highlight=("P${#pretext} ${#PREDISPLAY} bold")
 
 if [[ -n $keys ]]; then
   zle -R
@@ -38,9 +38,4 @@
   (( stat )) || REPLY=$BUFFER
 fi
 
-LBUFFER=$savelbuffer
-RBUFFER=$saverbuffer
-PREDISPLAY=$savepredisplay
-POSTDISPLAY=$savepostdisplay
-
 return $stat
Index: Functions/Zle/replace-string-again
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/replace-string-again,v
retrieving revision 1.1
diff -u -r1.1 replace-string-again
--- Functions/Zle/replace-string-again	10 Oct 2006 11:24:38 -0000	1.1
+++ Functions/Zle/replace-string-again	2 Apr 2008 19:31:57 -0000
@@ -9,7 +9,8 @@
 local -a match mbegin mend
 
 if [[ -z $_replace_string_src ]]; then
-  zle -M No string to replace.
+  zle -M "No string to replace."
+  return 1
 fi
 
 if [[ $curwidget = *pattern* ]]; then
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.119
diff -u -r1.119 zsh.h
--- Src/zsh.h	14 Mar 2008 11:40:59 -0000	1.119
+++ Src/zsh.h	2 Apr 2008 19:31:59 -0000
@@ -1941,6 +1941,8 @@
 #define TXTUNDERLINE  0x04
 #define TXTDIRTY      0x80
 
+#define TXT_ATTR_ON_MASK   0x07
+
 #define txtisset(X)  (txtattrmask & (X))
 #define txtset(X)    (txtattrmask |= (X))
 #define txtunset(X)  (txtattrmask &= ~(X))
@@ -1949,7 +1951,11 @@
 #define TXTNOSTANDOUT	0x20
 #define TXTNOUNDERLINE	0x40
 
-#define txtchangeisset(X)	(txtchange & (X))
+#define TXT_ATTR_OFF_MASK  0x70
+/* Bits to shift off right to get on */
+#define TXT_ATTR_OFF_ON_SHIFT (4)
+
+#define txtchangeisset(T,X)	((T) & (X))
 #define txtchangeset(X, Y)	(txtchange |= (X), txtchange &= ~(Y))
 
 /****************************************/
Index: Src/Zle/zle.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle.h,v
retrieving revision 1.35
diff -u -r1.35 zle.h
--- Src/Zle/zle.h	15 Apr 2007 21:09:37 -0000	1.35
+++ Src/Zle/zle.h	2 Apr 2008 19:31:59 -0000
@@ -325,6 +325,37 @@
     SUFTYP_NEGRNG		/* Range of characters not to match */
 };
 
+
+#ifdef MULTIBYTE_SUPPORT
+/*
+ * We use a wint_t here, since we need an invalid character as a
+ * placeholder and wint_t guarantees that we can use WEOF to do this.
+ */
+typedef wint_t REFRESH_CHAR;
+#else
+typedef char REFRESH_CHAR;
+#endif
+
+/*
+ * Description of one screen cell in zle_refresh.c
+ */
+typedef struct {
+    /* The (possibly wide) character */
+    REFRESH_CHAR chr;
+    /*
+     * Its attributes.  'On' attributes (TXT_ATTR_ON_MASK) are
+     * applied before the character, 'off' attributes (TXT_ATTR_OFF_MASK)
+     * after it.  'On' attributes are present for all characters that
+     * need the effect; 'off' attributes are only present for the
+     * last character in the sequence.
+     */
+    REFRESH_CHAR atr;
+} REFRESH_ELEMENT;
+
+/* A string of screen cells */
+typedef REFRESH_ELEMENT *REFRESH_STRING;
+
+
 #ifdef DEBUG
 #define METACHECK()		\
 	DPUTS(zlemetaline == NULL, "line not metafied")
Index: Src/Zle/zle_main.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_main.c,v
retrieving revision 1.104
diff -u -r1.104 zle_main.c
--- Src/Zle/zle_main.c	11 Mar 2008 14:31:10 -0000	1.104
+++ Src/Zle/zle_main.c	2 Apr 2008 19:32:00 -0000
@@ -1076,6 +1076,7 @@
 	freeheap();
     }
 
+    region_active = 0;
     popheap();
 }
 
@@ -1933,6 +1934,7 @@
     getkeyptr = NULL;
 
     zfree(clwords, clwsize * sizeof(char *));
+    zle_refresh_finish();
 
     return 0;
 }
Index: Src/Zle/zle_move.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_move.c,v
retrieving revision 1.9
diff -u -r1.9 zle_move.c
--- Src/Zle/zle_move.c	15 Apr 2007 20:04:58 -0000	1.9
+++ Src/Zle/zle_move.c	2 Apr 2008 19:32:00 -0000
@@ -183,7 +183,12 @@
 int
 setmarkcommand(UNUSED(char **args))
 {
+    if (zmult < 0) {
+	region_active = 0;
+	return 0;
+    }
     mark = zlecs;
+    region_active = 1;
     return 0;
 }
 
@@ -193,11 +198,17 @@
 {
     int x;
 
+    if (zmult == 0) {
+	region_active = 1;
+	return 0;
+    }
     x = mark;
     mark = zlecs;
     zlecs = x;
     if (zlecs > zlell)
 	zlecs = zlell;
+    if (zmult > 0)
+	region_active = 1;
     return 0;
 }
 
Index: Src/Zle/zle_params.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_params.c,v
retrieving revision 1.30
diff -u -r1.30 zle_params.c
--- Src/Zle/zle_params.c	19 Nov 2007 16:53:09 -0000	1.30
+++ Src/Zle/zle_params.c	2 Apr 2008 19:32:00 -0000
@@ -90,6 +90,9 @@
 
 static const struct gsu_array killring_gsu =
 { get_killring, set_killring, unset_killring };
+/* implementation is in zle_refresh.c */
+static const struct gsu_array region_highlight_gsu =
+{ get_region_highlight, set_region_highlight, unset_region_highlight };
 
 #define GSU(X) ( (GsuScalar)(void*)(&(X)) )
 static struct zleparam {
@@ -120,6 +123,7 @@
     { "PREBUFFER",  PM_SCALAR | PM_READONLY,  GSU(prebuffer_gsu), NULL },
     { "PREDISPLAY", PM_SCALAR, GSU(predisplay_gsu), NULL },
     { "RBUFFER", PM_SCALAR,  GSU(rbuffer_gsu), NULL },
+    { "region_highlight", PM_ARRAY, GSU(region_highlight_gsu), NULL },
     { "WIDGET", PM_SCALAR | PM_READONLY, GSU(widget_gsu), NULL },
     { "WIDGETFUNC", PM_SCALAR | PM_READONLY, GSU(widgetfunc_gsu), NULL },
     { "WIDGETSTYLE", PM_SCALAR | PM_READONLY, GSU(widgetstyle_gsu), NULL },
Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.53
diff -u -r1.53 zle_refresh.c
--- Src/Zle/zle_refresh.c	22 Jan 2008 10:14:02 -0000	1.53
+++ Src/Zle/zle_refresh.c	2 Apr 2008 19:32:01 -0000
@@ -29,67 +29,53 @@
 
 #include "zle.mdh"
 
-#ifdef MULTIBYTE_SUPPORT
-/*
- * We use a wint_t here, since we need an invalid character as a
- * placeholder and wint_t guarantees that we can use WEOF to do this.
- */
-typedef wint_t *REFRESH_STRING;
-typedef wint_t REFRESH_CHAR;
+#define ZR_equal(zr1, zr2) ((zr1).chr == (zr2).chr && (zr1).atr == (zr2).atr)
 
-/*
- * Unfortunately, that means the pointer is the wrong type for
- * wmemset and friends.
- */
 static void
-ZR_memset(wint_t *dst, wchar_t wc, int len)
+ZR_memset(REFRESH_ELEMENT *dst, REFRESH_ELEMENT rc, int len)
 {
     while (len--)
-	*dst++ = wc;
+	*dst++ = rc;
 }
-#define ZR_memcpy(d, s, l)  memcpy((d), (s), (l)*sizeof(wint_t))
+
+#define ZR_memcpy(d, s, l)  memcpy((d), (s), (l)*sizeof(REFRESH_ELEMENT))
+
 static void
-ZR_strcpy(wint_t *dst, wint_t *src)
+ZR_strcpy(REFRESH_ELEMENT *dst, const REFRESH_ELEMENT *src)
 {
-    while ((*dst++ = *src++) != L'\0')
+    while ((*dst++ = *src++).chr != ZWC('\0'))
 	;
 }
+
 static size_t
-ZR_strlen(wint_t *wstr)
+ZR_strlen(const REFRESH_ELEMENT *wstr)
 {
     int len = 0;
 
-    while (*wstr++ != L'\0')
+    while (wstr++->chr != ZWC('\0'))
 	len++;
 
     return len;
 }
+
 /*
  * Simplified strcmp: we don't need the sign, just whether
- * the strings are equal.
+ * the strings and their attributes are equal.
  */
 static int
-ZR_strncmp(wint_t *wstr1, wint_t *wstr2, int len)
+ZR_strncmp(const REFRESH_ELEMENT *wstr1, const REFRESH_ELEMENT *wstr2, int len)
 {
     while (len--) {
-	if (!*wstr1 || !*wstr2)
-	    return (*wstr1 == *wstr2) ? 0 : 1;
-	if (*wstr1++ != *wstr2++)
+	if (!wstr1->chr || !wstr2->chr)
+	    return !ZR_equal(*wstr1, *wstr2);
+	if (!ZR_equal(*wstr1, *wstr2))
 	    return 1;
+	wstr1++;
+	wstr2++;
     }
 
     return 0;
 }
-#else
-typedef char *REFRESH_STRING;
-typedef char REFRESH_CHAR;
-
-#define ZR_memset	memset
-#define ZR_memcpy	memcpy
-#define ZR_strcpy	strcpy
-#define ZR_strlen	strlen
-#define ZR_strncmp	strncmp
-#endif
 
 #include "zle_refresh.pro"
 
@@ -160,52 +146,408 @@
 int predisplaylen, postdisplaylen;
 
 
+/*
+ * Attributes used for highlighting special (unprintable) characters
+ * displayed on screen.
+ */
+
+static int special_atr_on, special_atr_off;
+
+/* Flags for the region_highlight structure */
+enum {
+    /* Offsets include predisplay */
+    ZRH_PREDISPLAY = 1
+};
+
+/*
+ * Attributes used for highlighting regions.
+ * and mark.
+ */
+struct region_highlight {
+    /* Attributes turned on in the region */
+    int atr;
+    /* Start of the region */
+    int start;
+    /*
+     * End of the region:  position of the first character not highlighted
+     * (the same system as for point and mark).
+     */
+    int end;
+    /*
+     * Any of the flags defined above.
+     */
+    int flags;
+};
+/*
+ * Array of region highlights, no special termination.
+ * The first element (0) always describes the region between
+ * point and mark.  Any other elements are set by the user
+ * via the parameter region_highlight.
+ */
+struct region_highlight *region_highlights;
+/*
+ * Number of elements in region_highlights.
+ * This includes the region between point and mark, element 0.
+ */
+int n_region_highlights;
+
+/*
+ * Flag that highlighting of the region is active.
+ */
+/**/
+int region_active;
+
 #ifdef HAVE_SELECT
 /* cost of last update */
 /**/
 int cost;
 
-# define SELECT_ADD_COST(X)	cost += X
-# define zputc(a)		zwcputc(a), cost++
-# define zwrite(a, b)		zwcwrite(a, b), cost += (b * ZLE_CHAR_SIZE)
+# define SELECT_ADD_COST(X)	(cost += X)
+# define zputc(a)		(zwcputc(a, NULL), cost++)
+# define zwrite(a, b)		(zwcwrite((a), (b)), \
+				 cost += ((b) * ZLE_CHAR_SIZE))
 #else
 # define SELECT_ADD_COST(X)
-# define zputc(a)		zwcputc(a)
-# define zwrite(a, b)		zwcwrite(a, b)
+# define zputc(a)		zwcputc(a, NULL)
+# define zwrite(a, b)		zwcwrite((a), (b))
 #endif
 
+static const REFRESH_ELEMENT zr_cr = { ZWC('\r'), 0 };
+static const REFRESH_ELEMENT zr_dt = { ZWC('.'), 0 };
+static const REFRESH_ELEMENT zr_nl = { ZWC('\n'), 0 };
+static const REFRESH_ELEMENT zr_sp = { ZWC(' '), 0 };
+static const REFRESH_ELEMENT zr_ht = { ZWC('\t'), 0 };
+static const REFRESH_ELEMENT zr_zr = { ZWC('\0'), 0 };
+
+/*
+ * Constant arrays to be copied into place: these are memcpy'd,
+ * so don't have terminating NULLs.
+ */
+static const REFRESH_ELEMENT zr_end_ellipsis[] = {
+    { ZWC(' '), 0 },
+    { ZWC('<'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC(' '), 0 },
+};
+#define ZR_END_ELLIPSIS_SIZE	\
+    (sizeof(zr_end_ellipsis)/sizeof(zr_end_ellipsis[0]))
+
+static const REFRESH_ELEMENT zr_mid_ellipsis1[] = {
+    { ZWC(' '), 0 },
+    { ZWC('<'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+};
+#define ZR_MID_ELLIPSIS1_SIZE	\
+    (sizeof(zr_mid_ellipsis1)/sizeof(zr_mid_ellipsis1[0]))
+
+static const REFRESH_ELEMENT zr_mid_ellipsis2[] = {
+    { ZWC('>'), 0 },
+    { ZWC(' '), 0 },
+};
+#define ZR_MID_ELLIPSIS2_SIZE	\
+    (sizeof(zr_mid_ellipsis2)/sizeof(zr_mid_ellipsis2[0]))
+
+static const REFRESH_ELEMENT zr_start_ellipsis[] = {
+    { ZWC('>'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+    { ZWC('.'), 0 },
+};
+#define ZR_START_ELLIPSIS_SIZE	\
+    (sizeof(zr_start_ellipsis)/sizeof(zr_start_ellipsis[0]))
+
+
+/* Defines the available types of highlighting */
+struct highlight {
+    const char *name;
+    int mask_on;
+    int mask_off;
+};
+
+static const struct highlight highlights[] = {
+    { "none", 0, TXT_ATTR_ON_MASK },
+    { "bold", TXTBOLDFACE, 0 },
+    { "standout", TXTSTANDOUT, 0 },
+    { "underline", TXTUNDERLINE, 0 },
+    { NULL, 0, 0 }
+};
+
+/*
+ * Match a set of highlights in the given teststr.
+ * Set *on_var to reflect the values found.
+ */
+
+static void
+match_highlight(const char *teststr, int *on_var)
+{
+    int found = 1;
+
+    *on_var = 0;
+    while (found && *teststr) {
+	const struct highlight *hl;
+
+	found = 0;
+	for (hl = highlights; hl->name; hl++) {
+	    if (strpfx(hl->name, teststr)) {
+		const char *val = teststr + strlen(hl->name);
+
+		if (*val == ',')
+		    val++;
+		else if (*val)
+		    break;
+
+		*on_var |= hl->mask_on;
+		*on_var &= ~hl->mask_off;
+		teststr = val;
+		found = 1;
+	    }
+	}
+    }
+}
+
+
+/*
+ * Parse the variable zle_highlight to decide how to highlight characters
+ * and regions.  Set defaults for anything not explicitly covered.
+ */
+
+/**/
+void zle_set_highlight(void)
+{
+    char **atrs = getaparam("zle_highlight");
+    int special_atr_on_set = 0;
+    int region_atr_on_set = 0;
+
+    special_atr_on = 0;
+    if (!region_highlights) {
+	region_highlights = (struct region_highlight *)
+	    zshcalloc(sizeof(struct region_highlight));
+	n_region_highlights = 1;
+    } else {
+	region_highlights->atr = 0;
+    }
+
+    if (atrs) {
+	for (; *atrs; atrs++) {
+	    if (!strcmp(*atrs, "none")) {
+		/* reset attributes for consistency... usually unnecessary */
+		special_atr_on = region_highlights->atr = 0;
+		special_atr_on_set = region_atr_on_set = 1;
+	    } else if (strpfx("special:", *atrs)) {
+		match_highlight(*atrs + 8, &special_atr_on);
+		special_atr_on_set = 1;
+	    } else if (strpfx("region:", *atrs)) {
+		match_highlight(*atrs + 7, &region_highlights->atr);
+		region_atr_on_set = 1;
+	    }
+	}
+    }
+
+    /* Defaults */
+    if (!special_atr_on_set)
+	special_atr_on = TXTSTANDOUT;
+    if (!region_atr_on_set)
+	region_highlights->atr = TXTSTANDOUT;
+    special_atr_off = special_atr_on << TXT_ATTR_OFF_ON_SHIFT;
+}
+
+
+/*
+ * Interface to the region_highlight ZLE parameter.
+ * Converts betwen a format like "P32 42 underline,bold" to
+ * the format in the region_highlights variable.  Note that
+ * the region_highlights variable stores the internal (point/mark)
+ * region in element zero.
+ */
+
+/**/
+char **
+get_region_highlight(UNUSED(Param pm))
+{
+    int arrsize = n_region_highlights;
+    char **retarr, **arrp;
+
+    /* region_highlights may not have been set yet */
+    if (!arrsize)
+	arrsize = 1;
+    arrp = retarr = (char **)zhalloc(arrsize);
+    /* ignore NULL termination */
+    arrsize--;
+    if (arrsize) {
+	struct region_highlight *rhp;
+
+	/* ignore point/mark at start */
+	for (rhp = region_highlights+1; arrsize--; rhp++, arrp++) {
+	    char digbuf1[DIGBUFSIZE], digbuf2[DIGBUFSIZE];
+	    int atrlen = 0, alloclen, done1;
+	    const struct highlight *hp;
+
+	    sprintf(digbuf1, "%d", rhp->start);
+	    sprintf(digbuf2, "%d", rhp->end);
+
+	    for (hp = highlights; hp->name; hp++) {
+		if (hp->mask_on & rhp->atr) {
+		    if (atrlen)
+			atrlen++; /* comma */
+		    atrlen += strlen(hp->name);
+		}
+	    }
+	    if (atrlen == 0)
+		atrlen = 4; /* none */
+	    alloclen = atrlen + strlen(digbuf1) + strlen(digbuf2) +
+		3; /* 2 spaces, 1 0 */
+	    if (rhp->flags & ZRH_PREDISPLAY)
+		alloclen += 2; /* "P " */
+	    *arrp = (char *)zhalloc(alloclen * sizeof(char));
+	    /*
+	     * On input we allow a space after the flags.
+	     * I haven't put a space here because I think it's
+	     * marginally easier to have the output always split
+	     * into three words, and then check the first to
+	     * see if there are flags.  However, it's arguable.
+	     */
+	    sprintf(*arrp, "%s%s %s ", 
+		    (rhp->flags & ZRH_PREDISPLAY) ? "P" : "",
+		    digbuf1, digbuf2);
+	    if (atrlen) {
+		for (hp = highlights, done1 = 0; hp->name; hp++) {
+		    if (hp->mask_on & rhp->atr) {
+			if (done1)
+			    strcat(*arrp, ",");
+			strcat(*arrp, hp->name);
+			done1 = 1;
+		    }
+		}
+	    } else
+		strcat(*arrp, "none");
+	    arrp++;
+	}
+    }
+    *arrp = '\0';
+    return retarr;
+}
+
+
 /**/
 void
-zwcputc(ZLE_INT_T c)
+set_region_highlight(UNUSED(Param pm), char **aval)
+{
+    int len;
+    struct region_highlight *rhp;
+
+    len = aval ? arrlen(aval) : 0;
+    if (n_region_highlights != len + 1) {
+	/* no null termination, but include point/mark region at start */
+	n_region_highlights = len + 1;
+	region_highlights = (struct region_highlight *)
+	    zrealloc(region_highlights,
+		     sizeof(struct region_highlight) * n_region_highlights);
+    }
+
+    if (!aval)
+	return;
+
+    for (rhp = region_highlights + 1; *aval; rhp++, aval++) {
+	char *strp, *oldstrp;
+
+	oldstrp = *aval;
+	if (*oldstrp == 'P') {
+	    rhp->flags = ZRH_PREDISPLAY;
+	    oldstrp++;
+	}
+	else
+	    rhp->flags = 0;
+	while (inblank(*oldstrp))
+	    oldstrp++;
+
+	rhp->start = (int)zstrtol(oldstrp, &strp, 10);
+	if (strp == oldstrp)
+	    rhp->start = -1;
+
+	while (inblank(*strp))
+	    strp++;
+
+	oldstrp = strp;
+	rhp->end = (int)zstrtol(strp, &strp, 10);
+	if (strp == oldstrp)
+	    rhp->end = -1;
+
+	while (inblank(*strp))
+	    strp++;
+
+	match_highlight(strp, &rhp->atr);
+    }
+}
+
+
+/**/
+void
+unset_region_highlight(Param pm, int exp)
+{
+    if (exp) {
+	set_region_highlight(pm, NULL);
+	stdunsetfn(pm, exp);
+    }
+}
+
+
+/**/
+void
+zwcputc(const REFRESH_ELEMENT *c, REFRESH_CHAR *curatrp)
 {
 #ifdef MULTIBYTE_SUPPORT
     mbstate_t mbstate;
     int i;
     VARARR(char, mbtmp, MB_CUR_MAX + 1);
+#endif
 
-    if (c == WEOF)
-	return;
+    /*
+     * Don't output "on" attributes in a string of characters with
+     * the same attributes.
+     */
+    if ((c->atr & TXT_ATTR_ON_MASK) &&
+	(!curatrp ||
+	 ((*curatrp & TXT_ATTR_ON_MASK) != (c->atr & TXT_ATTR_ON_MASK))))
+	settextattributes(c->atr & TXT_ATTR_ON_MASK);
 
-    memset(&mbstate, 0, sizeof(mbstate_t));
-    if ((i = wcrtomb(mbtmp, (wchar_t)c, &mbstate)) > 0)
-	fwrite(mbtmp, i, 1, shout);
+#ifdef MULTIBYTE_SUPPORT
+    if (c->chr != WEOF) {
+	memset(&mbstate, 0, sizeof(mbstate_t));
+	if ((i = wcrtomb(mbtmp, (wchar_t)c->chr, &mbstate)) > 0)
+	    fwrite(mbtmp, i, 1, shout);
+    }
 #else
-    fputc(c, shout);
+    fputc(c->chr, shout);
 #endif
+
+    if (c->atr & TXT_ATTR_OFF_MASK)
+	settextattributes(c->atr & TXT_ATTR_OFF_MASK);
+    if (curatrp) {
+	/*
+	 * Remember the current attributes:  those that are turned
+	 * on, less those that are turned off again.
+	 */
+	*curatrp = (c->atr & TXT_ATTR_ON_MASK) &
+	    ~((c->atr & TXT_ATTR_OFF_MASK) >> TXT_ATTR_OFF_ON_SHIFT);
+    }
 }
 
 static int
-zwcwrite(REFRESH_STRING s, size_t i)
+zwcwrite(const REFRESH_STRING s, size_t i)
 {
-#ifdef MULTIBYTE_SUPPORT
     size_t j;
+    REFRESH_CHAR curatr = 0;
 
     for (j = 0; j < i; j++)
-	zwcputc(s[j]);
+	zwcputc(s + j, &curatr);
     return i; /* TODO something better for error indication */
-#else
-    return fwrite(s, i, 1, shout);
-#endif
 }
 
 /* Oct/Nov 94: <mason> some code savagely redesigned to fix several bugs -
@@ -214,8 +556,8 @@
    any queries about updates to mason@primenet.com.au */
 
 static REFRESH_STRING 
-    *nbuf = NULL,		/* new video buffer line-by-line char array */
-    *obuf = NULL;		/* old video buffer line-by-line char array */
+    *nbuf = NULL,		/* new video buffer line-by-line array */
+    *obuf = NULL;		/* old video buffer line-by-line array */
 static int more_start,		/* more text before start of screen?	    */
     more_end,			/* more stuff after end of screen?	    */
     olnct,			/* previous number of lines		    */
@@ -228,14 +570,31 @@
     vmaxln,			/* video maximum number of lines	    */
     winw, winh, rwinh,		/* window width & height		    */
     winpos,			/* singlelinezle: line's position in window */
-    winprompt;			/* singlelinezle: part of lprompt showing   */
+    winprompt,			/* singlelinezle: part of lprompt showing   */
+    winw_alloc = -1,		/* allocated window width */
+    winh_alloc = -1;		/* allocates window height */
+
+static void
+freevideo(void)
+{
+    if (nbuf) {
+	int ln;
+	for (ln = 0; ln != winh_alloc; ln++) {
+	    zfree(nbuf[ln], (winw_alloc + 2) * sizeof(**nbuf));
+	    zfree(obuf[ln], (winw_alloc + 2) * sizeof(**obuf));
+	}
+	free(nbuf);
+	free(obuf);
+	nbuf = NULL;
+	obuf = NULL;
+    }
+}
 
 /**/
 void
 resetvideo(void)
 {
     int ln;
-    static int lwinw = -1, lwinh = -1;	/* last window width & height */
  
     winw = columns;  /* terminal width */
     if (termflags & TERM_SHORT)
@@ -245,31 +604,24 @@
     rwinh = lines;		/* keep the real number of lines */
     vln = vmaxln = winprompt = 0;
     winpos = -1;
-    if (lwinw != winw || lwinh != winh) {
-	if (nbuf) {
-	    for (ln = 0; ln != lwinh; ln++) {
-		zfree(nbuf[ln], (lwinw + 2) * sizeof(**nbuf));
-		zfree(obuf[ln], (lwinw + 2) * sizeof(**obuf));
-	    }
-	    free(nbuf);
-	    free(obuf);
-	}
+    if (winw_alloc != winw || winh_alloc != winh) {
+	freevideo();
 	nbuf = (REFRESH_STRING *)zshcalloc((winh + 1) * sizeof(*nbuf));
 	obuf = (REFRESH_STRING *)zshcalloc((winh + 1) * sizeof(*obuf));
 	nbuf[0] = (REFRESH_STRING)zalloc((winw + 2) * sizeof(**nbuf));
 	obuf[0] = (REFRESH_STRING)zalloc((winw + 2) * sizeof(**obuf));
 
-	lwinw = winw;
-	lwinh = winh;
+	winw_alloc = winw;
+	winh_alloc = winh;
     }
     for (ln = 0; ln != winh + 1; ln++) {
 	if (nbuf[ln]) {
-	    nbuf[ln][0] = ZWC('\n');
-	    nbuf[ln][1] = ZWC('\0');
+	    nbuf[ln][0] = zr_nl;
+	    nbuf[ln][1] = zr_zr;
 	}
 	if (obuf[ln]) {
-	    obuf[ln][0] = ZWC('\n');
-	    obuf[ln][1] = ZWC('\0');
+	    obuf[ln][0] = zr_nl;
+	    obuf[ln][1] = zr_zr;
 	}
     }
 
@@ -286,9 +638,9 @@
     }
 
     if (lpromptw) {
-    	ZR_memset(nbuf[0], ZWC(' '), lpromptw);
-	ZR_memset(obuf[0], ZWC(' '), lpromptw);
-	nbuf[0][lpromptw] = obuf[0][lpromptw] = ZWC('\0');
+    	ZR_memset(nbuf[0], zr_sp, lpromptw);
+	ZR_memset(obuf[0], zr_sp, lpromptw);
+	nbuf[0][lpromptw] = obuf[0][lpromptw] = zr_zr;
     }
 
     vcs = lpromptw;
@@ -350,8 +702,8 @@
 static int
 nextline(Rparams rpms, int wrapped)
 {
-    nbuf[rpms->ln][winw+1] = wrapped ? ZWC('\n') : ZWC('\0');
-    *rpms->s = ZWC('\0');
+    nbuf[rpms->ln][winw+1] = wrapped ? zr_nl : zr_zr;
+    *rpms->s = zr_zr;
     if (rpms->ln != winh - 1)
 	rpms->ln++;
     else {
@@ -383,7 +735,7 @@
 static void
 snextline(Rparams rpms)
 {
-    *rpms->s = ZWC('\0');
+    *rpms->s = zr_zr;
     if (rpms->ln != winh - 1)
 	rpms->ln++;
     else
@@ -415,19 +767,19 @@
 
 /**/
 static void
-settextattributes(void)
+settextattributes(int atr)
 {
-    if (txtchangeisset(TXTNOBOLDFACE))
+    if (txtchangeisset(atr, TXTNOBOLDFACE))
 	tsetcap(TCALLATTRSOFF, 0);
-    if (txtchangeisset(TXTNOSTANDOUT))
+    if (txtchangeisset(atr, TXTNOSTANDOUT))
 	tsetcap(TCSTANDOUTEND, 0);
-    if (txtchangeisset(TXTNOUNDERLINE))
+    if (txtchangeisset(atr, TXTNOUNDERLINE))
 	tsetcap(TCUNDERLINEEND, 0);
-    if (txtchangeisset(TXTBOLDFACE))
+    if (txtchangeisset(atr, TXTBOLDFACE))
 	tsetcap(TCBOLDFACEBEG, 0);
-    if (txtchangeisset(TXTSTANDOUT))
+    if (txtchangeisset(atr, TXTSTANDOUT))
 	tsetcap(TCSTANDOUTBEG, 0);
-    if (txtchangeisset(TXTUNDERLINE))
+    if (txtchangeisset(atr, TXTUNDERLINE))
 	tsetcap(TCUNDERLINEBEG, 0);
 }
 
@@ -435,20 +787,21 @@
 mod_export void
 zrefresh(void)
 {
-    static int inlist;		/* avoiding recursion                        */
-    int iln;			/* current line as index in loops            */
+    static int inlist;		/* avoiding recursion			     */
+    int iln;			/* current line as index in loops	     */
     int t0 = -1;		/* tmp					     */
-    ZLE_STRING_T tmpline,	/* line with added pre/post text             */
+    ZLE_STRING_T tmpline,	/* line with added pre/post text	     */
 	t,			/* pointer into the real buffer		     */
 	scs,			/* pointer to cursor position in real buffer */
-	u;			/* pointer for status line stuff             */
-    REFRESH_STRING 	*qbuf;	/* tmp					     */
+	u;			/* pointer for status line stuff	     */
+    REFRESH_STRING	*qbuf;	/* tmp					     */
     int tmpcs, tmpll;		/* ditto cursor position and line length     */
-    int tmpalloced;		/* flag to free tmpline when finished        */
-    int remetafy;		/* flag that zle line is metafied            */
+    int tmppos;			/* t - tmpline				     */
+    int tmpalloced;		/* flag to free tmpline when finished	     */
+    int remetafy;		/* flag that zle line is metafied	     */
     struct rparams rpms;
 #ifdef MULTIBYTE_SUPPORT
-    int width;                  /* width of wide character                   */
+    int width;			/* width of wide character		     */
 #endif
 
     
@@ -491,6 +844,21 @@
 	tmpalloced = 0;
     }
 
+    /* this will create region_highlights if it's still NULL */
+    zle_set_highlight();
+
+    if (region_active) {
+	if (zlecs <= mark) {
+	    region_highlights->start = zlecs;
+	    region_highlights->end = mark;
+	} else {
+	    region_highlights->start = mark;
+	    region_highlights->end = zlecs;
+	}
+    } else {
+	region_highlights->start = region_highlights->end = -1;
+    }
+
     if (clearlist && listshown > 0) {
 	if (tccan(TCCLEAREOD)) {
 	    int ovln = vln, ovcs = vcs;
@@ -574,10 +942,10 @@
 		zputs("\n", shout);	/* works with both hasam and !hasam */
 	} else {
 	    txtchange = pmpt_attr;
-	    settextattributes();
+	    settextattributes(txtchange);
 	}
 	if (clearflag) {
-	    zputc(ZWC('\r'));
+	    zputc(&zr_cr);
 	    vcs = 0;
 	    moveto(0, lpromptw);
 	}
@@ -615,9 +983,33 @@
     rpms.nvln = -1;
 
     rpms.s = nbuf[rpms.ln = 0] + lpromptw;
-    t = tmpline;
     rpms.sen = *nbuf + winw;
-    for (; t < tmpline+tmpll; t++) {
+    for (t = tmpline, tmppos = 0; tmppos < tmpll; t++, tmppos++) {
+	int base_atr_on = 0, base_atr_off = 0, ireg;
+	struct region_highlight *rhp;
+	/*
+	 * Calculate attribute based on region.
+	 * HERE: we may need to be smarter about turning
+	 * attributes off if bailing out before the end of the
+	 * region.
+	 */
+	for (ireg = 0, rhp = region_highlights;
+	     ireg < n_region_highlights;
+	     ireg++, rhp++) {
+	    int offset;
+	    if (rhp->flags & ZRH_PREDISPLAY)
+		offset = 0;	/* include predisplay in start end */
+	    else
+		offset = predisplaylen; /* increment over it */
+	    if (rhp->start + offset <= tmppos &&
+		tmppos < rhp->end + offset) {
+		base_atr_on |= rhp->atr;
+		if (tmppos == rhp->end + offset - 1 ||
+		    tmppos == tmpll - 1)
+		    base_atr_off |= rhp->atr << TXT_ATTR_OFF_ON_SHIFT;
+	    }
+	}
+
 	if (t == scs)			/* if cursor is here, remember it */
 	    rpms.nvcs = rpms.s - nbuf[rpms.nvln = rpms.ln];
 
@@ -631,20 +1023,32 @@
 		/* text wrapped */
 		if (nextline(&rpms, 1))
 		    break;
-	    } else
-		do
-		    *rpms.s++ = ZWC(' ');
-		while ((++t0) & 7);
+	    } else {
+		do {
+		    rpms.s->chr = ZWC(' ');
+		    rpms.s->atr = base_atr_on;
+		    rpms.s++;
+		} while ((++t0) & 7);
+		rpms.s[-1].atr |= base_atr_off;
+	    }
 	}
 #ifdef MULTIBYTE_SUPPORT
 	else if (iswprint(*t) && (width = wcwidth(*t)) > 0) {
 	    if (width > rpms.sen - rpms.s) {
+		int started = 0;
 		/*
 		 * Too wide to fit.  Insert spaces to end of current line.
 		 */
 		do {
-		    *rpms.s++ = ZWC(' ');
+		    /* HERE highlight */
+		    rpms.s->chr = ZWC(' ');
+		    if (!started)
+			started = 1;
+		    rpms.s->atr = special_atr_on | base_atr_on;
+		    rpms.s++;
 		} while (rpms.s < rpms.sen);
+		if (started)
+		    rpms.s[-1].atr |= special_atr_off | base_atr_off;
 		if (nextline(&rpms, 1))
 		    break;
 		if (t == scs) {
@@ -656,13 +1060,28 @@
 		/*
 		 * The screen width is too small to fit even one
 		 * occurrence.
+		 *
+		 * HERE highlight
 		 */
-		*rpms.s++ = ZWC('?');
+		rpms.s->chr = ZWC('?');
+		rpms.s->atr = special_atr_on | special_atr_off |
+		    base_atr_on | base_atr_off;
+		rpms.s++;
 	    } else {
 		/* We can fit it without reaching the end of the line. */
-		*rpms.s++ = *t;
-		while (--width > 0)
-		    *rpms.s++ = WEOF;
+		rpms.s->chr = *t;
+		/*
+		 * As we don't actually output the WEOF, we attach
+		 * any off attributes to the character itself.
+		 */
+		rpms.s->atr = base_atr_on | base_atr_off;
+		rpms.s++;
+		while (--width > 0) {
+		    rpms.s->chr = WEOF;
+		    /* Not used, but be consistent... */
+		    rpms.s->atr = base_atr_on | base_atr_off;
+		    rpms.s++;
+		}
 	    }
 	}
 #endif
@@ -671,23 +1090,34 @@
 		 && (unsigned)*t <= 0xffU
 #endif
 	    ) {	/* other control character */
-	    *rpms.s++ = ZWC('^');
+	    /* HERE highlight */
+	    rpms.s->chr = ZWC('^');
+	    rpms.s->atr = special_atr_on | base_atr_on;
+	    rpms.s++;
 	    if (rpms.s == rpms.sen) {
 		/* text wrapped */
+		rpms.s[-1].atr |= special_atr_off | base_atr_off;
 		if (nextline(&rpms, 1))
 		    break;
 	    }
-	    *rpms.s++ = (((unsigned int)*t & ~0x80u) > 31) ? ZWC('?') : (*t | ZWC('@'));
+	    rpms.s->chr = (((unsigned int)*t & ~0x80u) > 31) ?
+		ZWC('?') : (*t | ZWC('@'));
+	    rpms.s->atr = special_atr_on | special_atr_off |
+		base_atr_on | base_atr_off;
+	    rpms.s++;
 	}
 #ifdef MULTIBYTE_SUPPORT
 	else {
 	    /*
 	     * Not printable or zero width.
 	     * Resort to hackery.
+	     *
+	     * HERE: highlight
 	     */
 	    char dispchars[11];
 	    char *dispptr = dispchars;
 	    wchar_t wc;
+	    int started = 0;
 
 	    if ((unsigned)*t > 0xffffU) {
 		sprintf(dispchars, "<%.08x>", (unsigned)*t);
@@ -697,21 +1127,33 @@
 	    while (*dispptr) {
 		if (mbtowc(&wc, dispptr, 1) == 1 /* paranoia */)
 		{
-		    *rpms.s++ = wc;
+		    rpms.s->chr = wc;
+		    if (!started)
+			started = 1;
+		    rpms.s->atr = special_atr_on | base_atr_on;
+		    rpms.s++;
 		    if (rpms.s == rpms.sen) {
 			/* text wrapped */
+			if (started) {
+			    rpms.s[-1].atr |= special_atr_off | base_atr_off;
+			    started = 0;
+			}
 			if (nextline(&rpms, 1))
 			    break;
 		    }
 		}
 		dispptr++;
 	    }
+	    if (started)
+		rpms.s[-1].atr |= special_atr_off | base_atr_off;
 	    if (*dispptr) /* nextline said stop processing */
 		break;
 	}
 #else
 	else {			/* normal character */
-	    *rpms.s++ = *t;
+	    rpms.s->chr = *t;
+	    rpms.s->atr = base_atr_on | base_atr_off;
+	    rpms.s++;
 	}
 #endif
 	if (rpms.s == rpms.sen) {
@@ -726,7 +1168,7 @@
 	(rpms.nvcs = rpms.s - (nbuf[rpms.nvln = rpms.ln])) == winw) {
 	/* text wrapped */
 	(void)nextline(&rpms, 1);
-	*rpms.s = ZWC('\0');
+	*rpms.s = zr_zr;
 	rpms.nvcs = 0;
 	rpms.nvln++;
     }
@@ -736,7 +1178,7 @@
 
     if (statusline) {
 	rpms.tosln = rpms.ln + 1;
-	nbuf[rpms.ln][winw + 1] = ZWC('\0');	/* text not wrapped */
+	nbuf[rpms.ln][winw + 1] = zr_zr;	/* text not wrapped */
 	snextline(&rpms);
 	u = statusline;
 	for (; u < statusline + statusll; u++) {
@@ -746,32 +1188,49 @@
 		/* Handle wide characters as above */
 		if (width > rpms.sen - rpms.s) {
 		    do {
-			*rpms.s++ = ZWC(' ');
+			*rpms.s++ = zr_sp;
 		    } while (rpms.s < rpms.sen);
-		    nbuf[rpms.ln][winw + 1] = ZWC('\n');
+		    nbuf[rpms.ln][winw + 1] = zr_nl;
 		    snextline(&rpms);
 		}
 		if (width > rpms.sen - rpms.s) {
-		    *rpms.s++ = ZWC('?');
+		    /* HERE: highlight */
+		    rpms.s->chr = ZWC('?');
+		    rpms.s->atr = special_atr_on | special_atr_off;
+		    rpms.s++;
 		} else {
-		    *rpms.s++ = *u;
-		    while (--width > 0)
-			*rpms.s++ = WEOF;
+		    rpms.s->chr = *u;
+		    rpms.s->atr = 0;
+		    rpms.s++;
+		    while (--width > 0) {
+			rpms.s->chr = WEOF;
+			rpms.s->atr = 0;
+			rpms.s++;
+		    }
 		}
 	    }
 	    else
 #endif
 	    if (ZC_icntrl(*u)) { /* simplified processing in the status line */
-		*rpms.s++ = ZWC('^');
+		/* HERE: highlight */
+		rpms.s->chr = ZWC('^');
+		rpms.s->atr = special_atr_on;
+		rpms.s++;
 		if (rpms.s == rpms.sen) {
-		    nbuf[rpms.ln][winw + 1] = ZWC('\n');/* text wrapped */
+		    nbuf[rpms.ln][winw + 1] = zr_nl;/* text wrapped */
 		    snextline(&rpms);
 		}
-		*rpms.s++ = (((unsigned int)*u & ~0x80u) > 31) ? ZWC('?') : (*u | ZWC('@'));
-	    } else
-		*rpms.s++ = *u;
+		rpms.s->chr = (((unsigned int)*u & ~0x80u) > 31)
+		    ? ZWC('?') : (*u | ZWC('@'));
+		rpms.s->atr = special_atr_on | special_atr_off;
+		rpms.s++;
+	    } else {
+		rpms.s->chr = *u;
+		rpms.s->atr = 0;
+		rpms.s++;
+	    }
 	    if (rpms.s == rpms.sen) {
-		nbuf[rpms.ln][winw + 1] = ZWC('\n');	/* text wrapped */
+		nbuf[rpms.ln][winw + 1] = zr_nl;	/* text wrapped */
 		snextline(&rpms);
 	    }
 	}
@@ -783,42 +1242,82 @@
 	    snextline(&rpms);
 	}
     }
-    *rpms.s = ZWC('\0');
+    *rpms.s = zr_zr;
 
 /* insert <.... at end of last line if there is more text past end of screen */
-/* TODO: if we start overwriting in the middle of a wide character, mayhem
- * will ensue.
- */
     if (more_end) {
+#ifdef MULTIBYTE_SUPPORT
+	int extra_ellipsis = 0;
+#endif
 	if (!statusline)
 	    rpms.tosln = winh;
 	rpms.s = nbuf[rpms.tosln - 1];
 	rpms.sen = rpms.s + winw - 7;
 	for (; rpms.s < rpms.sen; rpms.s++) {
-	    if (*rpms.s == ZWC('\0')) {
-		for (; rpms.s < rpms.sen; )
-		    *rpms.s++ = ZWC(' ');
+	    if (rpms.s->chr == ZWC('\0')) {
+		ZR_memset(rpms.s, zr_sp, rpms.sen - rpms.s);
+		/* make sure we don't trigger the WEOF test */
+		rpms.sen->chr = ZWC('\0');
 		break;
 	    }
 	}
-	ZR_memcpy(rpms.sen, ZWS(" <.... "), 7);
-	nbuf[rpms.tosln - 1][winw] = nbuf[rpms.tosln - 1][winw + 1]
-	    = ZWC('\0');
+	/* rpms.s is no longer needed */
+#ifdef MULTIBYTE_SUPPORT
+	/*
+	 * Ensure we don't start overwriting in the middle of a wide
+	 * character.
+	 */
+	while(rpms.sen > nbuf[rpms.tosln - 1] && rpms.sen->chr == WEOF) {
+	    extra_ellipsis++;
+	    rpms.sen--;
+	}
+#endif
+	ZR_memcpy(rpms.sen, zr_end_ellipsis, ZR_END_ELLIPSIS_SIZE);
+#ifdef MULTIBYTE_SUPPORT
+	/* Extend to the end if we backed off for a wide character */
+	if (extra_ellipsis) {
+	    rpms.sen += ZR_END_ELLIPSIS_SIZE;
+	    ZR_memset(rpms.sen, zr_dt, extra_ellipsis);
+	}
+#endif
+	nbuf[rpms.tosln - 1][winw] = nbuf[rpms.tosln - 1][winw + 1] = zr_zr;
     }
 
 /* insert <....> at end of first status line if status is too big */
     if (rpms.more_status) {
+#ifdef MULTIBYTE_SUPPORT
+	int extra_ellipsis = 0;
+#endif
 	rpms.s = nbuf[rpms.tosln];
 	rpms.sen = rpms.s + winw - 8;
 	for (; rpms.s < rpms.sen; rpms.s++) {
-	    if (*rpms.s == ZWC('\0')) {
-		for (; rpms.s < rpms.sen; )
-		    *rpms.s++ = ZWC(' ');
+	    if (rpms.s->chr == ZWC('\0')) {
+		ZR_memset(rpms.s, zr_sp, rpms.sen - rpms.s);
 		break;
 	    }
 	}
-	ZR_memcpy(rpms.sen, ZWS(" <....> "), 8);
-	nbuf[rpms.tosln][winw] = nbuf[rpms.tosln][winw + 1] = ZWC('\0');
+	/* rpms.s is no longer needed */
+#ifdef MULTIBYTE_SUPPORT
+	/*
+	 * Ensure we don't start overwriting in the middle of a wide
+	 * character.
+	 */
+	while(rpms.sen > nbuf[rpms.tosln - 1] && rpms.sen->chr == WEOF) {
+	    extra_ellipsis++;
+	    rpms.sen--;
+	}
+#endif
+	ZR_memcpy(rpms.sen, zr_mid_ellipsis1, ZR_MID_ELLIPSIS1_SIZE);
+	rpms.sen += ZR_MID_ELLIPSIS1_SIZE;
+#ifdef MULTIBYTE_SUPPORT
+	/* Extend if we backed off for a wide character */
+	if (extra_ellipsis) {
+	    ZR_memset(rpms.sen, zr_dt, extra_ellipsis);
+	    rpms.sen += extra_ellipsis;
+	}
+#endif
+	ZR_memcpy(rpms.sen, zr_mid_ellipsis2, ZR_MID_ELLIPSIS2_SIZE);
+	nbuf[rpms.tosln][winw] = nbuf[rpms.tosln][winw + 1] = zr_zr;
     }
 
     nlnct = rpms.ln + 1;
@@ -837,12 +1336,12 @@
 		(int)ZR_strlen(nbuf[0]) + rpromptw < winw - 1;
     } else {
 /* insert >.... on first line if there is more text before start of screen */
-	ZR_memset(nbuf[0], ZWC(' '), lpromptw);
+	ZR_memset(nbuf[0], zr_sp, lpromptw);
 	t0 = winw - lpromptw;
-	t0 = t0 > 5 ? 5 : t0;
-	ZR_memcpy(nbuf[0] + lpromptw, ZWS(">...."), t0);
-	ZR_memset(nbuf[0] + lpromptw + t0, ZWC(' '), winw - t0 - lpromptw);
-	nbuf[0][winw] = nbuf[0][winw + 1] = ZWC('\0');
+	t0 = t0 > ZR_START_ELLIPSIS_SIZE ? ZR_START_ELLIPSIS_SIZE : t0;
+	ZR_memcpy(nbuf[0] + lpromptw, zr_start_ellipsis, t0);
+	ZR_memset(nbuf[0] + lpromptw + t0, zr_sp, winw - t0 - lpromptw);
+	nbuf[0][winw] = nbuf[0][winw + 1] = zr_zr;
     }
 
     for (iln = 0; iln < nlnct; iln++) {
@@ -858,7 +1357,7 @@
 	    nbuf[iln] && obuf[iln] &&
 	    ZR_strncmp(nbuf[iln], obuf[iln], 16)) {
 	    if (tccan(TCDELLINE) && obuf[iln + 1] &&
-		obuf[iln + 1][0] && nbuf[iln] &&
+		obuf[iln + 1][0].chr && nbuf[iln] &&
 		!ZR_strncmp(nbuf[iln], obuf[iln + 1], 16)) {
 		moveto(iln, 0);
 		tcout(TCDELLINE);
@@ -893,17 +1392,23 @@
 	    vcs = winw - 1;
 	/* reset character attributes to that set by the main prompt */
 	    txtchange = pmpt_attr;
-	    if (txtchangeisset(TXTNOBOLDFACE) && (rpmpt_attr & TXTBOLDFACE))
+	    if (txtchangeisset(txtchange, TXTNOBOLDFACE) &&
+		(rpmpt_attr & TXTBOLDFACE))
 		tsetcap(TCALLATTRSOFF, 0);
-	    if (txtchangeisset(TXTNOSTANDOUT) && (rpmpt_attr & TXTSTANDOUT))
+	    if (txtchangeisset(txtchange, TXTNOSTANDOUT) &&
+		(rpmpt_attr & TXTSTANDOUT))
 		tsetcap(TCSTANDOUTEND, 0);
-	    if (txtchangeisset(TXTNOUNDERLINE) && (rpmpt_attr & TXTUNDERLINE))
+	    if (txtchangeisset(txtchange, TXTNOUNDERLINE) &&
+		(rpmpt_attr & TXTUNDERLINE))
 		tsetcap(TCUNDERLINEEND, 0);
-	    if (txtchangeisset(TXTBOLDFACE) && (rpmpt_attr & TXTNOBOLDFACE))
+	    if (txtchangeisset(txtchange, TXTBOLDFACE) &&
+		(rpmpt_attr & TXTNOBOLDFACE))
 		tsetcap(TCBOLDFACEBEG, 0);
-	    if (txtchangeisset(TXTSTANDOUT) && (rpmpt_attr & TXTNOSTANDOUT))
+	    if (txtchangeisset(txtchange, TXTSTANDOUT) &&
+		(rpmpt_attr & TXTNOSTANDOUT))
 		tsetcap(TCSTANDOUTBEG, 0);
-	    if (txtchangeisset(TXTUNDERLINE) && (rpmpt_attr & TXTNOUNDERLINE))
+	    if (txtchangeisset(txtchange, TXTUNDERLINE) &&
+		(rpmpt_attr & TXTNOUNDERLINE))
 		tsetcap(TCUNDERLINEBEG, 0);
 	}
     }
@@ -920,7 +1425,7 @@
 /* reset character attributes */
     if (clearf && postedit) {
 	if ((txtchange = pmpt_attr ? pmpt_attr : rpmpt_attr))
-	    settextattributes();
+	    settextattributes(txtchange);
     }
     clearf = 0;
     oput_rpmpt = put_rpmpt;
@@ -942,7 +1447,7 @@
     fflush(shout);		/* make sure everything is written out */
 
     if (tmpalloced)
-	zfree(tmpline, tmpll);
+	zfree(tmpline, tmpll * sizeof(*tmpline));
 
     /* if we have a new list showing, note it; if part of the list has been
     overwritten, redisplay it. We have to metafy line back before calling
@@ -972,11 +1477,11 @@
 #define tc_leftcurs(X)	(void) tcmultout(TCLEFT, TCMULTLEFT, (X))
 
 static int
-wpfxlen(REFRESH_STRING s, REFRESH_STRING t)
+wpfxlen(const REFRESH_ELEMENT *s, const REFRESH_ELEMENT *t)
 {
     int i = 0;
 
-    while (*s && *s == *t)
+    while (s->chr && ZR_equal(*s, *t))
 	s++, t++, i++;
     return i;
 }
@@ -1004,7 +1509,7 @@
 	ollen = ZR_strlen(ol);
     }
     else {
-	static REFRESH_CHAR nullchr = ZWC('\0');
+	static REFRESH_ELEMENT nullchr = { ZWC('\0'), 0 };
 	ol = &nullchr;
 	ollen = 0;
     }
@@ -1027,9 +1532,9 @@
 	p1 = zhalloc((winw + 2) * sizeof(*p1));
 	if (nllen)
 	    ZR_memcpy(p1, nl, nllen);
-	ZR_memset(p1 + nllen, ZWC(' '), winw - nllen);
-	p1[winw] = ZWC('\0');
-	p1[winw + 1] = (nllen < winw) ? ZWC('\0') : nl[winw + 1];
+	ZR_memset(p1 + nllen, zr_sp, winw - nllen);
+	p1[winw] = zr_zr;
+	p1[winw + 1] = (nllen < winw) ? zr_zr : nl[winw + 1];
 	if (ln && nbuf[ln])
 	    ZR_memcpy(nl, p1, winw + 2);	/* next time obuf will be up-to-date */
 	else
@@ -1038,8 +1543,8 @@
     } else if (ollen > nllen) { /* make new line at least as long as old */
 	p1 = zhalloc((ollen + 1) * sizeof(*p1));
 	ZR_memcpy(p1, nl, nllen);
-	ZR_memset(p1 + nllen, ZWC(' '), ollen - nllen);
-	p1[ollen] = ZWC('\0');
+	ZR_memset(p1 + nllen, zr_sp, ollen - nllen);
+	p1[ollen] = zr_zr;
 	nl = p1;
 	nllen = ollen;
     }
@@ -1054,10 +1559,13 @@
     else {
 	col_cleareol = -1;
 	if (tccan(TCCLEAREOL) && (nllen == winw || put_rpmpt != oput_rpmpt)) {
-	    for (i = nllen; i && nl[i - 1] == ZWC(' '); i--);
-	    for (j = ollen; j && ol[j - 1] == ZWC(' '); j--);
+	    /* HERE: watch for change of attributes */
+	    for (i = nllen; i && ZR_equal(nl[i - 1], zr_sp); i--)
+		;
+	    for (j = ollen; j && ZR_equal(ol[j - 1], zr_sp); j--)
+		;
 	    if ((j > i + tclen[TCCLEAREOL])	/* new buf has enough spaces */
-		|| (nllen == winw && nl[winw - 1] == ZWC(' ')))
+		|| (nllen == winw && ZR_equal(nl[winw - 1], zr_sp)))
 		col_cleareol = i;
 	}
     }
@@ -1065,21 +1573,21 @@
 /* 2b: first a new trick for automargin niceness - good for cut and paste */
 
     if (hasam && vcs == winw) {
-	if (nbuf[vln] && nbuf[vln][vcs + 1] == ZWC('\n')) {
+	if (nbuf[vln] && nbuf[vln][vcs + 1].chr == ZWC('\n')) {
 	    vln++, vcs = 1;
-            if (nbuf[vln]  && *nbuf[vln]) {
-		zputc(*nbuf[vln]);
+            if (nbuf[vln]  && nbuf[vln]->chr) {
+		zputc(nbuf[vln]);
 	    } else
-		zputc(ZWC(' '));  /* I don't think this should happen */
+		zputc(&zr_sp);  /* I don't think this should happen */
 	    if (ln == vln) {	/* better safe than sorry */
 		nl++;
-		if (*ol)
+		if (ol->chr)
 		    ol++;
 		ccs = 1;
 	    }			/* else  hmmm... I wonder what happened */
 	} else {
 	    vln++, vcs = 0;
-	    zputc(ZWC('\n'));
+	    zputc(&zr_nl);
 	}
     }
     ins_last = 0;
@@ -1100,9 +1608,9 @@
      * Realign to a real character after any jiggery pokery at
      * the start of the line.
      */
-    while (*nl == WEOF) {
+    while (nl->chr == WEOF) {
 	nl++, ccs++, vcs++;
-	if (*ol)
+	if (ol->chr)
 	    ol++;
     }
 #endif
@@ -1111,18 +1619,19 @@
 
     for (;;) {
 #ifdef MULTIBYTE_SUPPORT
-	if ((!*nl || *nl != WEOF) && (!*ol || *ol != WEOF)) {
+	if ((!nl->chr || nl->chr != WEOF) && (!ol->chr || ol->chr != WEOF)) {
 #endif
-	    if (*nl && *ol && nl[1] == ol[1]) {
+	    if (nl->chr && ol->chr && ZR_equal(nl[1], ol[1])) {
 		/* skip only if second chars match */
 #ifdef MULTIBYTE_SUPPORT
 		int ccs_was = ccs;
 #endif
 		/* skip past all matching characters */
-		for (; *nl && (*nl == *ol); nl++, ol++, ccs++) ;
+		for (; nl->chr && ZR_equal(*nl, *ol); nl++, ol++, ccs++)
+		    ;
 #ifdef MULTIBYTE_SUPPORT
 		/* Make sure ol and nl are pointing to real characters */
-		while ((*nl == WEOF || *ol == WEOF) && ccs > ccs_was) {
+		while ((nl->chr == WEOF || ol->chr == WEOF) && ccs > ccs_was) {
 		    nl--;
 		    ol--;
 		    ccs--;
@@ -1130,12 +1639,12 @@
 #endif
 	    }
 
-	    if (!*nl) {
+	    if (!nl->chr) {
 		if (ccs == winw && hasam && char_ins > 0 && ins_last
 		    && vcs != winw) {
 		    nl--;           /* we can assume we can go back here */
 		    moveto(ln, winw - 1);
-		    zputc(*nl);
+		    zputc(nl);
 		    vcs++;
 		    return;         /* write last character in line */
 		}
@@ -1156,21 +1665,21 @@
 	    }
 
 	    /* we've written out the new but yet to clear rubbish due to inserts */
-	    if (!*nl) {
+	    if (!nl->chr) {
 		i = (winw - ccs < char_ins) ? (winw - ccs) : char_ins;
 		if (tccan(TCDEL) && (tcdelcost(i) <= i + 1))
 		    tc_delchars(i);
 		else {
 		    vcs += i;
 		    while (i-- > 0)
-			zputc(ZWC(' '));
+			zputc(&zr_sp);
 		}
 		return;
 	    }
 
 	    /* if we've reached the end of the old buffer, then there are few tricks
 	       we can do, so we just dump out what we must and clear if we can */
-	    if (!*ol) {
+	    if (!ol->chr) {
 		i = (col_cleareol >= 0) ? col_cleareol : nllen;
 		i -= vcs;
 		if (i < 0) {
@@ -1192,26 +1701,35 @@
 	    /* inserting & deleting chars: we can if there's no right-prompt */
 	    if ((ln || !put_rpmpt || !oput_rpmpt) 
 #ifdef MULTIBYTE_SUPPORT
-		&& *ol != WEOF && *nl != WEOF
+		&& ol->chr != WEOF && nl->chr != WEOF
 #endif
-		&& nl[1] && ol[1] && nl[1] != ol[1]) { 
+		&& nl[1].chr && ol[1].chr && !ZR_equal(nl[1], ol[1])) { 
 
 		/* deleting characters - see if we can find a match series that
 		   makes it cheaper to delete intermediate characters
 		   eg. oldline: hifoobar \ hopefully cheaper here to delete two
 		   newline: foobar	 / characters, then we have six matches */
 		if (tccan(TCDEL)) {
-		    for (i = 1; *(ol + i); i++)
+		    for (i = 1; ol[i].chr; i++)
 			if (tcdelcost(i) < wpfxlen(ol + i, nl)) {
 			    tc_delchars(i);
 			    ol += i;
 			    char_ins -= i;
 #ifdef MULTIBYTE_SUPPORT
-			    while (*ol == WEOF) {
+			    while (ol->chr == WEOF) {
 				ol++;
 				char_ins--;
 			    }
 #endif
+			    /*
+			     * If the sequence we're deleting ended
+			     * by turning off an attribute, make sure
+			     * it stays turned off.  I don't think we
+			     * should need this.
+			     */
+			    if (ol[-1].atr & TXT_ATTR_OFF_MASK)
+				settextattributes(ol[-1].atr &
+						  TXT_ATTR_OFF_MASK);
 			    i = 0;
 			    break;
 			}
@@ -1223,13 +1741,13 @@
 		   undesired scrolling occurs due to `illegal' characters on screen */
 
 		if (tccan(TCINS) && (vln != lines - 1)) {	/* not on last line */
-		    for (i = 1; *(nl + i); i++)
+		    for (i = 1; nl[i].chr; i++)
 			if (tcinscost(i) < wpfxlen(nl + i, ol)) {
 			    tc_inschars(i);
 			    zwrite(nl, i);
 			    nl += i;
 #ifdef MULTIBYTE_SUPPORT
-			    while (*nl == WEOF) {
+			    while (nl->chr == WEOF) {
 				nl++;
 				i++;
 			    }
@@ -1237,13 +1755,13 @@
 			    char_ins += i;
 			    ccs = (vcs += i);
 			    /* if we've pushed off the right, truncate oldline */
-			    for (i = 0; *(ol + i) && i < winw - ccs; i++);
+			    for (i = 0; ol[i].chr && i < winw - ccs; i++);
 #ifdef MULTIBYTE_SUPPORT
-			    while (ol[i] == WEOF)
+			    while (ol[i].chr == WEOF)
 				i++;
 #endif
 			    if (i >= winw - ccs) {
-				*(ol + i) = ZWC('\0');
+				ol[i] = zr_zr;
 				ins_last = 1;
 			    }
 			    i = 0;
@@ -1263,11 +1781,19 @@
 	 * in case we were tidying up a funny-width character when we
 	 * reached the end of the new line...
 	 */
-	if (!*nl)
+	if (!nl->chr)
 	    break;
 	do {
 #endif
-	    zputc(*nl);
+	    /*
+	     * If an attribute was on here but isn't any more,
+	     * output the sequence to turn it off.
+	     */
+	    int now_off = ol->atr & ~nl->atr & TXT_ATTR_ON_MASK;
+	    if (now_off)
+		settextattributes(now_off << TXT_ATTR_OFF_ON_SHIFT);
+
+	    zputc(nl);
 	    nl++, ol++;
 	    ccs++, vcs++;
 #ifdef MULTIBYTE_SUPPORT
@@ -1275,7 +1801,8 @@
 	     * Make sure we always overwrite the complete width of
 	     * a character that was there before.
 	     */
-	} while ((*ol == WEOF && *nl) || (*nl == WEOF && *ol));
+	} while ((ol->chr == WEOF && nl->chr) ||
+		 (nl->chr == WEOF && ol->chr));
 #endif
     }
 }
@@ -1287,22 +1814,22 @@
 void
 moveto(int ln, int cl)
 {
-    ZLE_INT_T c;
+    const REFRESH_ELEMENT *rep;
 
     if (vcs == winw) {
 	vln++, vcs = 0;
 	if (!hasam) {
-	    zputc(ZWC('\r'));
-	    zputc(ZWC('\n'));
+	    zputc(&zr_cr);
+	    zputc(&zr_nl);
 	} else {
-	    if ((vln < nlnct) && nbuf[vln] && *nbuf[vln])
-		c = *nbuf[vln];
+	    if ((vln < nlnct) && nbuf[vln] && nbuf[vln]->chr)
+		rep = nbuf[vln];
 	    else
-		c = ZWC(' ');
-	    zputc(c);
-	    zputc(ZWC('\r'));
-	    if ((vln < olnct) && obuf[vln] && *obuf[vln])
-		*obuf[vln] = c;
+		rep = &zr_sp;
+	    zputc(rep);
+	    zputc(&zr_cr);
+	    if ((vln < olnct) && obuf[vln] && obuf[vln]->chr)
+		*obuf[vln] = *rep;
 	}
     }
 
@@ -1330,9 +1857,9 @@
 		continue;
 	    }
 	}
-	zputc(ZWC('\r')), vcs = 0; /* safety precaution */
+	zputc(&zr_cr), vcs = 0; /* safety precaution */
 	while (ln > vln) {
-	    zputc(ZWC('\n'));
+	    zputc(&zr_nl);
 	    vln++;
 	}
     }
@@ -1420,7 +1947,7 @@
 		tcout(TCRIGHT);
         else {
 	    if (i != 0)
-		zputc('\r');
+		zputc(&zr_cr);
 	    tc_upcurs(lprompth - 1);
 	    zputs(lpromptbuf, shout);
 	    if (lpromptwof == winw)
@@ -1431,13 +1958,13 @@
     }
 
     if (nbuf[vln]) {
-	for (j = 0, t = nbuf[vln]; *t && (j < i); j++, t++);
+	for (j = 0, t = nbuf[vln]; t->chr && (j < i); j++, t++);
 	if (j == i)
-	    for ( ; *t && ct; ct--, t++)
-		zputc(*t);
+	    for ( ; t->chr && ct; ct--, t++)
+		zputc(t);
     }
     while (ct--)
-	zputc(ZWC(' '));	/* not my fault your terminal can't go right */
+	zputc(&zr_sp);	/* not my fault your terminal can't go right */
 }
 
 /**/
@@ -1448,8 +1975,8 @@
 
     if (ct && !tcmultout(TCDOWN, TCMULTDOWN, ct)) {
 	while (ct--)
-	    zputc(ZWC('\n'));
-	zputc(ZWC('\r')), ret = -1;
+	    zputc(&zr_nl);
+	zputc(&zr_cr), ret = -1;
     }
     return ret;
 }
@@ -1488,13 +2015,21 @@
 redisplay(UNUSED(char **args))
 {
     moveto(0, 0);
-    zputc(ZWC('\r'));		/* extra care */
+    zputc(&zr_cr);		/* extra care */
     tc_upcurs(lprompth - 1);
     resetneeded = 1;
     clearflag = 0;
     return 0;
 }
 
+/*
+ * Show as much of the line buffer as we can in single line mode.
+ * TBD: all termcap effects are turned off in this mode, so
+ * there's no point in using character attributes.  We should
+ * decide what we're going to do and either remove the handling
+ * from here or enable it in tsetcap().
+ */
+
 /**/
 static void
 singlerefresh(ZLE_STRING_T tmpline, int tmpll, int tmpcs)
@@ -1530,38 +2065,88 @@
     }
 
     /* prompt is not directly copied into the video buffer */
-    ZR_memset(vbuf, ZWC(' '), lpromptw);
+    ZR_memset(vbuf, zr_sp, lpromptw);
     vp = vbuf + lpromptw;
-    *vp = ZWC('\0');
+    *vp = zr_zr;
 
     for (t0 = 0; t0 < tmpll; t0++) {
+	int base_atr_on = 0, base_atr_off = 0, ireg;
+	struct region_highlight *rhp;
+	/*
+	 * Calculate attribute based on region.
+	 * HERE: we may need to be smarter about turning
+	 * attributes off if bailing out before the end of the
+	 * region.
+	 */
+	for (ireg = 0, rhp = region_highlights;
+	     ireg < n_region_highlights;
+	     ireg++, rhp++) {
+	    int offset;
+	    if (rhp->flags & ZRH_PREDISPLAY)
+		offset = 0;	/* include predisplay in start end */
+	    else
+		offset = predisplaylen; /* increment over it */
+	    if (rhp->start + offset <= t0 &&
+		t0 < rhp->end + offset) {
+		base_atr_on |= rhp->atr;
+		if (t0 == rhp->end + offset - 1 ||
+		    t0 == tmpll - 1)
+		    base_atr_off |= rhp->atr << TXT_ATTR_OFF_ON_SHIFT;
+	    }
+	}
+
 	if (tmpline[t0] == ZWC('\t')) {
-	    for (*vp++ = ZWC(' '); (vp - vbuf) & 7; )
-		*vp++ = ZWC(' ');
+	    REFRESH_ELEMENT sp = zr_sp;
+	    sp.atr = base_atr_on;
+	    for (*vp++ = zr_sp; (vp - vbuf) & 7; )
+		*vp++ = zr_sp;
+	    vp[-1].atr |= base_atr_off;
 	} else if (tmpline[t0] == ZWC('\n')) {
-	    *vp++ = ZWC('\\');
-	    *vp++ = ZWC('n');
+	    /* HERE highlight */
+	    vp->chr = ZWC('\\');
+	    vp->atr = special_atr_on | base_atr_on;
+	    vp++;
+	    vp->chr = ZWC('n');
+	    vp->atr = special_atr_on | special_atr_off |
+		base_atr_on | base_atr_off;
+	    vp++;
 #ifdef MULTIBYTE_SUPPORT
 	} else if (iswprint(tmpline[t0])) {
 	    int width;
-	    *vp++ = tmpline[t0];
+	    vp->chr = tmpline[t0];
+	    vp->atr = base_atr_on;
+	    vp++;
 	    width = wcwidth(tmpline[t0]);
-	    while (--width > 0)
-		*vp++ = WEOF;
+	    while (--width > 0) {
+		vp->chr = WEOF;
+		vp->atr = base_atr_on;
+		vp++;
+	    }
+	    vp[-1].atr |= base_atr_off;
 #endif
 	} else if (ZC_icntrl(tmpline[t0])) {
+	    /* HERE: highlight */
 	    ZLE_INT_T t = tmpline[++t0];
 
-	    *vp++ = ZWC('^');
-	    *vp++ = (((unsigned int)t & ~0x80u) > 31) ? ZWC('?') : (t | ZWC('@'));
-	} else
-	    *vp++ = tmpline[t0];
+	    vp->chr = ZWC('^');
+	    vp->atr = special_atr_on | base_atr_on;
+	    vp++;
+	    vp->chr = (((unsigned int)t & ~0x80u) > 31) ?
+		ZWC('?') : (t | ZWC('@'));
+	    vp->atr = special_atr_on | special_atr_off | base_atr_on |
+		base_atr_off;
+	    vp++;
+	} else {
+	    vp->chr = tmpline[t0];
+	    vp->atr = base_atr_on | base_atr_off;
+	    vp++;
+	}
 	if (t0 == tmpcs)
 	    nvcs = vp - vbuf - 1;
     }
     if (t0 == tmpcs)
 	nvcs = vp - vbuf;
-    *vp = ZWC('\0');
+    *vp = zr_zr;
 
 /* determine which part of the new line buffer we want for the display */
     if (winpos == -1)
@@ -1570,11 +2155,14 @@
 	if ((winpos = nvcs - ((winw - hasam) / 2)) < 0)
 	    winpos = 0;
     }
-    if (winpos)
-	vbuf[winpos] = ZWC('<');	/* line continues to the left */
+    if (winpos) {
+	vbuf[winpos].chr = ZWC('<');	/* line continues to the left */
+	vbuf[winpos].atr = 0;
+    }
     if ((int)ZR_strlen(vbuf + winpos) > (winw - hasam)) {
-	vbuf[winpos + winw - hasam - 1] = ZWC('>');	/* line continues to right */
-	vbuf[winpos + winw - hasam] = ZWC('\0');
+	vbuf[winpos + winw - hasam - 1].chr = ZWC('>');	/* line continues to right */
+	vbuf[winpos + winw - hasam - 1].atr = 0;
+	vbuf[winpos + winw - hasam] = zr_zr;
     }
     ZR_strcpy(nbuf[0], vbuf + winpos);
     zfree(vbuf, vsiz * sizeof(*vbuf));
@@ -1644,28 +2232,30 @@
 	 * nastiness may be around.
 	 */
 	if (vp - *nbuf >= owinprompt)
-	    for (; *vp && *vp == *refreshop; t0++, vp++, refreshop++) ;
+	    for (; vp->chr && ZR_equal(*vp, *refreshop);
+		 t0++, vp++, refreshop++)
+		;
 
-	if (!*vp && !*refreshop)
+	if (!vp->chr && !refreshop->chr)
 	    break;
 
 	singmoveto(t0);		/* move to where we do all output from */
 
-	if (!*refreshop) {
+	if (!refreshop->chr) {
 	    if ((t0 = ZR_strlen(vp)))
 		zwrite(vp, t0);
 	    vcs += t0;
 	    break;
 	}
-	if (!*vp) {
+	if (!vp->chr) {
 	    if (tccan(TCCLEAREOL))
 		tcout(TCCLEAREOL);
 	    else
-		for (; *refreshop++; vcs++)
-		    zputc(ZWC(' '));
+		for (; refreshop++->chr; vcs++)
+		    zputc(&zr_sp);
 	    break;
 	}
-	zputc(*vp);
+	zputc(vp);
 	vcs++, t0++;
 	vp++, refreshop++;
     }
@@ -1688,7 +2278,7 @@
    do this now because it's easier (to code) */
 
     if ((!tccan(TCMULTLEFT) || pos == 0) && (pos <= vcs / 2)) {
-	zputc(ZWC('\r'));
+	zputc(&zr_cr);
 	vcs = 0;
     }
 
@@ -1699,3 +2289,16 @@
 
     vcs = pos;
 }
+
+/* Provided for unloading the module in a modular fashion */
+
+/**/
+void
+zle_refresh_finish(void)
+{
+    freevideo();
+
+    if (region_highlights)
+	zfree(region_highlights,
+	      sizeof(struct region_highlight) * n_region_highlights);
+}
Index: Src/Zle/zle_utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_utils.c,v
retrieving revision 1.44
diff -u -r1.44 zle_utils.c
--- Src/Zle/zle_utils.c	6 Mar 2008 16:42:54 -0000	1.44
+++ Src/Zle/zle_utils.c	2 Apr 2008 19:32:01 -0000
@@ -384,6 +384,7 @@
 	if (mark > zlecs)
 	    mark += ct;
     }
+    region_active = 0;
 }
 
 /**/
@@ -408,6 +409,7 @@
 	}
 	zleline[zlell = to] = ZWC('\0');
     }
+    region_active = 0;
 }
 
 /**/
@@ -724,8 +726,12 @@
     else
 	c = ZC_tolower(c);
     /* echo response and return */
-    if (c != ZWC('\n'))
-	zwcputc(c);
+    if (c != ZWC('\n')) {
+	REFRESH_ELEMENT re;
+	re.chr = c;
+	re.atr = 0;
+	zwcputc(&re, NULL);
+    }
     return c == ZWC('y');
 }
 
@@ -903,6 +909,7 @@
 handlefeep(UNUSED(char **args))
 {
     zbeep();
+    region_active = 0;
     return 0;
 }
 

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-02 19:39 PATCH: command line highlighting Peter Stephenson
@ 2008-04-03 10:30 ` Oliver Kiddle
  2008-04-03 11:33   ` Peter Stephenson
  2008-04-04  9:13 ` Mikael Magnusson
  2008-04-06 20:31 ` Bart Schaefer
  2 siblings, 1 reply; 10+ messages in thread
From: Oliver Kiddle @ 2008-04-03 10:30 UTC (permalink / raw)
  To: Zsh hackers list

Peter Stephenson wrote:
> Now 4.3.6 is out of the way, here's something I've been working on.  It
> highlights the normal command line (not a completion list) in three

Very cool. This is fantastic. The fish shell will be going belly up once
this evolves a bit.

> +special array parameter tt(region_highglight); see

There's a typo there.

Can the region and unprintable characters have a different highlight
style? zle_highlight is an array, right? Or is that for start/end escape
sequences.

> - arbitrary chunks of the line:  controlled by the (zle special) array
>   region_highlight.  This is limited to static uses at the moment;
>   highlighting doesn't track movement.

This is probably the part that will be expanded on if things like syntax
highlighting will be done in shell code (completion system like).

As a test, I wrote a very rudimentary bracket matching widget:
  local i
  zle self-insert
  for ((i=$CURSOR;i;i--)) {
    [[ $BUFFER[$i] = '(' ]] && break
  }
  region_highlight=("$((i-1)) $i/ bold")  
With a zle -M command at the end of this, zsh coredumps. Let me know if
you can't reproduce it. I observed it on Solaris 10.

I'm not sure where and how to implement the unhighlight. Ideally, I
would like to have the matching paren highlighted for 1 second which
is what nedit does. And how can it be sure to remove its highlight
and not a region from some other widget. In the same way as compadd
adds completion matches instead of us having a $comp_matches, maybe a
builtin would be easier. It might allow more flexibility: timeout options,
group/tag identifiers, etc.

I'm not too familiar with the way syntax highlighting is typically done
in text editors. Maybe command-lines are small enough that reparsing
the whole line each time makes sense but that might imply full redraws
which could be bad over slow ssh connections. If the parsing is only
done partially as changes are made, then we'll need a good few hooks and
a way to make highlights sticky: the same word in the buffer is
highlighted even as text is inserted before it.

This is going to make the whole concept of PS2 prompts instead of proper
multiline editing seem even more out of place.

Oliver


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-03 10:30 ` Oliver Kiddle
@ 2008-04-03 11:33   ` Peter Stephenson
  2008-04-03 14:11     ` Oliver Kiddle
  2008-04-03 15:13     ` Peter Stephenson
  0 siblings, 2 replies; 10+ messages in thread
From: Peter Stephenson @ 2008-04-03 11:33 UTC (permalink / raw)
  To: Zsh hackers list

On Thu, 03 Apr 2008 12:30:14 +0200
Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> > +special array parameter tt(region_highglight); see
> 
> There's a typo there.

I'll fix that when I commit.

> Can the region and unprintable characters have a different highlight
> style? zle_highlight is an array, right? Or is that for start/end escape
> sequences.

zle_highlight is a global setting which can go in your .zshrc, so you can
set zle_highlight=(region:underline special:bold), or whatever.  That's why
it begins with zle_: the local special parameters in widgets are only
special there so don't have it.

> > - arbitrary chunks of the line:  controlled by the (zle special) array
> >   region_highlight.  This is limited to static uses at the moment;
> >   highlighting doesn't track movement.
> 
> This is probably the part that will be expanded on if things like syntax
> highlighting will be done in shell code (completion system like).

Yes, one day it should be possible to expand this to track changes, but I
think there are areas of the code that aren't very consistent at the
moment, such as undo.  Also, I'm not sure how consistent the completion
code is in calling the right code to change the command line: mostly I
think it's OK, but there's so much of it it's hard to be sure.  So for now
it's purely static.

> As a test, I wrote a very rudimentary bracket matching widget:
>   local i
>   zle self-insert
>   for ((i=$CURSOR;i;i--)) {
>     [[ $BUFFER[$i] = '(' ]] && break
>   }
>   region_highlight=("$((i-1)) $i/ bold")
> With a zle -M command at the end of this, zsh coredumps. Let me know if
> you can't reproduce it. I observed it on Solaris 10.

I couldn't easily get this to happen (that "/" probably isn't supposed to
be there).  However, I have seen at least one crash involving widgets and
region_highlight here which comes from get_region_highlight() being
broken.  I think I'll commit what I have:  it is to be assumed that the
interface will continue to change.

> I'm not sure where and how to implement the unhighlight. Ideally, I
> would like to have the matching paren highlighted for 1 second which
> is what nedit does.

We need to be able to see if a key is read within the time and if it is
unget it and return (like sit-for in Emacs).  Something like
  read -kt 1 && zle -U $REPLY
should work, but I haven't played around with it yet.

> And how can it be sure to remove its highlight
> and not a region from some other widget.

region_highlight could be an associative array, I suppose.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-03 11:33   ` Peter Stephenson
@ 2008-04-03 14:11     ` Oliver Kiddle
  2008-04-03 15:29       ` Peter Stephenson
  2008-04-03 15:13     ` Peter Stephenson
  1 sibling, 1 reply; 10+ messages in thread
From: Oliver Kiddle @ 2008-04-03 14:11 UTC (permalink / raw)
  To: Zsh hackers list

Peter wrote:

> zle_highlight is a global setting which can go in your .zshrc, so you can
> set zle_highlight=(region:underline special:bold), or whatever.  That's why

Thanks. Sorry, I see now that that is documented.

> > With a zle -M command at the end of this, zsh coredumps. Let me know if
> > you can't reproduce it. I observed it on Solaris 10.

> I couldn't easily get this to happen (that "/" probably isn't supposed to
> be there).  However, I have seen at least one crash involving widgets and
> region_highlight here which comes from get_region_highlight() being
> broken.  I think I'll commit what I have:  it is to be assumed that the
> interface will continue to change.

I've updated from cvs, rebuilt with debug enabled, removed the
spurious "/" and tried again. This time it works. Thanks.

By the way, the documented feature that accepting a line does not save
region_highlight doesn't appear to work. For me, the variable and the
highlights are persisting.

> We need to be able to see if a key is read within the time and if it is
> unget it and return (like sit-for in Emacs).  Something like
>   read -kt 1 && zle -U $REPLY
> should work, but I haven't played around with it yet.

I had to add a call to "zle -R" so that it doesn't wait till the end of
the widget to display the highlight but otherwise, that seems to work.
An additional key typed in less than a second will remove the
highlighting but that's actually fine. I've included the full function
below in case anyone else wants to play with it: handle quoting or
forward searching perhaps.

> > And how can it be sure to remove its highlight
> > and not a region from some other widget.
> 
> region_highlight could be an associative array, I suppose.

Yes, that would be another solution.

Oliver

local i nested=1
typeset -A match
match=(
  \) \(
  \" \"
  \' \'
  \] \[
  \} \{
)
zle self-insert
for ((i=$(( $CURSOR - 1));i;i--)) {
  [[ $BUFFER[$i] = $match[$KEYS] ]] && (( ! --nested )) && break
  [[ $BUFFER[$i] == $KEYS ]] && (( nested++ ))
}
region_highlight=("$((i-1)) $i standout")
zle -R
read -kt 1 && zle -U "$REPLY"
region_highlight=()


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-03 11:33   ` Peter Stephenson
  2008-04-03 14:11     ` Oliver Kiddle
@ 2008-04-03 15:13     ` Peter Stephenson
  1 sibling, 0 replies; 10+ messages in thread
From: Peter Stephenson @ 2008-04-03 15:13 UTC (permalink / raw)
  To: Zsh hackers list

On Thu, 3 Apr 2008 12:33:01 +0100
Peter Stephenson <pws@csr.com> wrote:
> We need to be able to see if a key is read within the time and if it is
> unget it and return (like sit-for in Emacs).  Something like
>   read -kt 1 && zle -U $REPLY
> should work, but I haven't played around with it yet.

The bad news is the timeout didn't work, at least for me.  The good news is
that fixing it up to use zle's (relatively robust) timeout mechanism when
it's reading from zle anyway isn't hard and is more interesting than trying
to understand PPDU sounding frames for 802.11n beamforming calibration.  I
have solid evidence for the last bit, anyway.

I thought about updating the doc for read -t about this, but decided that
would just be confusing: it should "just work".  The granularity and range
for the timeout are smaller than the non-zle case but for reading a key
that should be immaterial.

For reference, the following function highlight-current-word (written as an
autoload file) highlights the current word for 1 second or until a key is
pressed (which is then handled as normal).  This now seems to be working.

##start
highlight-current-word-sub()
{
  local REPLY highlight
  local -a matched_words region_highlight
  autoload -U match-words-by-style
  match-words-by-style

  integer before after

  if (( ${#matched_words[4]} )); then
    highlight=underline
    (( before = ${#matched_words[1]} + ${#matched_words[2]} ))
    (( after = before + ${#matched_words[3]} + ${#matched_words[4]} ))
  else
    highlight=standout
    if (( ${#matched_words[3]} )); then
      (( before = ${#matched_words[1]} + ${#matched_words[2]} + \
	${#matched_words[3]} ))
      (( after = before + ${#matched_words[5]} ))
    else
      (( before = ${#matched_words[1]} ))
      (( after = before + ${#matched_words[2]} + ${#matched_words[5]} ))
    fi
  fi
  region_highlight=("$before $after $highlight")
  zle -R "current word highlighted..."
  read -k -t 1 && zle -U $REPLY
}

highlight-current-word() {
  # allow function to save and restore region_highlight
  highlight-current-word-sub "$@"
  # restore old display
  zle -R
}

highlight-current-word "$@"
##end


Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.189
diff -u -r1.189 builtin.c
--- Src/builtin.c	28 Mar 2008 09:59:09 -0000	1.189
+++ Src/builtin.c	3 Apr 2008 15:10:10 -0000
@@ -4739,6 +4739,7 @@
     int readchar = -1, val, resettty = 0;
     struct ttyinfo saveti;
     char d;
+    long izle_timeout = 0;
 #ifdef MULTIBYTE_SUPPORT
     wchar_t delim = L'\n', wc;
     mbstate_t mbs;
@@ -4900,18 +4901,32 @@
 		timeout = (zlong)mn.u.l * (zlong)1000000;
 	    }
 	}
-	if (readfd == -1 ||
-	    !read_poll(readfd, &readchar, keys && !zleactive, timeout)) {
-	    if (OPT_ISSET(ops,'k') && !zleactive && !isem)
-		settyinfo(&shttyinfo);
-	    else if (resettty && SHTTY != -1)
-	      settyinfo(&saveti);
-	    if (haso) {
-		fclose(shout);
-		shout = oshout;
-		SHTTY = -1;
+	if (izle) {
+	    /*
+	     * Timeout is in 100ths of a second rather than us.
+	     * See calc_timeout() in zle_main for format of this.
+	     */
+	    timeout = -(timeout/(zlong)10000 + 1L);
+	    izle_timeout = (long)timeout;
+#ifdef LONG_MAX
+	    /* saturate if range exceeded */
+	    if ((zlong)izle_timeout != timeout)
+		izle_timeout = LONG_MAX;
+#endif
+	} else {
+	    if (readfd == -1 ||
+		!read_poll(readfd, &readchar, keys && !zleactive, timeout)) {
+		if (OPT_ISSET(ops,'k') && !zleactive && !isem)
+		    settyinfo(&shttyinfo);
+		else if (resettty && SHTTY != -1)
+		    settyinfo(&saveti);
+		if (haso) {
+		    fclose(shout);
+		    shout = oshout;
+		    SHTTY = -1;
+		}
+		return 1;
 	    }
-	    return 1;
 	}
     }
 
@@ -4927,7 +4942,7 @@
 
 	do {
 	    if (izle) {
-		if ((val = getkeyptr(0, NULL)) < 0) {
+		if ((val = getkeyptr(izle_timeout, NULL)) < 0) {
 		    eof = 1;
 		    break;
 		}
@@ -5033,7 +5048,7 @@
 #ifdef MULTIBYTE_SUPPORT
 	    int key;
 
-	    while ((key = getkeyptr(0, NULL)) >= 0) {
+	    while ((key = getkeyptr(izle_timeout, NULL)) >= 0) {
 		char c = (char)key;
 		/*
 		 * If multibyte, it can't be y, so we don't care
@@ -5044,7 +5059,7 @@
 		    break;
 	    }
 #else
-	    int key = getkeyptr(0, NULL);
+	    int key = getkeyptr(izle_timeout, NULL);
 #endif
 
 	    readbuf[0] = (key == 'y' ? 'y' : 'n');
@@ -5087,7 +5102,7 @@
 #endif
 	/* get input, a character at a time */
 	while (!gotnl) {
-	    c = zread(izle, &readchar);
+	    c = zread(izle, &readchar, izle_timeout);
 	    /* \ at the end of a line indicates a continuation *
 	     * line, except in raw mode (-r option)            */
 #ifdef MULTIBYTE_SUPPORT
@@ -5277,7 +5292,7 @@
     if (!gotnl) {
 	sigset_t s = child_unblock();
 	for (;;) {
-	    c = zread(izle, &readchar);
+	    c = zread(izle, &readchar, izle_timeout);
 #ifdef MULTIBYTE_SUPPORT
 	    if (c == EOF) {
 		/* not waiting to be completed any more */
@@ -5418,13 +5433,13 @@
 
 /**/
 static int
-zread(int izle, int *readchar)
+zread(int izle, int *readchar, long izle_timeout)
 {
     char cc, retry = 0;
     int ret;
 
     if (izle) {
-	int c = getkeyptr(0, NULL);
+	int c = getkeyptr(izle_timeout, NULL);
 
 	return (c < 0 ? EOF : c);
     }
Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.80
diff -u -r1.80 init.c
--- Src/init.c	12 Dec 2007 13:46:37 -0000	1.80
+++ Src/init.c	3 Apr 2008 15:10:11 -0000
@@ -82,7 +82,7 @@
 /* Pointer to read-key function from zle */
 
 /**/
-mod_export int (*getkeyptr) _((int, int *));
+mod_export int (*getkeyptr) _((long, int *));
 
 /* SIGCHLD mask */
 
Index: Src/Zle/zle_keymap.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_keymap.c,v
retrieving revision 1.28
diff -u -r1.28 zle_keymap.c
--- Src/Zle/zle_keymap.c	10 Sep 2006 18:10:49 -0000	1.28
+++ Src/Zle/zle_keymap.c	3 Apr 2008 15:10:11 -0000
@@ -1363,7 +1363,7 @@
 static int
 getkeybuf(int w)
 {
-    int c = getbyte(w, NULL);
+    int c = getbyte((long)w, NULL);
 
     if(c < 0)
 	return EOF;
Index: Src/Zle/zle_main.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_main.c,v
retrieving revision 1.105
diff -u -r1.105 zle_main.c
--- Src/Zle/zle_main.c	3 Apr 2008 11:39:12 -0000	1.105
+++ Src/Zle/zle_main.c	3 Apr 2008 15:10:12 -0000
@@ -451,13 +451,22 @@
 /*
  * See if we need a timeout either for a key press or for a
  * timed function.
+ *
+ * do_keytmout is passed down from getbyte() here.  If it is positive,
+ * we the keytimeout value, which is in 100ths of a second (directly set
+ * from the parameter).  If it is negative, we use -(do_keytmout+1)
+ * (i.e. the one's complement, to allow a zero value to be set).  This
+ * is only used when calling into zle from outside to specify an
+ * explicit timeout.  This is also in 100ths of a second.
  */
 
 static void
-calc_timeout(struct ztmout *tmoutp, int do_keytmout)
+calc_timeout(struct ztmout *tmoutp, long do_keytmout)
 {
-    if (do_keytmout && keytimeout > 0) {
-	if (keytimeout > ZMAXTIMEOUT * 100 /* 24 days for a keypress???? */)
+    if (do_keytmout && (keytimeout > 0 || do_keytmout < 0)) {
+	if (do_keytmout < 0)
+	    tmoutp->exp100ths = (time_t)-do_keytmout;
+	else if (keytimeout > ZMAXTIMEOUT * 100 /* 24 days for a keypress???? */)
 	    tmoutp->exp100ths = ZMAXTIMEOUT * 100;
 	else
 	    tmoutp->exp100ths = keytimeout;
@@ -501,8 +510,10 @@
     }
 }
 
+/* see calc_timeout for use of do_keytmout */
+
 static int
-raw_getbyte(int do_keytmout, char *cptr)
+raw_getbyte(long do_keytmout, char *cptr)
 {
     int ret;
     struct ztmout tmout;
@@ -785,9 +796,11 @@
     return ret;
 }
 
+/* see calc_timeout for use of do_keytmout */
+
 /**/
 mod_export int
-getbyte(int do_keytmout, int *timeout)
+getbyte(long do_keytmout, int *timeout)
 {
     char cc;
     unsigned int ret;
@@ -899,7 +912,7 @@
 mod_export ZLE_INT_T
 getfullchar(int do_keytmout)
 {
-    int inchar = getbyte(do_keytmout, NULL);
+    int inchar = getbyte((long)do_keytmout, NULL);
 
 #ifdef MULTIBYTE_SUPPORT
     return getrestchar(inchar);
@@ -962,7 +975,7 @@
 	 * arrive together.  If we don't do this the input can
 	 * get stuck if an invalid byte sequence arrives.
 	 */
-	inchar = getbyte(1, &timeout);
+	inchar = getbyte(1L, &timeout);
 	/* getbyte deliberately resets lastchar_wide_valid */
 	lastchar_wide_valid = 1;
 	if (inchar == EOF) {
Index: Src/Zle/zle_misc.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_misc.c,v
retrieving revision 1.43
diff -u -r1.43 zle_misc.c
--- Src/Zle/zle_misc.c	30 Aug 2007 14:38:27 -0000	1.43
+++ Src/Zle/zle_misc.c	3 Apr 2008 15:10:12 -0000
@@ -619,7 +619,7 @@
      *
      * Hence for now this remains byte-by-byte.
      */
-    while ((gotk = getbyte(0, NULL)) != EOF) {
+    while ((gotk = getbyte(0L, NULL)) != EOF) {
 	if (gotk == '-' && !digcnt) {
 	    minus = -1;
 	    digcnt++;
Index: Src/Zle/zle_vi.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_vi.c,v
retrieving revision 1.14
diff -u -r1.14 zle_vi.c
--- Src/Zle/zle_vi.c	25 Apr 2006 15:00:27 -0000	1.14
+++ Src/Zle/zle_vi.c	3 Apr 2008 15:10:12 -0000
@@ -108,7 +108,7 @@
     char m[3], *str;
     Thingy cmd;
 
-    if (getbyte(0, NULL) == EOF)
+    if (getbyte(0L, NULL) == EOF)
 	return ZLEEOF;
 
     m[0] = lastchar;

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-03 14:11     ` Oliver Kiddle
@ 2008-04-03 15:29       ` Peter Stephenson
  0 siblings, 0 replies; 10+ messages in thread
From: Peter Stephenson @ 2008-04-03 15:29 UTC (permalink / raw)
  To: Zsh hackers list

On Thu, 03 Apr 2008 16:11:13 +0200
Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> By the way, the documented feature that accepting a line does not save
> region_highlight doesn't appear to work. For me, the variable and the
> highlights are persisting.

I got confused.  region_highlight doesn't get unset when it goes out of
scope: this is standard behaviour for ZLE variables and we need the effect
to persist for the life of the command line here, too.  So it needs to be
reset explicitly.  Clearly it won't be valid for a new line.

(For that reason, history could be a bit of a nightmare to get right.)

> > We need to be able to see if a key is read within the time and if it is
> > unget it and return (like sit-for in Emacs).  Something like
> >   read -kt 1 && zle -U $REPLY
> > should work, but I haven't played around with it yet.
> 
> I had to add a call to "zle -R" so that it doesn't wait till the end of
> the widget to display the highlight but otherwise, that seems to work.
> An additional key typed in less than a second will remove the
> highlighting but that's actually fine.

Yes, that was deliberate.

Index: Src/Zle/zle_main.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_main.c,v
retrieving revision 1.106
diff -u -r1.106 zle_main.c
--- Src/Zle/zle_main.c	3 Apr 2008 15:20:30 -0000	1.106
+++ Src/Zle/zle_main.c	3 Apr 2008 15:23:51 -0000
@@ -1225,6 +1225,8 @@
     zleline = NULL;
     forget_edits();
     errno = old_errno;
+    /* highlight no longer valid */
+    set_region_highlight(NULL, NULL);
     return s;
 }
 
Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.54
diff -u -r1.54 zle_refresh.c
--- Src/Zle/zle_refresh.c	3 Apr 2008 11:39:12 -0000	1.54
+++ Src/Zle/zle_refresh.c	3 Apr 2008 15:23:51 -0000
@@ -434,6 +434,11 @@
 }
 
 
+/*
+ * The parameter system requires the pm argument, but this
+ * may be NULL if called directly.
+ */
+
 /**/
 void
 set_region_highlight(UNUSED(Param pm), char **aval)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-02 19:39 PATCH: command line highlighting Peter Stephenson
  2008-04-03 10:30 ` Oliver Kiddle
@ 2008-04-04  9:13 ` Mikael Magnusson
  2008-04-04 16:28   ` Peter Stephenson
  2008-04-06 20:31 ` Bart Schaefer
  2 siblings, 1 reply; 10+ messages in thread
From: Mikael Magnusson @ 2008-04-04  9:13 UTC (permalink / raw)
  To: Zsh hackers list

On 02/04/2008, Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> Now 4.3.6 is out of the way, here's something I've been working on.  It
>  highlights the normal command line (not a completion list) in three
>  ways:

+item(tt(special))(
+Individual characters that have no direct printable
+representation but are shown in a special manner by the line editor.
+These characters are described below.)
+enditem()

It appears a newline is needed between the . and the ), or the output becomes:
  These characters are described below..'
same for these parts
`tt(^)' followed by the base character.)
on the operating system.)


The region hilighting acts a bit odd for me, if I type 'hello' press
ctrl-space and go left, everything appears fine until the cursor
reaches the "e", at this point the "h" is also hilighted. Tried in
urxvt and xterm, with zsh -f, with the same result. Going back and
forth seems to leave some characters hilighted too.

Other than that, this seems really nice. :)

-- 
Mikael Magnusson


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-04  9:13 ` Mikael Magnusson
@ 2008-04-04 16:28   ` Peter Stephenson
  0 siblings, 0 replies; 10+ messages in thread
From: Peter Stephenson @ 2008-04-04 16:28 UTC (permalink / raw)
  To: Zsh hackers list

On Fri, 4 Apr 2008 11:13:51 +0200
"Mikael Magnusson" <mikachu@gmail.com> wrote:
> The region hilighting acts a bit odd for me, if I type 'hello' press
> ctrl-space and go left, everything appears fine until the cursor
> reaches the "e", at this point the "h" is also hilighted. Tried in
> urxvt and xterm, with zsh -f, with the same result.

This seems to fix it.  The code is written to try to be safe about the
state without being too agressive about writing attributes unnecessarily
and without excessive code.  I don't claim to have balanced the three
things well.

Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.55
diff -u -r1.55 zle_refresh.c
--- Src/Zle/zle_refresh.c	3 Apr 2008 15:33:39 -0000	1.55
+++ Src/Zle/zle_refresh.c	4 Apr 2008 16:22:53 -0000
@@ -506,20 +506,34 @@
 void
 zwcputc(const REFRESH_ELEMENT *c, REFRESH_CHAR *curatrp)
 {
+    /*
+     * Safety: turn attributes off if last heard of turned on.
+     * This differs from *curatrp, which is an optimisation for
+     * writing lots of stuff at once.
+     */
+    static int lastatr;
 #ifdef MULTIBYTE_SUPPORT
     mbstate_t mbstate;
     int i;
     VARARR(char, mbtmp, MB_CUR_MAX + 1);
 #endif
 
+    if (lastatr & ~c->atr) {
+	/* Stuff on we don't want, turn it off */
+	settextattributes((lastatr & ~c->atr) << TXT_ATTR_OFF_ON_SHIFT);
+	lastatr = 0;
+    }
+
     /*
      * Don't output "on" attributes in a string of characters with
      * the same attributes.
      */
     if ((c->atr & TXT_ATTR_ON_MASK) &&
 	(!curatrp ||
-	 ((*curatrp & TXT_ATTR_ON_MASK) != (c->atr & TXT_ATTR_ON_MASK))))
-	settextattributes(c->atr & TXT_ATTR_ON_MASK);
+	 ((*curatrp & TXT_ATTR_ON_MASK) != (c->atr & TXT_ATTR_ON_MASK)))) {
+	lastatr = c->atr & TXT_ATTR_ON_MASK;
+	settextattributes(lastatr);
+    }
 
 #ifdef MULTIBYTE_SUPPORT
     if (c->chr != WEOF) {
@@ -531,8 +545,10 @@
     fputc(c->chr, shout);
 #endif
 
-    if (c->atr & TXT_ATTR_OFF_MASK)
+    if (c->atr & TXT_ATTR_OFF_MASK) {
 	settextattributes(c->atr & TXT_ATTR_OFF_MASK);
+	lastatr &= ~((c->atr & TXT_ATTR_OFF_MASK) >> TXT_ATTR_OFF_ON_SHIFT);
+    }
     if (curatrp) {
 	/*
 	 * Remember the current attributes:  those that are turned



-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-02 19:39 PATCH: command line highlighting Peter Stephenson
  2008-04-03 10:30 ` Oliver Kiddle
  2008-04-04  9:13 ` Mikael Magnusson
@ 2008-04-06 20:31 ` Bart Schaefer
  2008-04-07  9:36   ` Peter Stephenson
  2 siblings, 1 reply; 10+ messages in thread
From: Bart Schaefer @ 2008-04-06 20:31 UTC (permalink / raw)
  To: Zsh hackers list

On Apr 2,  8:39pm, Peter Stephenson wrote:
}
} Now 4.3.6 is out of the way, here's something I've been working on.  It
} highlights the normal command line (not a completion list)

Now all we need to make all the would-be highlighters happy is
highlighting in PREDISPLAY and POSTDISPLAY, I think.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: PATCH: command line highlighting
  2008-04-06 20:31 ` Bart Schaefer
@ 2008-04-07  9:36   ` Peter Stephenson
  0 siblings, 0 replies; 10+ messages in thread
From: Peter Stephenson @ 2008-04-07  9:36 UTC (permalink / raw)
  To: Zsh hackers list

Bart Schaefer wrote:
> On Apr 2,  8:39pm, Peter Stephenson wrote:
> }
> } Now 4.3.6 is out of the way, here's something I've been working on.  It
> } highlights the normal command line (not a completion list)
> 
> Now all we need to make all the would-be highlighters happy is
> highlighting in PREDISPLAY and POSTDISPLAY, I think.

This should work already.  A "P" in the region_highlight indicates the
index includes PRESDISPLAY, and counting is supposed to continue over
POSTDISPLAY.  See, for example, the highlighting added to
read-from-minibuffer.  I should make it more explicit that POSTDISPLAY
is included, however.

Completely separate matter that's just occurred to me... actually,I
think I've inadvertently made read-from-minibuffer behave slightly
differently.  The documentation says it "will work correctly as a widget
in its own right".  However, I've rewritten so that saving and restoring
of special parameters is done by making them local.  This doesn't
actually work when it's called directly as a widget: the parameters are
already local at the level in question and won't be restored when they
go out of scope, because of the way special zle widgets work.  It would
probably be safer to put it back.  It's actually not very useful as a
widget in its own right so it's hardly worth worrying about, but I might
as well get out of the corner I've documented myself into.

pws

Index: Doc/Zsh/zle.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/zle.yo,v
retrieving revision 1.60
diff -u -r1.60 zle.yo
--- Doc/Zsh/zle.yo	4 Apr 2008 09:29:32 -0000	1.60
+++ Doc/Zsh/zle.yo	7 Apr 2008 09:34:53 -0000
@@ -778,8 +778,12 @@
 item(tt(region_highlight) (array))(
 Each element of this array may be set to a string that describes
 highlighting for an arbitrary region of the command line that will
-take effect the next time the command line is redisplayed.  Each
-string consists of the following parts:
+take effect the next time the command line is redisplayed.  Highlighting
+of the non-editable parts of the command line in tt(PREDISPLAY)
+and tt(POSTDISPLAY) are possible, but note that the tt(P) flag
+is needed for character indexing to include tt(PREDISPLAY).
+
+Each string consists of the following parts:
 
 startlist()
 list(Optionally, a `tt(P)' to signify that the start and end offset that
Index: Functions/Zle/read-from-minibuffer
===================================================================
RCS file: /cvsroot/zsh/zsh/Functions/Zle/read-from-minibuffer,v
retrieving revision 1.6
diff -u -r1.6 read-from-minibuffer
--- Functions/Zle/read-from-minibuffer	3 Apr 2008 11:39:11 -0000	1.6
+++ Functions/Zle/read-from-minibuffer	7 Apr 2008 09:34:54 -0000
@@ -21,21 +21,41 @@
 
   local pretext="$PREDISPLAY$LBUFFER$RBUFFER$POSTDISPLAY
 "
-local LBUFFER="$2"
-local RBUFFER="$3"
-local PREDISPLAY="$pretext${1:-? }"
-local POSTDISPLAY=
-local -a region_highlight
-region_highlight=("P${#pretext} ${#PREDISPLAY} bold")
+# We could use the local variables mechanism to save these
+# values, but if read-from-minibuffer is called as a widget
+# (which isn't actually all that useful) the values won't be
+# restored because the variables are already local at the current
+# level and don't get restored when they go out of scope.
+# We could do it with an additional function level.
+  local save_lbuffer=$LBUFFER
+  local save_rbuffer=$RBUFFER
+  local save_predisplay=$PREDISPLAY
+  local save_postdisplay=$POSTDISPLAY
+  local -a save_region_highlight
+  save_region_highlight=("${region_highlight[@]}")
 
-if [[ -n $keys ]]; then
-  zle -R
-  read -k $keys
-  stat=$?
-else
-  zle recursive-edit -K main
-  stat=$?
-  (( stat )) || REPLY=$BUFFER
-fi
+{
+  LBUFFER="$2"
+  RBUFFER="$3"
+  PREDISPLAY="$pretext${1:-? }"
+  POSTDISPLAY=
+  region_highlight=("P${#pretext} ${#PREDISPLAY} bold")
+
+  if [[ -n $keys ]]; then
+    zle -R
+    read -k $keys
+    stat=$?
+  else
+    zle recursive-edit -K main
+    stat=$?
+    (( stat )) || REPLY=$BUFFER
+  fi
+} always {
+  LBUFFER=$save_lbuffer
+  RBUFFER=$save_rbuffer
+  PREDISPLAY=$save_predisplay
+  POSTDISPLAY=$save_postdisplay
+  region_highlight=("${save_region_highlight[@]}")
+}
 
 return $stat

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2008-04-07  9:38 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-04-02 19:39 PATCH: command line highlighting Peter Stephenson
2008-04-03 10:30 ` Oliver Kiddle
2008-04-03 11:33   ` Peter Stephenson
2008-04-03 14:11     ` Oliver Kiddle
2008-04-03 15:29       ` Peter Stephenson
2008-04-03 15:13     ` Peter Stephenson
2008-04-04  9:13 ` Mikael Magnusson
2008-04-04 16:28   ` Peter Stephenson
2008-04-06 20:31 ` Bart Schaefer
2008-04-07  9:36   ` Peter Stephenson

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).