[BUG?] If true-color is used, overlapping colors do not work

[BUG?] If true-color is used, overlapping colors do not work

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 2180 bytes --]

Hello
The last test from the attached ztst reveals this. Before-last test
overlapping colors with standard fg=green/fg=red symbols. Result is a
word "true" with "r" in red the rest in green. You can see the colors
by doing cat /tmp/output.

With true color, I get:
--- /tmp/zsh.ztst.77121/ztst.out    2018-11-07 14:11:51.000000000 +0100
+++ /tmp/zsh.ztst.77121/ztst.tout    2018-11-07 14:11:51.000000000 +0100
@@ -2,5 +2,5 @@
 rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }
 zle -N rh_widget
 bindkey "\C-a" rh_widget
-0m32mt31mr39m32mue39m
+0mt38;2;204;0;0mr39mue

The minus-line is not yet updated, inherited from basic 8-colors case
(which is working and overlapped colors can be observed in the
minus-line). But look at the plus-line. It doesn't emit any color for
"t", only true-color red for "r", and then nothing for "ue".

With zsh/nearcolor loaded (test code pasted at the end of this email),
colors are correctly overlapped:

--- /tmp/zsh.ztst.76836/ztst.out    2018-11-07 14:09:32.000000000 +0100
+++ /tmp/zsh.ztst.76836/ztst.tout    2018-11-07 14:09:32.000000000 +0100
@@ -3,5 +3,5 @@
 rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }
 zle -N rh_widget
 bindkey "\C-a" rh_widget
-0m32mt31mr39m32mue39m
+0m38;5;40mt38;5;160mr39m38;5;40mue39m

cat /tmp/output confirms this.

zsh/nearcolor test case:

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4
fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r'
in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_input $'\C-a'
  zpty_line 5   # 3 lines echoed back + 3 empty lines (zpty bug)
  zpty_line 1 p
  zpty_stop
0:overlapping region_highlight with true-color
>zmodload zsh/nearcolor
>rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }
>rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }
>zle -N rh_widget
>bindkey "\C-a" rh_widget
>0m32mt31mr39m32mue39m
>exit

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

[-- Attachment #2: X04highlight.ztst.txt --]
[-- Type: text/plain, Size: 5018 bytes --]

# Tests for region_highlight, true-color support, near-color support

%prep

  if [[ $OSTYPE == cygwin ]]; then
    ZTST_unimplemented='the zsh/zpty module does not work on Cygwin'
  elif zmodload zsh/zpty 2> /dev/null; then
    zpty_start() {
      export PS1= PS2=
      zpty -d
      zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV"
    }
    zpty_input() {
      zpty -w zsh "$1" ${${(M)2:#nl}:+$'\n'}
    }
    zpty_line() {
      setopt localoptions extendedglob noshwordsplit
      local REPLY cm=$'\r'
      integer i
      for (( i = 0; i < ${1:-1}; ++i )); do
        zpty -r zsh REPLY
        # P is for "preserve", induces keeping some
        # color codes to test region_highlight, etc.
        # The color codes are then made regular text
        [[ "$2" = "p" ]] && {
            print -rl -- "$REPLY" >> /tmp/output # THE LINE
            REPLY=${REPLY//$'\x1b'\[([2][0-9;]m|[JK]|\?[0-9]##(h|l))/}
            REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}}
        } || {
            REPLY=${REPLY//$'\x1b'\[([0-9;]##m|[JK]|\?[0-9]##(h|l))/}
        }
        # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2
        # otherwise replace with \1 stripped out of leading/trailing [[:space:]]
        REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}
        [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##}
      done
    }
    zpty_stop() {
      setopt localoptions extendedglob
      local REPLY cm=$'\r'
      zpty -w zsh $'exit\n'
      # zpty gives no output when piped without these braces (?)
      # The while loop with // substitution is to convert `e^Mexit'
      # into `exit' (see zpty_line). The sed commands remove escapes
      { zpty -r zsh } | sed $'/[^[:space:]]/!d; s/\r$//; s/\x1b\\[[0-9;]*m//g; s/\x1b\\[[JK]//g; s/\x1b\\[?[0-9]*[hl]//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -r -- "$REPLY"; done
      zpty -d
      :
    }
  else
    ZTST_unimplemented='the zsh/zpty module is not available'
  fi

%test

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_input $'\C-a'
  zpty_line 3
  zpty_line 1 p
  zpty_stop
0:basic region_highlight with 8 colors
>rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }
>zle -N rh_widget
>bindkey "\C-a" rh_widget
>0m32mtrue39m
>exit

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' nl
  zpty_input 'zle -N rh_widget' nl
  zpty_input 'bindkey "\C-a" rh_widget' nl
  zpty_input $'\C-a'
  zpty_line 6   # 3 lines echoed back + 3 empty lines (TODO: wrong -n use)
  zpty_line 1 p
  zpty_stop
0:basic region_highlight with true-color (hex-triplets)
>rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }
>zle -N rh_widget
>bindkey "\C-a" rh_widget
>0m38;2;4;8;16mtrue39m
>exit

  zpty_start
  zpty_input 'zmodload zsh/nearcolor' nl
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' nl
  zpty_input 'zle -N rh_widget' nl
  zpty_input 'bindkey "\C-a" rh_widget' nl
  zpty_input $'\C-a'
  zpty_line 8   # 5 lines echoed back + 5 empty lines (TODO: wrong -n use)
  zpty_line 1 p
  zpty_input 'zmodload -u zsh/nearcolor' nl
  zpty_line 2
  zpty_stop
0:basic region_highlight with near-color (hex-triplets at input)
>zmodload zsh/nearcolor
>rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }
>zle -N rh_widget
>bindkey "\C-a" rh_widget
>0m38;5;232mtrue39m
>zmodload -u zsh/nearcolor
>exit

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_input $'\C-a'
  zpty_line 4
  zpty_line 1 p
  zpty_stop
0:overlapping region_highlight with 8 colors
>rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }
>rh2() { region_highlight+=( "1 2 fg=red" ); }
>zle -N rh_widget
>bindkey "\C-a" rh_widget
>0m32mt31mr39m32mue39m
>exit

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_input $'\C-a'
  zpty_line 4
  zpty_line 1 p
  zpty_stop
0:overlapping region_highlight with true-color
>rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }
>rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }
>zle -N rh_widget
>bindkey "\C-a" rh_widget
>0m32mt31mr39m32mue39m
>exit

%clean

  zmodload -ui zsh/zpty

# vim:ft=zsh

Re: [BUG?] If true-color is used, overlapping colors do not work

[Sebastian Gniazdowski]

A simpler way to observe this:

% zsh -fi
% rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; };
rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }
zle -N rh_widget
bindkey "\C-a" rh_widget
% <press ctrl-a>
On Wed, 7 Nov 2018 at 14:19, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> Hello
> The last test from the attached ztst reveals this. Before-last test overlapping colors with standard fg=green/fg=red symbols. Result is a word "true" with "r" in red the rest in green. You can see the colors by doing cat /tmp/output.
>
> With true color, I get:
> --- /tmp/zsh.ztst.77121/ztst.out    2018-11-07 14:11:51.000000000 +0100
> +++ /tmp/zsh.ztst.77121/ztst.tout    2018-11-07 14:11:51.000000000 +0100
> @@ -2,5 +2,5 @@
>  rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }
>  zle -N rh_widget
>  bindkey "\C-a" rh_widget
> -0m32mt31mr39m32mue39m
> +0mt38;2;204;0;0mr39mue
>
> The minus-line is not yet updated, inherited from basic 8-colors case (which is working and overlapped colors can be observed in the minus-line). But look at the plus-line. It doesn't emit any color for "t", only true-color red for "r", and then nothing for "ue".
>
> With zsh/nearcolor loaded (test code pasted at the end of this email), colors are correctly overlapped:
>
> --- /tmp/zsh.ztst.76836/ztst.out    2018-11-07 14:09:32.000000000 +0100
> +++ /tmp/zsh.ztst.76836/ztst.tout    2018-11-07 14:09:32.000000000 +0100
> @@ -3,5 +3,5 @@
>  rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }
>  zle -N rh_widget
>  bindkey "\C-a" rh_widget
> -0m32mt31mr39m32mue39m
> +0m38;5;40mt38;5;160mr39m38;5;40mue39m
>
> cat /tmp/output confirms this.
>
> zsh/nearcolor test case:
>
>   zpty_start
>   zpty_input 'zmodload zsh/nearcolor'
>   zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
>   zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
>   zpty_input 'zle -N rh_widget'
>   zpty_input 'bindkey "\C-a" rh_widget'
>   zpty_input $'\C-a'
>   zpty_line 5   # 3 lines echoed back + 3 empty lines (zpty bug)
>   zpty_line 1 p
>   zpty_stop
> 0:overlapping region_highlight with true-color
> >zmodload zsh/nearcolor
> >rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }
> >rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }
> >zle -N rh_widget
> >bindkey "\C-a" rh_widget
> >0m32mt31mr39m32mue39m
> >exit
>
> --
> Sebastian Gniazdowski
> News: https://twitter.com/ZdharmaI
> IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
> Blog: http://zdharma.org



-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: [BUG?] If true-color is used, overlapping colors do not work

[Sebastian Gniazdowski]

I was testing on Linux, via docker. $termcap[Co] shows 8, the same
$terminfo[colors]. The zsh/nearcolor fails and maybe it could fail a
little better?
    ....
    -0m38;5;232mtrue39m
    +0m30mtrue39m
    ...
    Was testing: basic region_highlight with near-color (hex-triplets at input)

Color 30 is apparently black. Maybe zsh/nearcolor could use (compare
against) first 8 or 16 colors when $termcap[Co] <= 8?

Re: [BUG?] If true-color is used, overlapping colors do not work

[Oliver Kiddle]

Sebastian Gniazdowski wrote:
> The last test from the attached ztst reveals this. Before-last test
> overlapping colors with standard fg=green/fg=red symbols. Result is a

>  rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }

It isn't overlapping colours but the += assignment that fails.

I had missed that we have code for converting attributes back to text
form and appending to region_highlight apparently relies on that.

This does also mean that if you use a short form like fg=#c00, it gets
expanded to fg=#cc0000. It also means that with nearcolor,
region_highlight reflects the converted value, fg=160 in this case.
I don't think this is a problem. There was previously forms that would
not be preserved exactly.

Thanks for testing the changes.
By the way, your tests might be easier to manage if you set
fg_start_code, fg_end_code and so on in zle_highlight to some plain text
markers. Could be clearer to follow than $'\x1b'. None of your tests
work on my system. I think the expected results reflect the particulars
of the termcap database for your terminal on your system. zle -T may
help but to be portable, the tests may also need to search for a
256color terminal and skip if none is found.

Oliver

diff --git a/Src/prompt.c b/Src/prompt.c
index 284c02475..377015ad8 100644
--- a/Src/prompt.c
+++ b/Src/prompt.c
@@ -1646,7 +1646,8 @@ match_colour(const char **teststrp, int is_fg, int colour)
 		color.red = col >> 16;
 		color.green = (col & 0xff00) >> 8;
 		color.blue = col & 0xff;
-	    }
+	    } else
+		return TXT_ERROR;
 	    *teststrp = end;
 	    colour = runhookdef(GETCOLORATTR, &color) - 1;
 	    if (colour < 0) { /* no hook function added, try true color (24-bit) */
@@ -1744,11 +1745,12 @@ match_highlight(const char *teststr, zattr *on_var)
 
 /*
  * Count or output a string for colour information: used
- * by output_highlight().
+ * by output_highlight(). count when buf is NULL.
+ * returned count excludes the terminating null byte.
  */
 
 static int
-output_colour(int colour, int fg_bg, int use_tc, char *buf)
+output_colour(int colour, int fg_bg, int use_tc, int truecol, char *buf)
 {
     int atrlen = 3, len;
     char *ptr = buf;
@@ -1756,8 +1758,12 @@ output_colour(int colour, int fg_bg, int use_tc, char *buf)
 	strcpy(ptr, fg_bg == COL_SEQ_FG ? "fg=" : "bg=");
 	ptr += 3;
     }
+    if (truecol) {
+	/* length of hex triplet always 7, don't need sprintf to count */
+	atrlen += buf ? sprintf(ptr, "#%02x%02x%02x", colour >> 16,
+		(colour >> 8) & 0xff, colour & 0xff) : 7;
     /* colour should only be > 7 if using termcap but let's be safe */
-    if (use_tc || colour > 7) {
+    } else if (use_tc || colour > 7) {
 	char digbuf[DIGBUFSIZE];
 	sprintf(digbuf, "%d", colour);
 	len = strlen(digbuf);
@@ -1795,6 +1801,7 @@ output_highlight(zattr atr, char *buf)
 	len = output_colour(txtchangeget(atr, TXT_ATTR_FG_COL),
 			    COL_SEQ_FG,
 			    (atr & TXT_ATTR_FG_TERMCAP),
+			    (atr & TXT_ATTR_FG_24BIT),
 			    ptr);
 	atrlen += len;
 	if (buf)
@@ -1811,6 +1818,7 @@ output_highlight(zattr atr, char *buf)
 	len = output_colour(txtchangeget(atr, TXT_ATTR_BG_COL),
 			    COL_SEQ_BG,
 			    (atr & TXT_ATTR_BG_TERMCAP),
+			    (atr & TXT_ATTR_BG_24BIT),
 			    ptr);
 	atrlen += len;
 	if (buf)

Re: [BUG?] If true-color is used, overlapping colors do not work

[Oliver Kiddle]

Sebastian Gniazdowski wrote:
> I was testing on Linux, via docker. $termcap[Co] shows 8, the same
> $terminfo[colors]. The zsh/nearcolor fails and maybe it could fail a
> little better?

I admit that black, as opposed to the default colour isn't clever. The
patch changes this.

> Color 30 is apparently black. Maybe zsh/nearcolor could use (compare
> against) first 8 or 16 colors when $termcap[Co] <= 8?

The first 16 colours vary between terminals and are also commonly
configurable so nearcolor never compares against them. 

There is an escape sequence for getting colour definitions but I don't
think trying to use it gains us much.

Oliver


diff --git a/Src/Modules/nearcolor.c b/Src/Modules/nearcolor.c
index 128658e20..7ebc75365 100644
--- a/Src/Modules/nearcolor.c
+++ b/Src/Modules/nearcolor.c
@@ -117,13 +117,14 @@ mapRGBto256(int red, int green, int blue)
 static int
 getnearestcolor(UNUSED(Hookdef dummy), Color_rgb col)
 {
+    /* we add 1 to the colours so that colour 0 (black) is
+     * distinguished from runhookdef() indicating that no
+     * hook function is registered */
     if (tccolours == 256)
 	return mapRGBto256(col->red, col->green, col->blue) + 1;
     if (tccolours == 88)
 	return mapRGBto88(col->red, col->green, col->blue) + 1;
-    /* returning 1 indicates black rather than failure (0) so this
-     * module still serves to prevent fallback on true color */
-    return 1;
+    return -1;
 }
 
 static struct features module_features = {
diff --git a/Src/prompt.c b/Src/prompt.c
index 377015ad8..568bfc2a9 100644
--- a/Src/prompt.c
+++ b/Src/prompt.c
@@ -1650,10 +1650,12 @@ match_colour(const char **teststrp, int is_fg, int colour)
 		return TXT_ERROR;
 	    *teststrp = end;
 	    colour = runhookdef(GETCOLORATTR, &color) - 1;
-	    if (colour < 0) { /* no hook function added, try true color (24-bit) */
+	    if (colour == -1) { /* no hook function added, try true color (24-bit) */
 		colour = (((color.red << 8) + color.green) << 8) + color.blue;
 		return on | (is_fg ? TXT_ATTR_FG_24BIT : TXT_ATTR_BG_24BIT) |
 			(zattr)colour << shft;
+	    } else if (colour <= -2) {
+		return TXT_ERROR;
 	    }
 	} else if ((named = ialpha(**teststrp))) {
 	    colour = match_named_colour(teststrp);

Re: [BUG?] If true-color is used, overlapping colors do not work

[Sebastian Gniazdowski]

On Thu, 8 Nov 2018 at 04:03, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> (...)
> markers. Could be clearer to follow than $'\x1b'. None of your tests
> work on my system. I think the expected results reflect the particulars
> of the termcap database for your terminal on your system. zle -T may
> help but to be portable, the tests may also need to search for a
> 256color terminal and skip if none is found.

I hope the tests will be working system-wide, otherwise it's a show
stopper for them. Could you maybe provide the test wrong answer? If
you are sure that it's about TERM= value, then ok, I've observed this
too, and export TERM=xterm-256color fixed the tests (on Linux).
Searching for proper termcap seems a good idea. I might also find a
docker image with the same OS and test it.

Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: [BUG?] If true-color is used, overlapping colors do not work

[Sebastian Gniazdowski]

I've applied the patch and recompiled, and the tests or the Ctrl-a
(with Src/zsh -fi) snippet show now change?

Re: [BUG?] If true-color is used, overlapping colors do not work

[Oliver Kiddle]

Sebastian Gniazdowski wrote:
> I've applied the patch and recompiled, and the tests or the Ctrl-a
> (with Src/zsh -fi) snippet show now change?

Working through each of your tests manually, they all seem to work fine
for me if judged purely on the basis of what I see in a terminal. I've
checked more than one terminal programme including ones with and without
true colour support.

> I hope the tests will be working system-wide, otherwise it's a show
> stopper for them. Could you maybe provide the test wrong answer? If
> you are sure that it's about TERM= value, then ok, I've observed this
> too, and export TERM=xterm-256color fixed the tests (on Linux).
> Searching for proper termcap seems a good idea. I might also find a
> docker image with the same OS and test it.

I get differences before the tests even reach the final buffer state. The
fact that all the setup lines get repeated back in the test output
really isn't helping the matter. The differences are all escape
sequences which diff is not showing, e.g 1b 5b 6d occurring at the start
of each line. There are also further differences that can be put down to a
difference of termcap definition. The only hope would be to strip the output
down to just the section actually being tested (which comptest manages),
also using something like:
  zle_highlight=( fg_start_code:FGSTART fg_end_code:FGEND fg_default_code:FGDEF )
and perhaps zle -T somehow to map escape sequences to something readable
somehow contriving to set TERM to something that claims 256 colour support.
And, only relying on the portable subset of sed syntax.
It's not without reason that I was too lazy to attempt this in the first
place.

Oliver

Re: [BUG?] If true-color is used, overlapping colors do not work

[Sebastian Gniazdowski]

On Fri, 9 Nov 2018 at 02:28, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>
> Sebastian Gniazdowski wrote:
> > I've applied the patch and recompiled, and the tests or the Ctrl-a
> > (with Src/zsh -fi) snippet show now change?
>
> Working through each of your tests manually, they all seem to work fine
> for me if judged purely on the basis of what I see in a terminal. I've
> checked more than one terminal programme including ones with and without
> true colour support.

Ah I've applied only the black color patch, with also the first patch,
everything works fine.

> I get differences before the tests even reach the final buffer state. The
> fact that all the setup lines get repeated back in the test output
> really isn't helping the matter. The differences are all escape
> sequences which diff is not showing, e.g 1b 5b 6d occurring at the start
> of each line. There are also further differences that can be put down to a
> difference of termcap definition. The only hope would be to strip the output
> down to just the section actually being tested (which comptest manages),
> also using something like:
>   zle_highlight=( fg_start_code:FGSTART fg_end_code:FGEND fg_default_code:FGDEF )
> and perhaps zle -T somehow to map escape sequences to something readable
> somehow contriving to set TERM to something that claims 256 colour support.
> And, only relying on the portable subset of sed syntax.
> It's not without reason that I was too lazy to attempt this in the first
> place.

I'm stripping down the escape codes that Zsh/Zle normally produces
managing output. It appears that your setup generates some additional
codes. This should be only a matter of altering // substitution and
sed invocation, but maybe your right, maybe updating zle_highlight /
fg_start_code / fg_end_code is the right way.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: [BUG?] If true-color is used, overlapping colors do not work

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 2347 bytes --]

On Fri, 9 Nov 2018 at 02:28, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>(...)
>The only hope would be to strip the output
> down to just the section actually being tested (which comptest manages),
> also using something like:
>   zle_highlight=( fg_start_code:FGSTART fg_end_code:FGEND fg_default_code:FGDEF )
> and perhaps zle -T somehow to map escape sequences to something readable
> somehow contriving to set TERM to something that claims 256 colour support.
> And, only relying on the portable subset of sed syntax.
> It's not without reason that I was too lazy to attempt this in the first
> place.

I have new fresh test file which I hope brings the X04 test to some
new stage or level. The lines sent to zpty now aren't repeated,
because zsh is started with +Z option, and `setopt zle' is emitted
just before the "zpty -w zsh $'\C-a'" line, and the shell / zpty is
closed via Ctrl-D, so luckily only single one, the test line
(BUFFER="true", region_highlight+=...) is emitted and read.

I've used zle -T tc tcfunc, which sets REPLY="", i.e. discards the
codes. Should I change them to something? Because I'm only getting LE
termcap command, moving cursor to the left (after printing POSTDISPLAY
with zsh-autosuggestions text), some cd, ce, up commands and no codes
for colors. Do you think I should not strip the termcap codes and
specify them in test's expected output?

I'm currently removing following escapes from all zpty output:
- ^[[27m, ^[[24m, etc.
- ^[[J
- ^[[K
- ^[[?2004h
- e^Mexit -> exit
- ^[[?2004l

You seem to retrieve some other Zle-managing-output code (I think it's
this) and the tests fail? Maybe it is already fixed by the late-Zle
enabling, so you could try running the new test file?

There are 6 tests currently:

0:basic region_highlight with 8 colors
0:basic region_highlight with true-color (hex-triplets)
0:basic region_highlight with near-color (hex-triplets at input)
0:overlapping region_highlight with 8 colors
0:overlapping region_highlight with true-color
0:overlapping region_highlight with near-color (hex-triplets at input)

PS. Didn't yet search for terminfo file, just did export
TERM=xterm-256, not actually sure how to search for the definition
-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

[-- Attachment #2: X04highlight.ztst.txt --]
[-- Type: text/plain, Size: 5469 bytes --]

# Tests for region_highlight, true-color support, near-color support

%prep

  if [[ $OSTYPE == cygwin ]]; then
    ZTST_unimplemented='the zsh/zpty module does not work on Cygwin'
  elif zmodload zsh/zpty 2> /dev/null; then
    export TERM=xterm-256color
    zpty_start() {
      export PS1= PS2=
      zpty -d
      zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z"
    }
    zpty_input() {
      zpty ${${(M)2:#nonl}:+-n} -w zsh "$1"
    }
    zpty_enable_zle() {
      zpty -w zsh "tcfunc() { REPLY=""; }"
      zpty -w zsh "setopt zle; zle -T tc tcfunc"
    }
    zpty_line() {
      setopt localoptions extendedglob noshwordsplit
      local REPLY cm=$'\r'
      integer i
      for (( i = 0; i < ${1:-1}; ++i )); do
        zpty -r zsh REPLY
        # P is for "preserve", induces keeping some
        # color codes to test region_highlight, etc.
        # The color codes are then made regular text
        [[ "$2" = "p" ]] && {
            print -rl -- "$REPLY" >> /tmp/output # THE LINE
            REPLY=${REPLY//$'\x1b'\[([2][0-9;]m|[JK]|\?[0-9]##(h|l))/}
            REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}}
        } || {
            REPLY=${REPLY//$'\x1b'\[([0-9;]##m|[JK]|\?[0-9]##(h|l))/}
        }
        # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2
        # otherwise replace with \1 stripped out of leading/trailing [[:space:]]
        REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}
        [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##}
      done
    }
    zpty_stop() {
      setopt localoptions extendedglob
      local REPLY cm=$'\r'
      # Zle is active, can use Ctrl-D to exit
      zpty -n -w zsh $'\C-d'
      # zpty gives no output when piped without these braces (?)
      # The while loop with // substitution is to convert `e^Mexit'
      # into `exit' (see zpty_line). The sed commands remove escapes
      { zpty -r zsh } | sed $'/[^[:space:]]/!d; s/\r$//; s/\x1b\\[[0-9;]*m//g; s/\x1b\\[[JK]//g; s/\x1b\\[?[0-9]*[hl]//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done
      zpty -d
      :
    }
  else
    ZTST_unimplemented='the zsh/zpty module is not available'
  fi

%test

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with 8 colors
>0m32mtrue39m

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with true-color (hex-triplets)
>0m38;2;4;8;16mtrue39m

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with near-color (hex-triplets at input)
>0m38;5;232mtrue39m

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with 8 colors
>0m32mt31mr39m32mue39m

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with true-color
>0m38;2;0;204;0mt38;2;204;0;0mr39m38;2;0;204;0mue39m

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with near-color (hex-triplets at input)
>0m38;5;40mt38;5;160mr39m38;5;40mue39m

%clean

  zmodload -ui zsh/zpty

# vim:ft=zsh

Re: [BUG?] If true-color is used, overlapping colors do not work

[Sebastian Gniazdowski]

On Sun, 11 Nov 2018 at 01:43, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> I've used zle -T tc tcfunc, which sets REPLY="", i.e. discards the
> codes. Should I change them to something? Because I'm only getting LE
> termcap command, moving cursor to the left (after printing POSTDISPLAY
> with zsh-autosuggestions text), some cd, ce, up commands and no codes
> for colors. Do you think I should not strip the termcap codes and
> specify them in test's expected output?

That was for non-zsh-f full featured Zsh session. I've added storing
of the termcap commands within the test. It's a sequence of only 2
codes, cd & ce:

<tc=cd,arg=>
<tc=ce,arg=>
<tc=ce,arg=>
<tc=cd,arg=>
...

I've checked their codes, they are the ^[[J, ^[[K that I'm removing by
// and in other (separate in execution) place by analogous sed
invocation. No other termcap command gets invoked during make
TESTNUM=X04. I wonder if maybe on your setup there is some other
termcap command invoked?


> I'm currently removing following escapes from all zpty output:
> - ^[[27m, ^[[24m, etc.
> - ^[[J
> - ^[[K
> - ^[[?2004h
> - e^Mexit -> exit
> - ^[[?2004l
>
> You seem to retrieve some other Zle-managing-output code (I think it's
> this) and the tests fail? Maybe it is already fixed by the late-Zle
> enabling, so you could try running the new test file?
>
> There are 6 tests currently:
>
> 0:basic region_highlight with 8 colors
> 0:basic region_highlight with true-color (hex-triplets)
> 0:basic region_highlight with near-color (hex-triplets at input)
> 0:overlapping region_highlight with 8 colors
> 0:overlapping region_highlight with true-color
> 0:overlapping region_highlight with near-color (hex-triplets at input)
>
> PS. Didn't yet search for terminfo file, just did export
> TERM=xterm-256, not actually sure how to search for the definition
> --
> Sebastian Gniazdowski
> News: https://twitter.com/ZdharmaI
> IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
> Blog: http://zdharma.org



--
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

X04 zle highlight tests, near-color bug

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 1982 bytes --]

On Fri, 9 Nov 2018 at 02:28, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> difference of termcap definition. The only hope would be to strip the output
> down to just the section actually being tested (which comptest manages),
> also using something like:
>   zle_highlight=( fg_start_code:FGSTART fg_end_code:FGEND fg_default_code:FGDEF )
> and perhaps zle -T somehow to map escape sequences to something readable
> somehow contriving to set TERM to something that claims 256 colour support.
> And, only relying on the portable subset of sed syntax.
> It's not without reason that I was too lazy to attempt this in the first
> place.

I'm sending the same uplifted, bugs and not-nice things fixed (zpty
-n, Zle-echoing-back, etc. ) test file X04zlehighlight as in my answer
where I've elaborated on zle -T and enumerated codes being removed
like "^[[?2004h" (X-seq: 43811), but with:

      zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|"
fg_end_code:"|" bg_start_code:"BCDE|" bg_end_code:"|" )'

I adapted the visible-codes by readability on the terminal, example
new-code and it's previous form:

-0m38;2;4;8;16mtrue39m
+0mCDE|8;2;4;8;16|trueCDE|9|

(I've accepted this and other changes into the test, they're correct
with the zle_highlight contents).

And a bug appeared – every test works, instead of the two near-color
tests. The reported differences:

-0mCDE|8;5;232|trueCDE|9|
+0m38;5;232mtrueCDE|9|
(...)
Was testing: basic region_highlight with near-color (hex-triplets at input)

-0mCDE|8;5;40|tCDE|8;5;160|rCDE|9|CDE|8;5;40|ueCDE|9|
+0m38;5;40mt38;5;160mrCDE|9|38;5;40mueCDE|9|
Was testing: overlapping region_highlight with near-color
(hex-triplets at input)

The minus-lines are hand-crafted by me, holding the expected contents
after zle_highlight was customized.
-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

[-- Attachment #2: X04zlehighlight.ztst.txt --]
[-- Type: text/plain, Size: 5719 bytes --]

# Tests for region_highlight, true-color support, near-color support
# Version 0.5
%prep

  if [[ $OSTYPE == cygwin ]]; then
    ZTST_unimplemented='the zsh/zpty module does not work on Cygwin'
  elif zmodload zsh/zpty 2> /dev/null; then
    export TERM=xterm-256color
    zpty_start() {
      export PS1= PS2=
      zpty -d
      zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z"
      zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|" fg_end_code:"|" bg_start_code:"BCDE|" bg_end_code:"|" )'
    }
    zpty_input() {
      zpty ${${(M)2:#nonl}:+-n} -w zsh "$1"
    }
    zpty_enable_zle() {
      zpty -w zsh "tcfunc() { REPLY=""; }"
      zpty -w zsh "setopt zle; zle -T tc tcfunc"  # This line will not be echoed back, behaving like ! -o zle
    }
    zpty_line() {
      setopt localoptions extendedglob noshwordsplit
      local REPLY cm=$'\r'
      integer i
      for (( i = 0; i < ${1:-1}; ++i )); do
        zpty -r zsh REPLY
        # P is for "preserve", induces keeping some
        # color codes to test region_highlight, etc.
        # The color codes are then made regular text
        [[ "$2" = "p" ]] && {
            print -rl -- "$REPLY" >> /tmp/output # THE LINE
            REPLY=${REPLY//$'\x1b'\[([2][0-9;]m|[JK]|\?[0-9]##(h|l))/}
            REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}}
        } || {
            REPLY=${REPLY//$'\x1b'\[([0-9;]##m|[JK]|\?[0-9]##(h|l))/}
        }
        # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2
        # otherwise replace with \1 stripped out of leading/trailing [[:space:]]
        REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}
        [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##}
      done
    }
    zpty_stop() {
      setopt localoptions extendedglob
      local REPLY cm=$'\r'
      # Zle is active, can use Ctrl-D to exit
      zpty -n -w zsh $'\C-d'
      # zpty gives no output when piped without these braces (?)
      # The while loop with // substitution is to convert `e^Mexit'
      # into `exit' (see zpty_line). The sed commands remove escapes
      { zpty -r zsh } | sed $'/[^[:space:]]/!d; s/\r$//; s/\x1b\\[[0-9;]*m//g; s/\x1b\\[[JK]//g; s/\x1b\\[?[0-9]*[hl]//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done
      zpty -d
      :
    }
  else
    ZTST_unimplemented='the zsh/zpty module is not available'
  fi

%test

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with 8 colors
>0mCDE|2|trueCDE|9|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with true-color (hex-triplets)
>0mCDE|8;2;4;8;16|trueCDE|9|

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with near-color (hex-triplets at input)
>0mCDE|8;5;232|trueCDE|9|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with 8 colors
>0mCDE|2|tCDE|1|rCDE|9|CDE|2|ueCDE|9|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with true-color
>0mCDE|8;2;0;204;0|tCDE|8;2;204;0;0|rCDE|9|CDE|8;2;0;204;0|ueCDE|9|

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with near-color (hex-triplets at input)
>0mCDE|8;5;40|tCDE|8;5;160|rCDE|9|CDE|8;5;40|ueCDE|9|

%clean

  zmodload -ui zsh/zpty

# vim:ft=zsh

Re: X04 zle highlight tests, near-color bug

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 187 bytes --]

Had someone maybe ran the X04 test? The one attached in the topic email
should fail at 3 or 4th test like it is described. The no-zsh_higlight
version in workers/43811 should fully pass.

highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work)

[Oliver Kiddle]

On 11 Nov, Sebastian Gniazdowski wrote:
> I have new fresh test file which I hope brings the X04 test to some
> new stage or level. The lines sent to zpty now aren't repeated,

Sorry, I hadn't found time to look at this. Given that ensuring
portability has proved tricky with the existing zpty based tests I'm
rather leery of including new ones and would be happier if someone else
were to take a look.

The tests now pass on my system.

> I've used zle -T tc tcfunc, which sets REPLY="", i.e. discards the
> codes. Should I change them to something? Because I'm only getting LE

If removing them works for the particular tests then I'd have thought
this was fine.

> I'm currently removing following escapes from all zpty output:
> - ^[[?2004h
> - ^[[?2004l

unsetting zle_bracketed_paste will prevent zsh from generating these
which is better than removing them afterwards.

> PS. Didn't yet search for terminfo file, just did export
> TERM=xterm-256, not actually sure how to search for the definition

Searching probably isn't easy to do portably. Perhaps guessing like this
is better, but it would be good to check $termcap[Co] afterwards and
skip tests as appropriate.

>       { zpty -r zsh } | sed $'/[^[:space:]]/!d; s/\r$//; s/\x1b\\[[0-9;]*m//g; s/\x1b\\[[JK]//g; s/\x1b\\[?[0-9]*[hl]//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done

This use of sed isn't portable. In particular, the use of semi-colons to
separate commands and the [[:space:]] character class are extensions.
You can use literal spaces and multiple -e options. Either that
or do the substitutions in zsh.

The heirloom project (http://heirloom.sourceforge.net/tools.html) is a
good source for lowest common denominator versions of Unix tools if you
don't have an old OS available.

Oliver

Re: highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work)

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 2817 bytes --]

On Sat, 24 Nov 2018 at 18:32, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>
> On 11 Nov, Sebastian Gniazdowski wrote:
>
> > I've used zle -T tc tcfunc, which sets REPLY="", i.e. discards the
> > codes. Should I change them to something? Because I'm only getting LE
>
> If removing them works for the particular tests then I'd have thought
> this was fine.

Ok, so I've also removed the deleting of \[[J and \[[K codes (result:
simpler patterns and regexes), which are covered by the "zle -T ..."
sink.

> > I'm currently removing following escapes from all zpty output:
> > - ^[[?2004h
> > - ^[[?2004l
>
> unsetting zle_bracketed_paste will prevent zsh from generating these
> which is better than removing them afterwards.

I'm using this now, the codes are indeed not generated now. One thing,
I've had to call `unset zle_brack...' after `setopt zle', not before
it.

So the sed command is now quite simple, only three "-e" arguments:

{ zpty -r zsh } | sed -e $'/[^\t\r ]/!d' -e $'s/\r$//' -e
$'s/\x1b\\[[0-9;]*m//g' | while ...

> > PS. Didn't yet search for terminfo file, just did export
> > TERM=xterm-256, not actually sure how to search for the definition
>
> Searching probably isn't easy to do portably. Perhaps guessing like this
> is better, but it would be good to check $termcap[Co] afterwards and
> skip tests as appropriate.

This is now done, i.e. termcap[Co] is being check:

  export TERM=xterm-256color
  if [[ ${+termcap} != 1 || ${termcap[Co]} != <-> || ${termcap[Co]}
-lt 256 ]]; then
    ZTST_unimplemented="no termcap module OR termcap doesn't support
256 or more colors"
  elif [[ $OSTYPE == cygwin ]]; then
    ZTST_unimplemented='the zsh/zpty module does not work on Cygwin'
  elif zmodload zsh/zpty 2> /dev/null; then
    zpty_start() {
    ...

> This use of sed isn't portable. In particular, the use of semi-colons to
> separate commands and the [[:space:]] character class are extensions.
> You can use literal spaces and multiple -e options. Either that
> or do the substitutions in zsh.

The current sed invocation is pasted above – sed is now using literal
spaces instead of [[:space:]], multiple -e commands instead of
semicolon and is simplified.


Now, the bug with zle_highlight + zsh/nearcolor:

-0mCDE|38;5;232|trueCDE|39|
+0m38;5;232mtrueCDE|39|
Test ./X04zlehighlight.ztst failed: output differs from expected as
shown above for:
...
Was testing: basic region_highlight with near-color (hex-triplets at input)

The output doesn't follow zsh_highlight replacements for start-code
and end-code and still emits raw codes.

Could this be fixed?

--
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

[-- Attachment #2: X04zlehighlight.ztst.txt --]
[-- Type: text/plain, Size: 5915 bytes --]

# Tests for region_highlight, true-color support, near-color support
# Version 0.5
%prep

  export TERM=xterm-256color
  if [[ ${+termcap} != 1 || ${termcap[Co]} != <-> || ${termcap[Co]} -lt 256 ]]; then
    ZTST_unimplemented="no termcap module OR termcap doesn't support 256 or more colors"
  elif [[ $OSTYPE == cygwin ]]; then
    ZTST_unimplemented='the zsh/zpty module does not work on Cygwin'
  elif zmodload zsh/zpty 2> /dev/null; then
    zpty_start() {
      export PS1= PS2=
      zpty -d
      zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z"
      zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|" bg_start_code:"BCDE|4" bg_end_code:"|" )'
    }
    zpty_input() {
      zpty ${${(M)2:#nonl}:+-n} -w zsh "$1"
    }
    zpty_enable_zle() {
      zpty -w zsh "tcfunc() { REPLY=""; }"
      # This line will not be echoed back, behaving like ! -o zle
      zpty -w zsh "setopt zle; zle -T tc tcfunc; unset zle_bracketed_paste"
    }
    zpty_line() {
      setopt localoptions extendedglob noshwordsplit
      local REPLY cm=$'\r'
      integer i
      for (( i = 0; i < ${1:-1}; ++i )); do
        zpty -r zsh REPLY
        # P is for "preserve", and induces keeping some
        # expected color codes to test region_highlight,
        # etc. - the color codes are made a regular text.
        [[ "$2" = "p" ]] && {
            REPLY=${REPLY//$'\x1b'\[([2][0-9;]m|\?[0-9]##(h|l))/}  # remove only 2...m codes
            REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}}
        } || {
            REPLY=${REPLY//$'\x1b'\[([0-9;]##m|\?[0-9]##(h|l))/}   # remove all [0-9]...m codes
        }
        # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2
        # otherwise replace with \1 stripped out of leading/trailing [[:space:]]
        REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}
        [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##}
      done
    }
    zpty_stop() {
      setopt localoptions extendedglob
      local REPLY cm=$'\r'
      # Zle is active, can use Ctrl-D to exit
      zpty -n -w zsh $'\C-d'
      # zpty gives no output when piped without these braces (?)
      # The while loop with // substitution is to convert `e^Mexit'
      # into `exit' (see zpty_line). The sed commands remove escapes
      { zpty -r zsh } | sed -e $'/[^\t\r ]/!d' -e $'s/\r$//' -e $'s/\x1b\\[[0-9;]*m//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done
      zpty -d
      :
    }
  else
    ZTST_unimplemented='the zsh/zpty module is not available'
  fi

%test

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with 8 colors
>0mCDE|32|trueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with true-color (hex-triplets)
>0mCDE|38;2;4;8;16|trueCDE|39|

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with near-color (hex-triplets at input)
>0mCDE|38;5;232|trueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with 8 colors
>0mCDE|32|tCDE|31|rCDE|39|CDE|32|ueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with true-color
>0mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39|

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with near-color (hex-triplets at input)
>0mCDE|38;5;40|tCDE|38;5;160|rCDE|39|CDE|38;5;40|ueCDE|39|

%clean

  zmodload -ui zsh/zpty

# vim:ft=zsh

[BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases)

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

On Fri, 30 Nov 2018 at 01:34, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> The output doesn't follow zsh_highlight replacements for start-code
> and end-code and still emits raw codes.
>
> Could this be fixed?

Turns out this doesn't involve near-color at all, only 256 color
codes. The new standout-targeted test returns:

-0m27m24mtr7mu27me word2 word3
+0m27m24m38;5;196mtr7m38;5;196mu27meCDE|39| word2 word3
Was testing: region highlight - standout overlapping on other
region_highlight entry

So again no CDE|3...|, but raw ^[38;5;196m. So this is a general Zsh
bug, not near-color bug, as zsh/near-color module isn't loaded in this
test, it's only the 256 code "fg=196" that is being used.

The setting: zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|"
bg_start_code:"BCDE|4" bg_end_code:"|" )

The file *-256.ztst.txt contains this test case. The file
*-standout-only.ztst.txt is the expected test case which shows no
problems with standout-overlaping other styles.

PS. I now also do not remove 2xm codes (underline, standaout) but the
test cases are all adapted to this change.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

[-- Attachment #2: X04zlehighlight-256.ztst.txt --]
[-- Type: text/plain, Size: 6383 bytes --]

# Tests for region_highlight, true-color support, near-color support
# Version 0.7 2018-12-06
%prep

  export TERM=xterm-256color
  if [[ ${+termcap} != 1 || ${termcap[Co]} != <-> || ${termcap[Co]} -lt 256 ]]; then
    ZTST_unimplemented="no termcap module OR termcap doesn't support 256 or more colors"
  elif [[ $OSTYPE == cygwin ]]; then
    ZTST_unimplemented='the zsh/zpty module does not work on Cygwin'
  elif zmodload zsh/zpty 2> /dev/null; then
    zpty_start() {
      export PS1= PS2=
      zpty -d
      zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z"
      zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|" bg_start_code:"BCDE|4" bg_end_code:"|" )'
    }
    zpty_input() {
      zpty ${${(M)2:#nonl}:+-n} -w zsh "$1"
    }
    zpty_enable_zle() {
      zpty -w zsh "tcfunc() { REPLY=""; }"
      # This line will not be echoed back, behaving like ! -o zle
      zpty -w zsh "setopt zle; zle -T tc tcfunc; unset zle_bracketed_paste"
    }
    zpty_line() {
      setopt localoptions extendedglob noshwordsplit
      local REPLY cm=$'\r'
      integer i
      for (( i = 0; i < ${1:-1}; ++i )); do
        zpty -r zsh REPLY
        # P is for "preserve", and induces keeping some
        # expected color codes to test region_highlight,
        # etc. - the color codes are made a regular text.
        [[ "$2" = "p" ]] && {
            REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}}
        } || {
            REPLY=${REPLY//$'\x1b'\[[0-9;]##m/}   # remove all [0-9]...m codes
        }
        # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2
        # otherwise replace with \1 stripped out of leading/trailing [[:space:]]
        REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}
        [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##}
      done
    }
    zpty_stop() {
      setopt localoptions extendedglob
      local REPLY cm=$'\r'
      # Zle is active, can use Ctrl-D to exit
      zpty -n -w zsh $'\C-d'
      # zpty gives no output when piped without these braces (?)
      # The while loop with // substitution is to convert `e^Mexit'
      # into `exit' (see zpty_line). The sed commands remove escapes
      { zpty -r zsh } | sed -e $'/[^\t\r ]/!d' -e $'s/\r$//' -e $'s/\x1b\\[[0-9;]*m//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done
      zpty -d
      :
    }
  else
    ZTST_unimplemented='the zsh/zpty module is not available'
  fi

%test

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true word2 word3"; region_highlight+=( "0 4 fg=196" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "2 3 standout" ); };'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:region highlight - standout overlapping on other region_highlight entry
>0m27m24mtr7mu27me word2 word3

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with 8 colors
>0m27m24mCDE|32|trueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with true-color (hex-triplets)
>0m27m24mCDE|38;2;4;8;16|trueCDE|39|

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with near-color (hex-triplets at input)
>0m27m24mCDE|38;5;232|trueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with 8 colors
>0m27m24mCDE|32|tCDE|31|rCDE|39|CDE|32|ueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with true-color
>0m27m24mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39|

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with near-color (hex-triplets at input)
>0m27m24mCDE|38;5;40|tCDE|38;5;160|rCDE|39|CDE|38;5;40|ueCDE|39|

%clean

  zmodload -ui zsh/zpty

# vim:ft=zsh

[-- Attachment #3: X04zlehighlight-standout-only.ztst.txt --]
[-- Type: text/plain, Size: 6403 bytes --]

# Tests for region_highlight, true-color support, near-color support
# Version 0.7 2018-12-06
%prep

  export TERM=xterm-256color
  if [[ ${+termcap} != 1 || ${termcap[Co]} != <-> || ${termcap[Co]} -lt 256 ]]; then
    ZTST_unimplemented="no termcap module OR termcap doesn't support 256 or more colors"
  elif [[ $OSTYPE == cygwin ]]; then
    ZTST_unimplemented='the zsh/zpty module does not work on Cygwin'
  elif zmodload zsh/zpty 2> /dev/null; then
    zpty_start() {
      export PS1= PS2=
      zpty -d
      zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z"
      zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|" bg_start_code:"BCDE|4" bg_end_code:"|" )'
    }
    zpty_input() {
      zpty ${${(M)2:#nonl}:+-n} -w zsh "$1"
    }
    zpty_enable_zle() {
      zpty -w zsh "tcfunc() { REPLY=""; }"
      # This line will not be echoed back, behaving like ! -o zle
      zpty -w zsh "setopt zle; zle -T tc tcfunc; unset zle_bracketed_paste"
    }
    zpty_line() {
      setopt localoptions extendedglob noshwordsplit
      local REPLY cm=$'\r'
      integer i
      for (( i = 0; i < ${1:-1}; ++i )); do
        zpty -r zsh REPLY
        # P is for "preserve", and induces keeping some
        # expected color codes to test region_highlight,
        # etc. - the color codes are made a regular text.
        [[ "$2" = "p" ]] && {
            REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}}
        } || {
            REPLY=${REPLY//$'\x1b'\[[0-9;]##m/}   # remove all [0-9]...m codes
        }
        # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2
        # otherwise replace with \1 stripped out of leading/trailing [[:space:]]
        REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}
        [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##}
      done
    }
    zpty_stop() {
      setopt localoptions extendedglob
      local REPLY cm=$'\r'
      # Zle is active, can use Ctrl-D to exit
      zpty -n -w zsh $'\C-d'
      # zpty gives no output when piped without these braces (?)
      # The while loop with // substitution is to convert `e^Mexit'
      # into `exit' (see zpty_line). The sed commands remove escapes
      { zpty -r zsh } | sed -e $'/[^\t\r ]/!d' -e $'s/\r$//' -e $'s/\x1b\\[[0-9;]*m//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done
      zpty -d
      :
    }
  else
    ZTST_unimplemented='the zsh/zpty module is not available'
  fi

%test

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true word2 word3"; region_highlight+=( "0 4 fg=196" ); rh2; }'
  zpty_input 'rh2() { region_highlight=( "2 3 standout" ); };' # note the =, not +=
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:region highlight - standout overlapping on other region_highlight entry
>0m27m24mtr7mu27me word2 word3

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with 8 colors
>0m27m24mCDE|32|trueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with true-color (hex-triplets)
>0m27m24mCDE|38;2;4;8;16|trueCDE|39|

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }'
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:basic region_highlight with near-color (hex-triplets at input)
>0m27m24mCDE|38;5;232|trueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with 8 colors
>0m27m24mCDE|32|tCDE|31|rCDE|39|CDE|32|ueCDE|39|

  zpty_start
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with true-color
>0m27m24mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39|

  zpty_start
  zpty_input 'zmodload zsh/nearcolor'
  zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }'
  zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long
  zpty_input 'zle -N rh_widget'
  zpty_input 'bindkey "\C-a" rh_widget'
  zpty_enable_zle
  zpty_input $'\C-a'  # emits newline, which executes BUFFER="true" command
  zpty_line 1 p       # the line of interest, preserving escapes ("p")
  zpty_stop
0:overlapping region_highlight with near-color (hex-triplets at input)
>0m27m24mCDE|38;5;40|tCDE|38;5;160|rCDE|39|CDE|38;5;40|ueCDE|39|

%clean

  zmodload -ui zsh/zpty

# vim:ft=zsh

Re: [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases)

[Sebastian Gniazdowski]

On Fri, 7 Dec 2018 at 02:55, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> So again no CDE|3...|, but raw ^[38;5;196m. So this is a general Zsh
> bug, not near-color bug, as zsh/near-color module isn't loaded in this
> test, it's only the 256 code "fg=196" that is being used.

Following Daniel's request of a minimal test case that makes the
problem visible:

widget() { zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|"
bg_start_code:"BCDE|4" bg_end_code:"|" ); BUFFER="true word2 word3";
region_highlight=( "0 4 fg=196" ); }
zle -N widget
bindkey '^T' widget

And after pasting and accepting in zsh -f, press Ctrl-T. You will see
red word "true" followed by "CDE|39|" and the other words, i.e.

% trueCDE|39| word2 word3

(the true is in red, but cannot visualize this in email). So:
zsh_highlight was ignored for 256 color code, only code ^[[39m has
been processed by zle_highlight, while the fg=196 code was output
unprocessed, i.e. as ^[[38;5;196m.

Could the problem be fixed, .. ASAP? It collides with my work of
providing X04 tests and is a blocker for making a new Zsh
release. I think Oliver touched the responsible code lately, as for
true-color (not 256 color, with or withoutzsh/nearcolor), the
zle_highlight processing works, i.e. I'm getting, e.g.:

0m27m24mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39|

in an X04 test case. So all 24-bit escape codes are nicely covered
with CDE| etc. things from zle_highlight. So maybe it seems that the
"get text representation from already processed entries in
region_highlight" that Olivier did is used also here.

So basically 256-color doesn't obey zle_highlight, while first 8/16
colors and TrueColor are obeying it.

--
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases)

[Daniel Shahaf]

Sebastian Gniazdowski wrote on Fri, 07 Dec 2018 21:26 +0100:
> Following Daniel's request of a minimal test case that makes the
> problem visible:
> 

Thanks for the test case.

> Could the problem be fixed, .. ASAP? It collides with my work of
> providing X04 tests and is a blocker for making a new Zsh
> release

Why should it be a blocker?  (I don't disagree; I just don't see the reason.)

I ran the minimal reproducer in 5.6.2 and in current master and I get
the same behaviour in both of them ("trueCDE|39| word2 word3", the whole
thing in red), so it's not a regression.

> I think Oliver touched the responsible code lately, as for
> true-color (not 256 color, with or withoutzsh/nearcolor), the
> zle_highlight processing works, i.e. I'm getting, e.g.:
> 
> 0m27m24mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39|

Cheers,

Daniel

Re: highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work)

[Oliver Kiddle]

On 30 Nov, Sebastian Gniazdowski wrote:
> The output doesn't follow zsh_highlight replacements for start-code
> and end-code and still emits raw codes.
>
> Could this be fixed?

That was intentional. As I stated in 43759 which was the message
including the true color patch:

| The actual escape sequences appear to be quite standard and are, for
| now, hard coded. We should probably support some zle_highlight fields
| similar to fg_start_code to allow them to be configured but I'm unsure
| of what exact form that should take.

It may not be good enough to overload "fg_start_code" because the escape
sequences might vary for 16, 256 or true colours. Quite why termcap
isn't used for the first 16 colours, I couldn't say but I'd have thought
it is always better to be using termcap if possible - contrary to your
change in 43875.

Apart from that concern about overloading the existing fields, I don't
object to using zle_highlight as an override or fallback for termcap
if that's somehow useful. In the case of true colour, start and end
code may not be enough to encapsulate all the relevant details. The
sequence I used has decimal numbers separated by semi-colons but judging
from https://iterm2.com/documentation-escape-codes.html there's a
macOS terminal emulator that may want hex triplets. This is what I was
referring to when saying I was unsure what form the zle_highlight fields
should take. The current hard coded escape sequences work for everything
I've tried.

Oliver

Re: highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work)

[Sebastian Gniazdowski]

On Mon, 10 Dec 2018 at 03:54, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>
> On 30 Nov, Sebastian Gniazdowski wrote:
> > The output doesn't follow zsh_highlight replacements for start-code
> > and end-code and still emits raw codes.
> >
> > Could this be fixed?
>
> That was intentional. As I stated in 43759 which was the message
> including the true color patch:
>
> | The actual escape sequences appear to be quite standard and are, for
> | now, hard coded. We should probably support some zle_highlight fields
> | similar to fg_start_code to allow them to be configured but I'm unsure
> | of what exact form that should take.

That said, not everything was hard-coded – zle_highlight was respected
by the true color patch.

> it is always better to be using termcap if possible - contrary to your
> change in 43875.

The patch could be changed to use zle_highlight only if its values
aren't the currently-well-known defaults. This way, if user will not
customize the codes that are being sent, he will choose to use
termcap, and be protected in the unknown-to-exists situation of
running Zsh on a terminal with custom 256 color codes (i.e. not
\e[38;5;N).

But when user will customize zle_highlight, then termcap will not be
used but the customization instead will be. User customizes --> he
want's thing to be used (of course, he's demand could be selective,
only for 256 colors or only for True Color, hehe).

> The
> sequence I used has decimal numbers separated by semi-colons but judging
> from https://iterm2.com/documentation-escape-codes.html there's a
> macOS terminal emulator that may want hex triplets. This is what I was
> referring to when saying I was unsure what form the zle_highlight fields
> should take. The current hard coded escape sequences work for everything
> I've tried.

The iTerm2 code is for changing the terminal's palette. With the
"untouched zle_highlight -> termcap" condition, this would be covered
anyway. If user will customize zle_highlight for e.g. tests, he will
be able to do <<TERMINAL==XYZ -> ZTST_unimplemented='terminal with
non-standard-codes'>> and simply don't bother about a specific
machine. And, in this situation, he actually may be interested to
still run the tests, even on such custom terminal, as the codes output
by Zsh will be standard anyway (and TERM will be xterm-256color
anyway).

> Oliver

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work)

[Sebastian Gniazdowski]

On Tue, 11 Dec 2018 at 00:51, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Mon, 10 Dec 2018 at 03:54, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> >
> > On 30 Nov, Sebastian Gniazdowski wrote:
> > > The output doesn't follow zsh_highlight replacements for start-code
> > > and end-code and still emits raw codes.
> > >
> > > Could this be fixed?
> >
> > That was intentional. As I stated in 43759 which was the message
> > including the true color patch:
> >
> > | The actual escape sequences appear to be quite standard and are, for
> > | now, hard coded. We should probably support some zle_highlight fields
> > | similar to fg_start_code to allow them to be configured but I'm unsure
> > | of what exact form that should take.
>
> That said, not everything was hard-coded – zle_highlight was respected
> by the true color patch.

Oh, I know see you were referring to additional zle_highlight fields.
I think this means that the uncommon situation can be responded to by
an update to Zshell (i.e. introduction of additional zle_highlight
field(s)).

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases)

[Sebastian Gniazdowski]

BTW. The mail appeared 2 days after sending.

On Sun, 9 Dec 2018 at 20:23, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> Sebastian Gniazdowski wrote on Fri, 07 Dec 2018 21:26 +0100:
> > Could the problem be fixed, .. ASAP? It collides with my work of
> > providing X04 tests and is a blocker for making a new Zsh
> > release
>
> Why should it be a blocker?  (I don't disagree; I just don't see the reason.)

I was thinking that ignoring zle_highlight will be considered a bug,
and this would be some blocker, to do a new release openly with known
unfixed bug. But it turned out that zle_highlight isn't that much
trusted and the code deliberately opts for termcap instead of using
it. I think the compromise "use termcap for default zle_highlight, but
for customized zle_highlight use it, not the termcap" addresses this
with a balanced, old-man-wise compromise.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org