* [BUG?] If true-color is used, overlapping colors do not work @ 2018-11-07 13:19 Sebastian Gniazdowski 2018-11-07 14:40 ` Sebastian Gniazdowski 2018-11-08 3:03 ` Oliver Kiddle 0 siblings, 2 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-07 13:19 UTC (permalink / raw) To: Zsh hackers list [-- Attachment #1: Type: text/plain, Size: 2180 bytes --] Hello The last test from the attached ztst reveals this. Before-last test overlapping colors with standard fg=green/fg=red symbols. Result is a word "true" with "r" in red the rest in green. You can see the colors by doing cat /tmp/output. With true color, I get: --- /tmp/zsh.ztst.77121/ztst.out 2018-11-07 14:11:51.000000000 +0100 +++ /tmp/zsh.ztst.77121/ztst.tout 2018-11-07 14:11:51.000000000 +0100 @@ -2,5 +2,5 @@ rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } zle -N rh_widget bindkey "\C-a" rh_widget -0m32mt31mr39m32mue39m +0mt38;2;204;0;0mr39mue The minus-line is not yet updated, inherited from basic 8-colors case (which is working and overlapped colors can be observed in the minus-line). But look at the plus-line. It doesn't emit any color for "t", only true-color red for "r", and then nothing for "ue". With zsh/nearcolor loaded (test code pasted at the end of this email), colors are correctly overlapped: --- /tmp/zsh.ztst.76836/ztst.out 2018-11-07 14:09:32.000000000 +0100 +++ /tmp/zsh.ztst.76836/ztst.tout 2018-11-07 14:09:32.000000000 +0100 @@ -3,5 +3,5 @@ rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } zle -N rh_widget bindkey "\C-a" rh_widget -0m32mt31mr39m32mue39m +0m38;5;40mt38;5;160mr39m38;5;40mue39m cat /tmp/output confirms this. zsh/nearcolor test case: zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_input $'\C-a' zpty_line 5 # 3 lines echoed back + 3 empty lines (zpty bug) zpty_line 1 p zpty_stop 0:overlapping region_highlight with true-color >zmodload zsh/nearcolor >rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; } >rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } >zle -N rh_widget >bindkey "\C-a" rh_widget >0m32mt31mr39m32mue39m >exit -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org [-- Attachment #2: X04highlight.ztst.txt --] [-- Type: text/plain, Size: 5018 bytes --] # Tests for region_highlight, true-color support, near-color support %prep if [[ $OSTYPE == cygwin ]]; then ZTST_unimplemented='the zsh/zpty module does not work on Cygwin' elif zmodload zsh/zpty 2> /dev/null; then zpty_start() { export PS1= PS2= zpty -d zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV" } zpty_input() { zpty -w zsh "$1" ${${(M)2:#nl}:+$'\n'} } zpty_line() { setopt localoptions extendedglob noshwordsplit local REPLY cm=$'\r' integer i for (( i = 0; i < ${1:-1}; ++i )); do zpty -r zsh REPLY # P is for "preserve", induces keeping some # color codes to test region_highlight, etc. # The color codes are then made regular text [[ "$2" = "p" ]] && { print -rl -- "$REPLY" >> /tmp/output # THE LINE REPLY=${REPLY//$'\x1b'\[([2][0-9;]m|[JK]|\?[0-9]##(h|l))/} REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}} } || { REPLY=${REPLY//$'\x1b'\[([0-9;]##m|[JK]|\?[0-9]##(h|l))/} } # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2 # otherwise replace with \1 stripped out of leading/trailing [[:space:]] REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}} [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##} done } zpty_stop() { setopt localoptions extendedglob local REPLY cm=$'\r' zpty -w zsh $'exit\n' # zpty gives no output when piped without these braces (?) # The while loop with // substitution is to convert `e^Mexit' # into `exit' (see zpty_line). The sed commands remove escapes { zpty -r zsh } | sed $'/[^[:space:]]/!d; s/\r$//; s/\x1b\\[[0-9;]*m//g; s/\x1b\\[[JK]//g; s/\x1b\\[?[0-9]*[hl]//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -r -- "$REPLY"; done zpty -d : } else ZTST_unimplemented='the zsh/zpty module is not available' fi %test zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_input $'\C-a' zpty_line 3 zpty_line 1 p zpty_stop 0:basic region_highlight with 8 colors >rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); } >zle -N rh_widget >bindkey "\C-a" rh_widget >0m32mtrue39m >exit zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' nl zpty_input 'zle -N rh_widget' nl zpty_input 'bindkey "\C-a" rh_widget' nl zpty_input $'\C-a' zpty_line 6 # 3 lines echoed back + 3 empty lines (TODO: wrong -n use) zpty_line 1 p zpty_stop 0:basic region_highlight with true-color (hex-triplets) >rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); } >zle -N rh_widget >bindkey "\C-a" rh_widget >0m38;2;4;8;16mtrue39m >exit zpty_start zpty_input 'zmodload zsh/nearcolor' nl zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' nl zpty_input 'zle -N rh_widget' nl zpty_input 'bindkey "\C-a" rh_widget' nl zpty_input $'\C-a' zpty_line 8 # 5 lines echoed back + 5 empty lines (TODO: wrong -n use) zpty_line 1 p zpty_input 'zmodload -u zsh/nearcolor' nl zpty_line 2 zpty_stop 0:basic region_highlight with near-color (hex-triplets at input) >zmodload zsh/nearcolor >rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); } >zle -N rh_widget >bindkey "\C-a" rh_widget >0m38;5;232mtrue39m >zmodload -u zsh/nearcolor >exit zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_input $'\C-a' zpty_line 4 zpty_line 1 p zpty_stop 0:overlapping region_highlight with 8 colors >rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; } >rh2() { region_highlight+=( "1 2 fg=red" ); } >zle -N rh_widget >bindkey "\C-a" rh_widget >0m32mt31mr39m32mue39m >exit zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_input $'\C-a' zpty_line 4 zpty_line 1 p zpty_stop 0:overlapping region_highlight with true-color >rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; } >rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } >zle -N rh_widget >bindkey "\C-a" rh_widget >0m32mt31mr39m32mue39m >exit %clean zmodload -ui zsh/zpty # vim:ft=zsh ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-07 13:19 [BUG?] If true-color is used, overlapping colors do not work Sebastian Gniazdowski @ 2018-11-07 14:40 ` Sebastian Gniazdowski 2018-11-07 19:19 ` Sebastian Gniazdowski 2018-11-08 3:03 ` Oliver Kiddle 1 sibling, 1 reply; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-07 14:40 UTC (permalink / raw) To: Zsh hackers list A simpler way to observe this: % zsh -fi % rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }; rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } zle -N rh_widget bindkey "\C-a" rh_widget % <press ctrl-a> On Wed, 7 Nov 2018 at 14:19, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > Hello > The last test from the attached ztst reveals this. Before-last test overlapping colors with standard fg=green/fg=red symbols. Result is a word "true" with "r" in red the rest in green. You can see the colors by doing cat /tmp/output. > > With true color, I get: > --- /tmp/zsh.ztst.77121/ztst.out 2018-11-07 14:11:51.000000000 +0100 > +++ /tmp/zsh.ztst.77121/ztst.tout 2018-11-07 14:11:51.000000000 +0100 > @@ -2,5 +2,5 @@ > rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } > zle -N rh_widget > bindkey "\C-a" rh_widget > -0m32mt31mr39m32mue39m > +0mt38;2;204;0;0mr39mue > > The minus-line is not yet updated, inherited from basic 8-colors case (which is working and overlapped colors can be observed in the minus-line). But look at the plus-line. It doesn't emit any color for "t", only true-color red for "r", and then nothing for "ue". > > With zsh/nearcolor loaded (test code pasted at the end of this email), colors are correctly overlapped: > > --- /tmp/zsh.ztst.76836/ztst.out 2018-11-07 14:09:32.000000000 +0100 > +++ /tmp/zsh.ztst.76836/ztst.tout 2018-11-07 14:09:32.000000000 +0100 > @@ -3,5 +3,5 @@ > rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } > zle -N rh_widget > bindkey "\C-a" rh_widget > -0m32mt31mr39m32mue39m > +0m38;5;40mt38;5;160mr39m38;5;40mue39m > > cat /tmp/output confirms this. > > zsh/nearcolor test case: > > zpty_start > zpty_input 'zmodload zsh/nearcolor' > zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' > zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long > zpty_input 'zle -N rh_widget' > zpty_input 'bindkey "\C-a" rh_widget' > zpty_input $'\C-a' > zpty_line 5 # 3 lines echoed back + 3 empty lines (zpty bug) > zpty_line 1 p > zpty_stop > 0:overlapping region_highlight with true-color > >zmodload zsh/nearcolor > >rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; } > >rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } > >zle -N rh_widget > >bindkey "\C-a" rh_widget > >0m32mt31mr39m32mue39m > >exit > > -- > Sebastian Gniazdowski > News: https://twitter.com/ZdharmaI > IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin > Blog: http://zdharma.org -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-07 14:40 ` Sebastian Gniazdowski @ 2018-11-07 19:19 ` Sebastian Gniazdowski 2018-11-08 3:48 ` Oliver Kiddle 0 siblings, 1 reply; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-07 19:19 UTC (permalink / raw) To: Zsh hackers list I was testing on Linux, via docker. $termcap[Co] shows 8, the same $terminfo[colors]. The zsh/nearcolor fails and maybe it could fail a little better? .... -0m38;5;232mtrue39m +0m30mtrue39m ... Was testing: basic region_highlight with near-color (hex-triplets at input) Color 30 is apparently black. Maybe zsh/nearcolor could use (compare against) first 8 or 16 colors when $termcap[Co] <= 8? ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-07 19:19 ` Sebastian Gniazdowski @ 2018-11-08 3:48 ` Oliver Kiddle 2018-11-08 9:25 ` Sebastian Gniazdowski 0 siblings, 1 reply; 22+ messages in thread From: Oliver Kiddle @ 2018-11-08 3:48 UTC (permalink / raw) To: Zsh hackers list; +Cc: Sebastian Gniazdowski Sebastian Gniazdowski wrote: > I was testing on Linux, via docker. $termcap[Co] shows 8, the same > $terminfo[colors]. The zsh/nearcolor fails and maybe it could fail a > little better? I admit that black, as opposed to the default colour isn't clever. The patch changes this. > Color 30 is apparently black. Maybe zsh/nearcolor could use (compare > against) first 8 or 16 colors when $termcap[Co] <= 8? The first 16 colours vary between terminals and are also commonly configurable so nearcolor never compares against them. There is an escape sequence for getting colour definitions but I don't think trying to use it gains us much. Oliver diff --git a/Src/Modules/nearcolor.c b/Src/Modules/nearcolor.c index 128658e20..7ebc75365 100644 --- a/Src/Modules/nearcolor.c +++ b/Src/Modules/nearcolor.c @@ -117,13 +117,14 @@ mapRGBto256(int red, int green, int blue) static int getnearestcolor(UNUSED(Hookdef dummy), Color_rgb col) { + /* we add 1 to the colours so that colour 0 (black) is + * distinguished from runhookdef() indicating that no + * hook function is registered */ if (tccolours == 256) return mapRGBto256(col->red, col->green, col->blue) + 1; if (tccolours == 88) return mapRGBto88(col->red, col->green, col->blue) + 1; - /* returning 1 indicates black rather than failure (0) so this - * module still serves to prevent fallback on true color */ - return 1; + return -1; } static struct features module_features = { diff --git a/Src/prompt.c b/Src/prompt.c index 377015ad8..568bfc2a9 100644 --- a/Src/prompt.c +++ b/Src/prompt.c @@ -1650,10 +1650,12 @@ match_colour(const char **teststrp, int is_fg, int colour) return TXT_ERROR; *teststrp = end; colour = runhookdef(GETCOLORATTR, &color) - 1; - if (colour < 0) { /* no hook function added, try true color (24-bit) */ + if (colour == -1) { /* no hook function added, try true color (24-bit) */ colour = (((color.red << 8) + color.green) << 8) + color.blue; return on | (is_fg ? TXT_ATTR_FG_24BIT : TXT_ATTR_BG_24BIT) | (zattr)colour << shft; + } else if (colour <= -2) { + return TXT_ERROR; } } else if ((named = ialpha(**teststrp))) { colour = match_named_colour(teststrp); ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-08 3:48 ` Oliver Kiddle @ 2018-11-08 9:25 ` Sebastian Gniazdowski 0 siblings, 0 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-08 9:25 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list I've applied the patch and recompiled, and the tests or the Ctrl-a (with Src/zsh -fi) snippet show now change? ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-07 13:19 [BUG?] If true-color is used, overlapping colors do not work Sebastian Gniazdowski 2018-11-07 14:40 ` Sebastian Gniazdowski @ 2018-11-08 3:03 ` Oliver Kiddle 2018-11-08 9:19 ` Sebastian Gniazdowski 1 sibling, 1 reply; 22+ messages in thread From: Oliver Kiddle @ 2018-11-08 3:03 UTC (permalink / raw) To: Zsh hackers list; +Cc: Sebastian Gniazdowski Sebastian Gniazdowski wrote: > The last test from the attached ztst reveals this. Before-last test > overlapping colors with standard fg=green/fg=red symbols. Result is a > rh2() { region_highlight+=( "1 2 fg=#cc0000" ); } It isn't overlapping colours but the += assignment that fails. I had missed that we have code for converting attributes back to text form and appending to region_highlight apparently relies on that. This does also mean that if you use a short form like fg=#c00, it gets expanded to fg=#cc0000. It also means that with nearcolor, region_highlight reflects the converted value, fg=160 in this case. I don't think this is a problem. There was previously forms that would not be preserved exactly. Thanks for testing the changes. By the way, your tests might be easier to manage if you set fg_start_code, fg_end_code and so on in zle_highlight to some plain text markers. Could be clearer to follow than $'\x1b'. None of your tests work on my system. I think the expected results reflect the particulars of the termcap database for your terminal on your system. zle -T may help but to be portable, the tests may also need to search for a 256color terminal and skip if none is found. Oliver diff --git a/Src/prompt.c b/Src/prompt.c index 284c02475..377015ad8 100644 --- a/Src/prompt.c +++ b/Src/prompt.c @@ -1646,7 +1646,8 @@ match_colour(const char **teststrp, int is_fg, int colour) color.red = col >> 16; color.green = (col & 0xff00) >> 8; color.blue = col & 0xff; - } + } else + return TXT_ERROR; *teststrp = end; colour = runhookdef(GETCOLORATTR, &color) - 1; if (colour < 0) { /* no hook function added, try true color (24-bit) */ @@ -1744,11 +1745,12 @@ match_highlight(const char *teststr, zattr *on_var) /* * Count or output a string for colour information: used - * by output_highlight(). + * by output_highlight(). count when buf is NULL. + * returned count excludes the terminating null byte. */ static int -output_colour(int colour, int fg_bg, int use_tc, char *buf) +output_colour(int colour, int fg_bg, int use_tc, int truecol, char *buf) { int atrlen = 3, len; char *ptr = buf; @@ -1756,8 +1758,12 @@ output_colour(int colour, int fg_bg, int use_tc, char *buf) strcpy(ptr, fg_bg == COL_SEQ_FG ? "fg=" : "bg="); ptr += 3; } + if (truecol) { + /* length of hex triplet always 7, don't need sprintf to count */ + atrlen += buf ? sprintf(ptr, "#%02x%02x%02x", colour >> 16, + (colour >> 8) & 0xff, colour & 0xff) : 7; /* colour should only be > 7 if using termcap but let's be safe */ - if (use_tc || colour > 7) { + } else if (use_tc || colour > 7) { char digbuf[DIGBUFSIZE]; sprintf(digbuf, "%d", colour); len = strlen(digbuf); @@ -1795,6 +1801,7 @@ output_highlight(zattr atr, char *buf) len = output_colour(txtchangeget(atr, TXT_ATTR_FG_COL), COL_SEQ_FG, (atr & TXT_ATTR_FG_TERMCAP), + (atr & TXT_ATTR_FG_24BIT), ptr); atrlen += len; if (buf) @@ -1811,6 +1818,7 @@ output_highlight(zattr atr, char *buf) len = output_colour(txtchangeget(atr, TXT_ATTR_BG_COL), COL_SEQ_BG, (atr & TXT_ATTR_BG_TERMCAP), + (atr & TXT_ATTR_BG_24BIT), ptr); atrlen += len; if (buf) ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-08 3:03 ` Oliver Kiddle @ 2018-11-08 9:19 ` Sebastian Gniazdowski 2018-11-09 1:28 ` Oliver Kiddle 0 siblings, 1 reply; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-08 9:19 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list On Thu, 8 Nov 2018 at 04:03, Oliver Kiddle <okiddle@yahoo.co.uk> wrote: > (...) > markers. Could be clearer to follow than $'\x1b'. None of your tests > work on my system. I think the expected results reflect the particulars > of the termcap database for your terminal on your system. zle -T may > help but to be portable, the tests may also need to search for a > 256color terminal and skip if none is found. I hope the tests will be working system-wide, otherwise it's a show stopper for them. Could you maybe provide the test wrong answer? If you are sure that it's about TERM= value, then ok, I've observed this too, and export TERM=xterm-256color fixed the tests (on Linux). Searching for proper termcap seems a good idea. I might also find a docker image with the same OS and test it. Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-08 9:19 ` Sebastian Gniazdowski @ 2018-11-09 1:28 ` Oliver Kiddle 2018-11-09 15:39 ` Sebastian Gniazdowski ` (2 more replies) 0 siblings, 3 replies; 22+ messages in thread From: Oliver Kiddle @ 2018-11-09 1:28 UTC (permalink / raw) To: Zsh hackers list; +Cc: Sebastian Gniazdowski Sebastian Gniazdowski wrote: > I've applied the patch and recompiled, and the tests or the Ctrl-a > (with Src/zsh -fi) snippet show now change? Working through each of your tests manually, they all seem to work fine for me if judged purely on the basis of what I see in a terminal. I've checked more than one terminal programme including ones with and without true colour support. > I hope the tests will be working system-wide, otherwise it's a show > stopper for them. Could you maybe provide the test wrong answer? If > you are sure that it's about TERM= value, then ok, I've observed this > too, and export TERM=xterm-256color fixed the tests (on Linux). > Searching for proper termcap seems a good idea. I might also find a > docker image with the same OS and test it. I get differences before the tests even reach the final buffer state. The fact that all the setup lines get repeated back in the test output really isn't helping the matter. The differences are all escape sequences which diff is not showing, e.g 1b 5b 6d occurring at the start of each line. There are also further differences that can be put down to a difference of termcap definition. The only hope would be to strip the output down to just the section actually being tested (which comptest manages), also using something like: zle_highlight=( fg_start_code:FGSTART fg_end_code:FGEND fg_default_code:FGDEF ) and perhaps zle -T somehow to map escape sequences to something readable somehow contriving to set TERM to something that claims 256 colour support. And, only relying on the portable subset of sed syntax. It's not without reason that I was too lazy to attempt this in the first place. Oliver ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-09 1:28 ` Oliver Kiddle @ 2018-11-09 15:39 ` Sebastian Gniazdowski 2018-11-11 0:43 ` Sebastian Gniazdowski 2018-11-11 5:38 ` X04 zle highlight tests, near-color bug Sebastian Gniazdowski 2 siblings, 0 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-09 15:39 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list On Fri, 9 Nov 2018 at 02:28, Oliver Kiddle <okiddle@yahoo.co.uk> wrote: > > Sebastian Gniazdowski wrote: > > I've applied the patch and recompiled, and the tests or the Ctrl-a > > (with Src/zsh -fi) snippet show now change? > > Working through each of your tests manually, they all seem to work fine > for me if judged purely on the basis of what I see in a terminal. I've > checked more than one terminal programme including ones with and without > true colour support. Ah I've applied only the black color patch, with also the first patch, everything works fine. > I get differences before the tests even reach the final buffer state. The > fact that all the setup lines get repeated back in the test output > really isn't helping the matter. The differences are all escape > sequences which diff is not showing, e.g 1b 5b 6d occurring at the start > of each line. There are also further differences that can be put down to a > difference of termcap definition. The only hope would be to strip the output > down to just the section actually being tested (which comptest manages), > also using something like: > zle_highlight=( fg_start_code:FGSTART fg_end_code:FGEND fg_default_code:FGDEF ) > and perhaps zle -T somehow to map escape sequences to something readable > somehow contriving to set TERM to something that claims 256 colour support. > And, only relying on the portable subset of sed syntax. > It's not without reason that I was too lazy to attempt this in the first > place. I'm stripping down the escape codes that Zsh/Zle normally produces managing output. It appears that your setup generates some additional codes. This should be only a matter of altering // substitution and sed invocation, but maybe your right, maybe updating zle_highlight / fg_start_code / fg_end_code is the right way. -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-09 1:28 ` Oliver Kiddle 2018-11-09 15:39 ` Sebastian Gniazdowski @ 2018-11-11 0:43 ` Sebastian Gniazdowski 2018-11-11 5:11 ` Sebastian Gniazdowski 2018-11-24 17:32 ` highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) Oliver Kiddle 2018-11-11 5:38 ` X04 zle highlight tests, near-color bug Sebastian Gniazdowski 2 siblings, 2 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-11 0:43 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list [-- Attachment #1: Type: text/plain, Size: 2347 bytes --] On Fri, 9 Nov 2018 at 02:28, Oliver Kiddle <okiddle@yahoo.co.uk> wrote: >(...) >The only hope would be to strip the output > down to just the section actually being tested (which comptest manages), > also using something like: > zle_highlight=( fg_start_code:FGSTART fg_end_code:FGEND fg_default_code:FGDEF ) > and perhaps zle -T somehow to map escape sequences to something readable > somehow contriving to set TERM to something that claims 256 colour support. > And, only relying on the portable subset of sed syntax. > It's not without reason that I was too lazy to attempt this in the first > place. I have new fresh test file which I hope brings the X04 test to some new stage or level. The lines sent to zpty now aren't repeated, because zsh is started with +Z option, and `setopt zle' is emitted just before the "zpty -w zsh $'\C-a'" line, and the shell / zpty is closed via Ctrl-D, so luckily only single one, the test line (BUFFER="true", region_highlight+=...) is emitted and read. I've used zle -T tc tcfunc, which sets REPLY="", i.e. discards the codes. Should I change them to something? Because I'm only getting LE termcap command, moving cursor to the left (after printing POSTDISPLAY with zsh-autosuggestions text), some cd, ce, up commands and no codes for colors. Do you think I should not strip the termcap codes and specify them in test's expected output? I'm currently removing following escapes from all zpty output: - ^[[27m, ^[[24m, etc. - ^[[J - ^[[K - ^[[?2004h - e^Mexit -> exit - ^[[?2004l You seem to retrieve some other Zle-managing-output code (I think it's this) and the tests fail? Maybe it is already fixed by the late-Zle enabling, so you could try running the new test file? There are 6 tests currently: 0:basic region_highlight with 8 colors 0:basic region_highlight with true-color (hex-triplets) 0:basic region_highlight with near-color (hex-triplets at input) 0:overlapping region_highlight with 8 colors 0:overlapping region_highlight with true-color 0:overlapping region_highlight with near-color (hex-triplets at input) PS. Didn't yet search for terminfo file, just did export TERM=xterm-256, not actually sure how to search for the definition -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org [-- Attachment #2: X04highlight.ztst.txt --] [-- Type: text/plain, Size: 5469 bytes --] # Tests for region_highlight, true-color support, near-color support %prep if [[ $OSTYPE == cygwin ]]; then ZTST_unimplemented='the zsh/zpty module does not work on Cygwin' elif zmodload zsh/zpty 2> /dev/null; then export TERM=xterm-256color zpty_start() { export PS1= PS2= zpty -d zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z" } zpty_input() { zpty ${${(M)2:#nonl}:+-n} -w zsh "$1" } zpty_enable_zle() { zpty -w zsh "tcfunc() { REPLY=""; }" zpty -w zsh "setopt zle; zle -T tc tcfunc" } zpty_line() { setopt localoptions extendedglob noshwordsplit local REPLY cm=$'\r' integer i for (( i = 0; i < ${1:-1}; ++i )); do zpty -r zsh REPLY # P is for "preserve", induces keeping some # color codes to test region_highlight, etc. # The color codes are then made regular text [[ "$2" = "p" ]] && { print -rl -- "$REPLY" >> /tmp/output # THE LINE REPLY=${REPLY//$'\x1b'\[([2][0-9;]m|[JK]|\?[0-9]##(h|l))/} REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}} } || { REPLY=${REPLY//$'\x1b'\[([0-9;]##m|[JK]|\?[0-9]##(h|l))/} } # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2 # otherwise replace with \1 stripped out of leading/trailing [[:space:]] REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}} [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##} done } zpty_stop() { setopt localoptions extendedglob local REPLY cm=$'\r' # Zle is active, can use Ctrl-D to exit zpty -n -w zsh $'\C-d' # zpty gives no output when piped without these braces (?) # The while loop with // substitution is to convert `e^Mexit' # into `exit' (see zpty_line). The sed commands remove escapes { zpty -r zsh } | sed $'/[^[:space:]]/!d; s/\r$//; s/\x1b\\[[0-9;]*m//g; s/\x1b\\[[JK]//g; s/\x1b\\[?[0-9]*[hl]//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done zpty -d : } else ZTST_unimplemented='the zsh/zpty module is not available' fi %test zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with 8 colors >0m32mtrue39m zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with true-color (hex-triplets) >0m38;2;4;8;16mtrue39m zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with near-color (hex-triplets at input) >0m38;5;232mtrue39m zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with 8 colors >0m32mt31mr39m32mue39m zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with true-color >0m38;2;0;204;0mt38;2;204;0;0mr39m38;2;0;204;0mue39m zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with near-color (hex-triplets at input) >0m38;5;40mt38;5;160mr39m38;5;40mue39m %clean zmodload -ui zsh/zpty # vim:ft=zsh ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG?] If true-color is used, overlapping colors do not work 2018-11-11 0:43 ` Sebastian Gniazdowski @ 2018-11-11 5:11 ` Sebastian Gniazdowski 2018-11-24 17:32 ` highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) Oliver Kiddle 1 sibling, 0 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-11 5:11 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list On Sun, 11 Nov 2018 at 01:43, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > I've used zle -T tc tcfunc, which sets REPLY="", i.e. discards the > codes. Should I change them to something? Because I'm only getting LE > termcap command, moving cursor to the left (after printing POSTDISPLAY > with zsh-autosuggestions text), some cd, ce, up commands and no codes > for colors. Do you think I should not strip the termcap codes and > specify them in test's expected output? That was for non-zsh-f full featured Zsh session. I've added storing of the termcap commands within the test. It's a sequence of only 2 codes, cd & ce: <tc=cd,arg=> <tc=ce,arg=> <tc=ce,arg=> <tc=cd,arg=> ... I've checked their codes, they are the ^[[J, ^[[K that I'm removing by // and in other (separate in execution) place by analogous sed invocation. No other termcap command gets invoked during make TESTNUM=X04. I wonder if maybe on your setup there is some other termcap command invoked? > I'm currently removing following escapes from all zpty output: > - ^[[27m, ^[[24m, etc. > - ^[[J > - ^[[K > - ^[[?2004h > - e^Mexit -> exit > - ^[[?2004l > > You seem to retrieve some other Zle-managing-output code (I think it's > this) and the tests fail? Maybe it is already fixed by the late-Zle > enabling, so you could try running the new test file? > > There are 6 tests currently: > > 0:basic region_highlight with 8 colors > 0:basic region_highlight with true-color (hex-triplets) > 0:basic region_highlight with near-color (hex-triplets at input) > 0:overlapping region_highlight with 8 colors > 0:overlapping region_highlight with true-color > 0:overlapping region_highlight with near-color (hex-triplets at input) > > PS. Didn't yet search for terminfo file, just did export > TERM=xterm-256, not actually sure how to search for the definition > -- > Sebastian Gniazdowski > News: https://twitter.com/ZdharmaI > IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin > Blog: http://zdharma.org -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 22+ messages in thread
* highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) 2018-11-11 0:43 ` Sebastian Gniazdowski 2018-11-11 5:11 ` Sebastian Gniazdowski @ 2018-11-24 17:32 ` Oliver Kiddle 2018-11-30 0:34 ` Sebastian Gniazdowski 1 sibling, 1 reply; 22+ messages in thread From: Oliver Kiddle @ 2018-11-24 17:32 UTC (permalink / raw) To: Sebastian Gniazdowski; +Cc: Zsh hackers list On 11 Nov, Sebastian Gniazdowski wrote: > I have new fresh test file which I hope brings the X04 test to some > new stage or level. The lines sent to zpty now aren't repeated, Sorry, I hadn't found time to look at this. Given that ensuring portability has proved tricky with the existing zpty based tests I'm rather leery of including new ones and would be happier if someone else were to take a look. The tests now pass on my system. > I've used zle -T tc tcfunc, which sets REPLY="", i.e. discards the > codes. Should I change them to something? Because I'm only getting LE If removing them works for the particular tests then I'd have thought this was fine. > I'm currently removing following escapes from all zpty output: > - ^[[?2004h > - ^[[?2004l unsetting zle_bracketed_paste will prevent zsh from generating these which is better than removing them afterwards. > PS. Didn't yet search for terminfo file, just did export > TERM=xterm-256, not actually sure how to search for the definition Searching probably isn't easy to do portably. Perhaps guessing like this is better, but it would be good to check $termcap[Co] afterwards and skip tests as appropriate. > { zpty -r zsh } | sed $'/[^[:space:]]/!d; s/\r$//; s/\x1b\\[[0-9;]*m//g; s/\x1b\\[[JK]//g; s/\x1b\\[?[0-9]*[hl]//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done This use of sed isn't portable. In particular, the use of semi-colons to separate commands and the [[:space:]] character class are extensions. You can use literal spaces and multiple -e options. Either that or do the substitutions in zsh. The heirloom project (http://heirloom.sourceforge.net/tools.html) is a good source for lowest common denominator versions of Unix tools if you don't have an old OS available. Oliver ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) 2018-11-24 17:32 ` highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) Oliver Kiddle @ 2018-11-30 0:34 ` Sebastian Gniazdowski 2018-12-07 1:55 ` [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases) Sebastian Gniazdowski 2018-12-10 2:54 ` highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) Oliver Kiddle 0 siblings, 2 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-30 0:34 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list [-- Attachment #1: Type: text/plain, Size: 2817 bytes --] On Sat, 24 Nov 2018 at 18:32, Oliver Kiddle <okiddle@yahoo.co.uk> wrote: > > On 11 Nov, Sebastian Gniazdowski wrote: > > > I've used zle -T tc tcfunc, which sets REPLY="", i.e. discards the > > codes. Should I change them to something? Because I'm only getting LE > > If removing them works for the particular tests then I'd have thought > this was fine. Ok, so I've also removed the deleting of \[[J and \[[K codes (result: simpler patterns and regexes), which are covered by the "zle -T ..." sink. > > I'm currently removing following escapes from all zpty output: > > - ^[[?2004h > > - ^[[?2004l > > unsetting zle_bracketed_paste will prevent zsh from generating these > which is better than removing them afterwards. I'm using this now, the codes are indeed not generated now. One thing, I've had to call `unset zle_brack...' after `setopt zle', not before it. So the sed command is now quite simple, only three "-e" arguments: { zpty -r zsh } | sed -e $'/[^\t\r ]/!d' -e $'s/\r$//' -e $'s/\x1b\\[[0-9;]*m//g' | while ... > > PS. Didn't yet search for terminfo file, just did export > > TERM=xterm-256, not actually sure how to search for the definition > > Searching probably isn't easy to do portably. Perhaps guessing like this > is better, but it would be good to check $termcap[Co] afterwards and > skip tests as appropriate. This is now done, i.e. termcap[Co] is being check: export TERM=xterm-256color if [[ ${+termcap} != 1 || ${termcap[Co]} != <-> || ${termcap[Co]} -lt 256 ]]; then ZTST_unimplemented="no termcap module OR termcap doesn't support 256 or more colors" elif [[ $OSTYPE == cygwin ]]; then ZTST_unimplemented='the zsh/zpty module does not work on Cygwin' elif zmodload zsh/zpty 2> /dev/null; then zpty_start() { ... > This use of sed isn't portable. In particular, the use of semi-colons to > separate commands and the [[:space:]] character class are extensions. > You can use literal spaces and multiple -e options. Either that > or do the substitutions in zsh. The current sed invocation is pasted above – sed is now using literal spaces instead of [[:space:]], multiple -e commands instead of semicolon and is simplified. Now, the bug with zle_highlight + zsh/nearcolor: -0mCDE|38;5;232|trueCDE|39| +0m38;5;232mtrueCDE|39| Test ./X04zlehighlight.ztst failed: output differs from expected as shown above for: ... Was testing: basic region_highlight with near-color (hex-triplets at input) The output doesn't follow zsh_highlight replacements for start-code and end-code and still emits raw codes. Could this be fixed? -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org [-- Attachment #2: X04zlehighlight.ztst.txt --] [-- Type: text/plain, Size: 5915 bytes --] # Tests for region_highlight, true-color support, near-color support # Version 0.5 %prep export TERM=xterm-256color if [[ ${+termcap} != 1 || ${termcap[Co]} != <-> || ${termcap[Co]} -lt 256 ]]; then ZTST_unimplemented="no termcap module OR termcap doesn't support 256 or more colors" elif [[ $OSTYPE == cygwin ]]; then ZTST_unimplemented='the zsh/zpty module does not work on Cygwin' elif zmodload zsh/zpty 2> /dev/null; then zpty_start() { export PS1= PS2= zpty -d zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z" zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|" bg_start_code:"BCDE|4" bg_end_code:"|" )' } zpty_input() { zpty ${${(M)2:#nonl}:+-n} -w zsh "$1" } zpty_enable_zle() { zpty -w zsh "tcfunc() { REPLY=""; }" # This line will not be echoed back, behaving like ! -o zle zpty -w zsh "setopt zle; zle -T tc tcfunc; unset zle_bracketed_paste" } zpty_line() { setopt localoptions extendedglob noshwordsplit local REPLY cm=$'\r' integer i for (( i = 0; i < ${1:-1}; ++i )); do zpty -r zsh REPLY # P is for "preserve", and induces keeping some # expected color codes to test region_highlight, # etc. - the color codes are made a regular text. [[ "$2" = "p" ]] && { REPLY=${REPLY//$'\x1b'\[([2][0-9;]m|\?[0-9]##(h|l))/} # remove only 2...m codes REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}} } || { REPLY=${REPLY//$'\x1b'\[([0-9;]##m|\?[0-9]##(h|l))/} # remove all [0-9]...m codes } # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2 # otherwise replace with \1 stripped out of leading/trailing [[:space:]] REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}} [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##} done } zpty_stop() { setopt localoptions extendedglob local REPLY cm=$'\r' # Zle is active, can use Ctrl-D to exit zpty -n -w zsh $'\C-d' # zpty gives no output when piped without these braces (?) # The while loop with // substitution is to convert `e^Mexit' # into `exit' (see zpty_line). The sed commands remove escapes { zpty -r zsh } | sed -e $'/[^\t\r ]/!d' -e $'s/\r$//' -e $'s/\x1b\\[[0-9;]*m//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done zpty -d : } else ZTST_unimplemented='the zsh/zpty module is not available' fi %test zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with 8 colors >0mCDE|32|trueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with true-color (hex-triplets) >0mCDE|38;2;4;8;16|trueCDE|39| zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with near-color (hex-triplets at input) >0mCDE|38;5;232|trueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with 8 colors >0mCDE|32|tCDE|31|rCDE|39|CDE|32|ueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with true-color >0mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39| zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with near-color (hex-triplets at input) >0mCDE|38;5;40|tCDE|38;5;160|rCDE|39|CDE|38;5;40|ueCDE|39| %clean zmodload -ui zsh/zpty # vim:ft=zsh ^ permalink raw reply [flat|nested] 22+ messages in thread
* [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases) 2018-11-30 0:34 ` Sebastian Gniazdowski @ 2018-12-07 1:55 ` Sebastian Gniazdowski 2018-12-07 20:26 ` Sebastian Gniazdowski 2018-12-10 2:54 ` highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) Oliver Kiddle 1 sibling, 1 reply; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-12-07 1:55 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list [-- Attachment #1: Type: text/plain, Size: 1254 bytes --] On Fri, 30 Nov 2018 at 01:34, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > The output doesn't follow zsh_highlight replacements for start-code > and end-code and still emits raw codes. > > Could this be fixed? Turns out this doesn't involve near-color at all, only 256 color codes. The new standout-targeted test returns: -0m27m24mtr7mu27me word2 word3 +0m27m24m38;5;196mtr7m38;5;196mu27meCDE|39| word2 word3 Was testing: region highlight - standout overlapping on other region_highlight entry So again no CDE|3...|, but raw ^[38;5;196m. So this is a general Zsh bug, not near-color bug, as zsh/near-color module isn't loaded in this test, it's only the 256 code "fg=196" that is being used. The setting: zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|" bg_start_code:"BCDE|4" bg_end_code:"|" ) The file *-256.ztst.txt contains this test case. The file *-standout-only.ztst.txt is the expected test case which shows no problems with standout-overlaping other styles. PS. I now also do not remove 2xm codes (underline, standaout) but the test cases are all adapted to this change. -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org [-- Attachment #2: X04zlehighlight-256.ztst.txt --] [-- Type: text/plain, Size: 6383 bytes --] # Tests for region_highlight, true-color support, near-color support # Version 0.7 2018-12-06 %prep export TERM=xterm-256color if [[ ${+termcap} != 1 || ${termcap[Co]} != <-> || ${termcap[Co]} -lt 256 ]]; then ZTST_unimplemented="no termcap module OR termcap doesn't support 256 or more colors" elif [[ $OSTYPE == cygwin ]]; then ZTST_unimplemented='the zsh/zpty module does not work on Cygwin' elif zmodload zsh/zpty 2> /dev/null; then zpty_start() { export PS1= PS2= zpty -d zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z" zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|" bg_start_code:"BCDE|4" bg_end_code:"|" )' } zpty_input() { zpty ${${(M)2:#nonl}:+-n} -w zsh "$1" } zpty_enable_zle() { zpty -w zsh "tcfunc() { REPLY=""; }" # This line will not be echoed back, behaving like ! -o zle zpty -w zsh "setopt zle; zle -T tc tcfunc; unset zle_bracketed_paste" } zpty_line() { setopt localoptions extendedglob noshwordsplit local REPLY cm=$'\r' integer i for (( i = 0; i < ${1:-1}; ++i )); do zpty -r zsh REPLY # P is for "preserve", and induces keeping some # expected color codes to test region_highlight, # etc. - the color codes are made a regular text. [[ "$2" = "p" ]] && { REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}} } || { REPLY=${REPLY//$'\x1b'\[[0-9;]##m/} # remove all [0-9]...m codes } # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2 # otherwise replace with \1 stripped out of leading/trailing [[:space:]] REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}} [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##} done } zpty_stop() { setopt localoptions extendedglob local REPLY cm=$'\r' # Zle is active, can use Ctrl-D to exit zpty -n -w zsh $'\C-d' # zpty gives no output when piped without these braces (?) # The while loop with // substitution is to convert `e^Mexit' # into `exit' (see zpty_line). The sed commands remove escapes { zpty -r zsh } | sed -e $'/[^\t\r ]/!d' -e $'s/\r$//' -e $'s/\x1b\\[[0-9;]*m//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done zpty -d : } else ZTST_unimplemented='the zsh/zpty module is not available' fi %test zpty_start zpty_input 'rh_widget() { BUFFER="true word2 word3"; region_highlight+=( "0 4 fg=196" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "2 3 standout" ); };' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:region highlight - standout overlapping on other region_highlight entry >0m27m24mtr7mu27me word2 word3 zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with 8 colors >0m27m24mCDE|32|trueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with true-color (hex-triplets) >0m27m24mCDE|38;2;4;8;16|trueCDE|39| zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with near-color (hex-triplets at input) >0m27m24mCDE|38;5;232|trueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with 8 colors >0m27m24mCDE|32|tCDE|31|rCDE|39|CDE|32|ueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with true-color >0m27m24mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39| zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with near-color (hex-triplets at input) >0m27m24mCDE|38;5;40|tCDE|38;5;160|rCDE|39|CDE|38;5;40|ueCDE|39| %clean zmodload -ui zsh/zpty # vim:ft=zsh [-- Attachment #3: X04zlehighlight-standout-only.ztst.txt --] [-- Type: text/plain, Size: 6403 bytes --] # Tests for region_highlight, true-color support, near-color support # Version 0.7 2018-12-06 %prep export TERM=xterm-256color if [[ ${+termcap} != 1 || ${termcap[Co]} != <-> || ${termcap[Co]} -lt 256 ]]; then ZTST_unimplemented="no termcap module OR termcap doesn't support 256 or more colors" elif [[ $OSTYPE == cygwin ]]; then ZTST_unimplemented='the zsh/zpty module does not work on Cygwin' elif zmodload zsh/zpty 2> /dev/null; then zpty_start() { export PS1= PS2= zpty -d zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z" zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|" bg_start_code:"BCDE|4" bg_end_code:"|" )' } zpty_input() { zpty ${${(M)2:#nonl}:+-n} -w zsh "$1" } zpty_enable_zle() { zpty -w zsh "tcfunc() { REPLY=""; }" # This line will not be echoed back, behaving like ! -o zle zpty -w zsh "setopt zle; zle -T tc tcfunc; unset zle_bracketed_paste" } zpty_line() { setopt localoptions extendedglob noshwordsplit local REPLY cm=$'\r' integer i for (( i = 0; i < ${1:-1}; ++i )); do zpty -r zsh REPLY # P is for "preserve", and induces keeping some # expected color codes to test region_highlight, # etc. - the color codes are made a regular text. [[ "$2" = "p" ]] && { REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}} } || { REPLY=${REPLY//$'\x1b'\[[0-9;]##m/} # remove all [0-9]...m codes } # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2 # otherwise replace with \1 stripped out of leading/trailing [[:space:]] REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}} [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##} done } zpty_stop() { setopt localoptions extendedglob local REPLY cm=$'\r' # Zle is active, can use Ctrl-D to exit zpty -n -w zsh $'\C-d' # zpty gives no output when piped without these braces (?) # The while loop with // substitution is to convert `e^Mexit' # into `exit' (see zpty_line). The sed commands remove escapes { zpty -r zsh } | sed -e $'/[^\t\r ]/!d' -e $'s/\r$//' -e $'s/\x1b\\[[0-9;]*m//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done zpty -d : } else ZTST_unimplemented='the zsh/zpty module is not available' fi %test zpty_start zpty_input 'rh_widget() { BUFFER="true word2 word3"; region_highlight+=( "0 4 fg=196" ); rh2; }' zpty_input 'rh2() { region_highlight=( "2 3 standout" ); };' # note the =, not += zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:region highlight - standout overlapping on other region_highlight entry >0m27m24mtr7mu27me word2 word3 zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with 8 colors >0m27m24mCDE|32|trueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with true-color (hex-triplets) >0m27m24mCDE|38;2;4;8;16|trueCDE|39| zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with near-color (hex-triplets at input) >0m27m24mCDE|38;5;232|trueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with 8 colors >0m27m24mCDE|32|tCDE|31|rCDE|39|CDE|32|ueCDE|39| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with true-color >0m27m24mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39| zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with near-color (hex-triplets at input) >0m27m24mCDE|38;5;40|tCDE|38;5;160|rCDE|39|CDE|38;5;40|ueCDE|39| %clean zmodload -ui zsh/zpty # vim:ft=zsh ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases) 2018-12-07 1:55 ` [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases) Sebastian Gniazdowski @ 2018-12-07 20:26 ` Sebastian Gniazdowski 2018-12-09 19:13 ` Daniel Shahaf 0 siblings, 1 reply; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-12-07 20:26 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list On Fri, 7 Dec 2018 at 02:55, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > So again no CDE|3...|, but raw ^[38;5;196m. So this is a general Zsh > bug, not near-color bug, as zsh/near-color module isn't loaded in this > test, it's only the 256 code "fg=196" that is being used. Following Daniel's request of a minimal test case that makes the problem visible: widget() { zle_highlight=( fg_start_code:"CDE|3" fg_end_code:"|" bg_start_code:"BCDE|4" bg_end_code:"|" ); BUFFER="true word2 word3"; region_highlight=( "0 4 fg=196" ); } zle -N widget bindkey '^T' widget And after pasting and accepting in zsh -f, press Ctrl-T. You will see red word "true" followed by "CDE|39|" and the other words, i.e. % trueCDE|39| word2 word3 (the true is in red, but cannot visualize this in email). So: zsh_highlight was ignored for 256 color code, only code ^[[39m has been processed by zle_highlight, while the fg=196 code was output unprocessed, i.e. as ^[[38;5;196m. Could the problem be fixed, .. ASAP? It collides with my work of providing X04 tests and is a blocker for making a new Zsh release. I think Oliver touched the responsible code lately, as for true-color (not 256 color, with or withoutzsh/nearcolor), the zle_highlight processing works, i.e. I'm getting, e.g.: 0m27m24mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39| in an X04 test case. So all 24-bit escape codes are nicely covered with CDE| etc. things from zle_highlight. So maybe it seems that the "get text representation from already processed entries in region_highlight" that Olivier did is used also here. So basically 256-color doesn't obey zle_highlight, while first 8/16 colors and TrueColor are obeying it. -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases) 2018-12-07 20:26 ` Sebastian Gniazdowski @ 2018-12-09 19:13 ` Daniel Shahaf 2018-12-11 8:06 ` Sebastian Gniazdowski 0 siblings, 1 reply; 22+ messages in thread From: Daniel Shahaf @ 2018-12-09 19:13 UTC (permalink / raw) To: zsh-workers Sebastian Gniazdowski wrote on Fri, 07 Dec 2018 21:26 +0100: > Following Daniel's request of a minimal test case that makes the > problem visible: > Thanks for the test case. > Could the problem be fixed, .. ASAP? It collides with my work of > providing X04 tests and is a blocker for making a new Zsh > release Why should it be a blocker? (I don't disagree; I just don't see the reason.) I ran the minimal reproducer in 5.6.2 and in current master and I get the same behaviour in both of them ("trueCDE|39| word2 word3", the whole thing in red), so it's not a regression. > I think Oliver touched the responsible code lately, as for > true-color (not 256 color, with or withoutzsh/nearcolor), the > zle_highlight processing works, i.e. I'm getting, e.g.: > > 0m27m24mCDE|38;2;0;204;0|tCDE|38;2;204;0;0|rCDE|39|CDE|38;2;0;204;0|ueCDE|39| Cheers, Daniel ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases) 2018-12-09 19:13 ` Daniel Shahaf @ 2018-12-11 8:06 ` Sebastian Gniazdowski 0 siblings, 0 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-12-11 8:06 UTC (permalink / raw) To: Daniel Shahaf; +Cc: Zsh hackers list BTW. The mail appeared 2 days after sending. On Sun, 9 Dec 2018 at 20:23, Daniel Shahaf <d.s@daniel.shahaf.name> wrote: > > Sebastian Gniazdowski wrote on Fri, 07 Dec 2018 21:26 +0100: > > Could the problem be fixed, .. ASAP? It collides with my work of > > providing X04 tests and is a blocker for making a new Zsh > > release > > Why should it be a blocker? (I don't disagree; I just don't see the reason.) I was thinking that ignoring zle_highlight will be considered a bug, and this would be some blocker, to do a new release openly with known unfixed bug. But it turned out that zle_highlight isn't that much trusted and the code deliberately opts for termcap instead of using it. I think the compromise "use termcap for default zle_highlight, but for customized zle_highlight use it, not the termcap" addresses this with a balanced, old-man-wise compromise. -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) 2018-11-30 0:34 ` Sebastian Gniazdowski 2018-12-07 1:55 ` [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases) Sebastian Gniazdowski @ 2018-12-10 2:54 ` Oliver Kiddle 2018-12-10 23:51 ` Sebastian Gniazdowski 1 sibling, 1 reply; 22+ messages in thread From: Oliver Kiddle @ 2018-12-10 2:54 UTC (permalink / raw) To: Sebastian Gniazdowski; +Cc: Zsh hackers list On 30 Nov, Sebastian Gniazdowski wrote: > The output doesn't follow zsh_highlight replacements for start-code > and end-code and still emits raw codes. > > Could this be fixed? That was intentional. As I stated in 43759 which was the message including the true color patch: | The actual escape sequences appear to be quite standard and are, for | now, hard coded. We should probably support some zle_highlight fields | similar to fg_start_code to allow them to be configured but I'm unsure | of what exact form that should take. It may not be good enough to overload "fg_start_code" because the escape sequences might vary for 16, 256 or true colours. Quite why termcap isn't used for the first 16 colours, I couldn't say but I'd have thought it is always better to be using termcap if possible - contrary to your change in 43875. Apart from that concern about overloading the existing fields, I don't object to using zle_highlight as an override or fallback for termcap if that's somehow useful. In the case of true colour, start and end code may not be enough to encapsulate all the relevant details. The sequence I used has decimal numbers separated by semi-colons but judging from https://iterm2.com/documentation-escape-codes.html there's a macOS terminal emulator that may want hex triplets. This is what I was referring to when saying I was unsure what form the zle_highlight fields should take. The current hard coded escape sequences work for everything I've tried. Oliver ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) 2018-12-10 2:54 ` highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) Oliver Kiddle @ 2018-12-10 23:51 ` Sebastian Gniazdowski 2018-12-10 23:54 ` Sebastian Gniazdowski 0 siblings, 1 reply; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-12-10 23:51 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list On Mon, 10 Dec 2018 at 03:54, Oliver Kiddle <okiddle@yahoo.co.uk> wrote: > > On 30 Nov, Sebastian Gniazdowski wrote: > > The output doesn't follow zsh_highlight replacements for start-code > > and end-code and still emits raw codes. > > > > Could this be fixed? > > That was intentional. As I stated in 43759 which was the message > including the true color patch: > > | The actual escape sequences appear to be quite standard and are, for > | now, hard coded. We should probably support some zle_highlight fields > | similar to fg_start_code to allow them to be configured but I'm unsure > | of what exact form that should take. That said, not everything was hard-coded – zle_highlight was respected by the true color patch. > it is always better to be using termcap if possible - contrary to your > change in 43875. The patch could be changed to use zle_highlight only if its values aren't the currently-well-known defaults. This way, if user will not customize the codes that are being sent, he will choose to use termcap, and be protected in the unknown-to-exists situation of running Zsh on a terminal with custom 256 color codes (i.e. not \e[38;5;N). But when user will customize zle_highlight, then termcap will not be used but the customization instead will be. User customizes --> he want's thing to be used (of course, he's demand could be selective, only for 256 colors or only for True Color, hehe). > The > sequence I used has decimal numbers separated by semi-colons but judging > from https://iterm2.com/documentation-escape-codes.html there's a > macOS terminal emulator that may want hex triplets. This is what I was > referring to when saying I was unsure what form the zle_highlight fields > should take. The current hard coded escape sequences work for everything > I've tried. The iTerm2 code is for changing the terminal's palette. With the "untouched zle_highlight -> termcap" condition, this would be covered anyway. If user will customize zle_highlight for e.g. tests, he will be able to do <<TERMINAL==XYZ -> ZTST_unimplemented='terminal with non-standard-codes'>> and simply don't bother about a specific machine. And, in this situation, he actually may be interested to still run the tests, even on such custom terminal, as the codes output by Zsh will be standard anyway (and TERM will be xterm-256color anyway). > Oliver -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) 2018-12-10 23:51 ` Sebastian Gniazdowski @ 2018-12-10 23:54 ` Sebastian Gniazdowski 0 siblings, 0 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-12-10 23:54 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list On Tue, 11 Dec 2018 at 00:51, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > On Mon, 10 Dec 2018 at 03:54, Oliver Kiddle <okiddle@yahoo.co.uk> wrote: > > > > On 30 Nov, Sebastian Gniazdowski wrote: > > > The output doesn't follow zsh_highlight replacements for start-code > > > and end-code and still emits raw codes. > > > > > > Could this be fixed? > > > > That was intentional. As I stated in 43759 which was the message > > including the true color patch: > > > > | The actual escape sequences appear to be quite standard and are, for > > | now, hard coded. We should probably support some zle_highlight fields > > | similar to fg_start_code to allow them to be configured but I'm unsure > > | of what exact form that should take. > > That said, not everything was hard-coded – zle_highlight was respected > by the true color patch. Oh, I know see you were referring to additional zle_highlight fields. I think this means that the uncommon situation can be responded to by an update to Zshell (i.e. introduction of additional zle_highlight field(s)). -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 22+ messages in thread
* X04 zle highlight tests, near-color bug 2018-11-09 1:28 ` Oliver Kiddle 2018-11-09 15:39 ` Sebastian Gniazdowski 2018-11-11 0:43 ` Sebastian Gniazdowski @ 2018-11-11 5:38 ` Sebastian Gniazdowski 2018-11-18 15:34 ` Sebastian Gniazdowski 2 siblings, 1 reply; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-11 5:38 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list [-- Attachment #1: Type: text/plain, Size: 1982 bytes --] On Fri, 9 Nov 2018 at 02:28, Oliver Kiddle <okiddle@yahoo.co.uk> wrote: > difference of termcap definition. The only hope would be to strip the output > down to just the section actually being tested (which comptest manages), > also using something like: > zle_highlight=( fg_start_code:FGSTART fg_end_code:FGEND fg_default_code:FGDEF ) > and perhaps zle -T somehow to map escape sequences to something readable > somehow contriving to set TERM to something that claims 256 colour support. > And, only relying on the portable subset of sed syntax. > It's not without reason that I was too lazy to attempt this in the first > place. I'm sending the same uplifted, bugs and not-nice things fixed (zpty -n, Zle-echoing-back, etc. ) test file X04zlehighlight as in my answer where I've elaborated on zle -T and enumerated codes being removed like "^[[?2004h" (X-seq: 43811), but with: zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|" fg_end_code:"|" bg_start_code:"BCDE|" bg_end_code:"|" )' I adapted the visible-codes by readability on the terminal, example new-code and it's previous form: -0m38;2;4;8;16mtrue39m +0mCDE|8;2;4;8;16|trueCDE|9| (I've accepted this and other changes into the test, they're correct with the zle_highlight contents). And a bug appeared – every test works, instead of the two near-color tests. The reported differences: -0mCDE|8;5;232|trueCDE|9| +0m38;5;232mtrueCDE|9| (...) Was testing: basic region_highlight with near-color (hex-triplets at input) -0mCDE|8;5;40|tCDE|8;5;160|rCDE|9|CDE|8;5;40|ueCDE|9| +0m38;5;40mt38;5;160mrCDE|9|38;5;40mueCDE|9| Was testing: overlapping region_highlight with near-color (hex-triplets at input) The minus-lines are hand-crafted by me, holding the expected contents after zle_highlight was customized. -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org [-- Attachment #2: X04zlehighlight.ztst.txt --] [-- Type: text/plain, Size: 5719 bytes --] # Tests for region_highlight, true-color support, near-color support # Version 0.5 %prep if [[ $OSTYPE == cygwin ]]; then ZTST_unimplemented='the zsh/zpty module does not work on Cygwin' elif zmodload zsh/zpty 2> /dev/null; then export TERM=xterm-256color zpty_start() { export PS1= PS2= zpty -d zpty zsh "${(q)ZTST_testdir}/../Src/zsh -fiV +Z" zpty -w zsh 'zle_highlight=( fg_start_code:"CDE|" fg_end_code:"|" bg_start_code:"BCDE|" bg_end_code:"|" )' } zpty_input() { zpty ${${(M)2:#nonl}:+-n} -w zsh "$1" } zpty_enable_zle() { zpty -w zsh "tcfunc() { REPLY=""; }" zpty -w zsh "setopt zle; zle -T tc tcfunc" # This line will not be echoed back, behaving like ! -o zle } zpty_line() { setopt localoptions extendedglob noshwordsplit local REPLY cm=$'\r' integer i for (( i = 0; i < ${1:-1}; ++i )); do zpty -r zsh REPLY # P is for "preserve", induces keeping some # color codes to test region_highlight, etc. # The color codes are then made regular text [[ "$2" = "p" ]] && { print -rl -- "$REPLY" >> /tmp/output # THE LINE REPLY=${REPLY//$'\x1b'\[([2][0-9;]m|[JK]|\?[0-9]##(h|l))/} REPLY=${REPLY//(#b)$'\x1b'\[([0-9;]##m)/${match[1]}} } || { REPLY=${REPLY//$'\x1b'\[([0-9;]##m|[JK]|\?[0-9]##(h|l))/} } # Fix e^Mexit - match ((?)\r(?)), if \2 == \3, then replace with \2 # otherwise replace with \1 stripped out of leading/trailing [[:space:]] REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}} [[ -n "$REPLY" ]] && print -r -- ${${REPLY%%[[:space:]]##}##[[:space:]]##} done } zpty_stop() { setopt localoptions extendedglob local REPLY cm=$'\r' # Zle is active, can use Ctrl-D to exit zpty -n -w zsh $'\C-d' # zpty gives no output when piped without these braces (?) # The while loop with // substitution is to convert `e^Mexit' # into `exit' (see zpty_line). The sed commands remove escapes { zpty -r zsh } | sed $'/[^[:space:]]/!d; s/\r$//; s/\x1b\\[[0-9;]*m//g; s/\x1b\\[[JK]//g; s/\x1b\\[?[0-9]*[hl]//g' | while read REPLY; do REPLY=${REPLY//(#b)((?(#c0,1))$cm(?(#c0,1)))/${${${(M)match[2]:#${match[3]}}:+${match[2]}}:-${${match[1]##[[:space:]]##}%%[[:space:]]##}}}; print -rn -- "$REPLY"; done zpty -d : } else ZTST_unimplemented='the zsh/zpty module is not available' fi %test zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with 8 colors >0mCDE|2|trueCDE|9| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with true-color (hex-triplets) >0mCDE|8;2;4;8;16|trueCDE|9| zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#040810" ); }' zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:basic region_highlight with near-color (hex-triplets at input) >0mCDE|8;5;232|trueCDE|9| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=green" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=red" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with 8 colors >0mCDE|2|tCDE|1|rCDE|9|CDE|2|ueCDE|9| zpty_start zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with true-color >0mCDE|8;2;0;204;0|tCDE|8;2;204;0;0|rCDE|9|CDE|8;2;0;204;0|ueCDE|9| zpty_start zpty_input 'zmodload zsh/nearcolor' zpty_input 'rh_widget() { BUFFER="true"; region_highlight+=( "0 4 fg=#00cc00" ); rh2; }' zpty_input 'rh2() { region_highlight+=( "1 2 fg=#cc0000" ); }' # `r' in red; the above line would be too long zpty_input 'zle -N rh_widget' zpty_input 'bindkey "\C-a" rh_widget' zpty_enable_zle zpty_input $'\C-a' # emits newline, which executes BUFFER="true" command zpty_line 1 p # the line of interest, preserving escapes ("p") zpty_stop 0:overlapping region_highlight with near-color (hex-triplets at input) >0mCDE|8;5;40|tCDE|8;5;160|rCDE|9|CDE|8;5;40|ueCDE|9| %clean zmodload -ui zsh/zpty # vim:ft=zsh ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: X04 zle highlight tests, near-color bug 2018-11-11 5:38 ` X04 zle highlight tests, near-color bug Sebastian Gniazdowski @ 2018-11-18 15:34 ` Sebastian Gniazdowski 0 siblings, 0 replies; 22+ messages in thread From: Sebastian Gniazdowski @ 2018-11-18 15:34 UTC (permalink / raw) To: Oliver Kiddle; +Cc: Zsh hackers list [-- Attachment #1: Type: text/plain, Size: 187 bytes --] Had someone maybe ran the X04 test? The one attached in the topic email should fail at 3 or 4th test like it is described. The no-zsh_higlight version in workers/43811 should fully pass. ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2018-12-11 8:06 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-11-07 13:19 [BUG?] If true-color is used, overlapping colors do not work Sebastian Gniazdowski 2018-11-07 14:40 ` Sebastian Gniazdowski 2018-11-07 19:19 ` Sebastian Gniazdowski 2018-11-08 3:48 ` Oliver Kiddle 2018-11-08 9:25 ` Sebastian Gniazdowski 2018-11-08 3:03 ` Oliver Kiddle 2018-11-08 9:19 ` Sebastian Gniazdowski 2018-11-09 1:28 ` Oliver Kiddle 2018-11-09 15:39 ` Sebastian Gniazdowski 2018-11-11 0:43 ` Sebastian Gniazdowski 2018-11-11 5:11 ` Sebastian Gniazdowski 2018-11-24 17:32 ` highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) Oliver Kiddle 2018-11-30 0:34 ` Sebastian Gniazdowski 2018-12-07 1:55 ` [BUG] General 256 colors bug – zle_highlight / fg_start_code, etc. is not respected (was: highlight test cases) Sebastian Gniazdowski 2018-12-07 20:26 ` Sebastian Gniazdowski 2018-12-09 19:13 ` Daniel Shahaf 2018-12-11 8:06 ` Sebastian Gniazdowski 2018-12-10 2:54 ` highlight test cases (was Re: [BUG?] If true-color is used, overlapping colors do not work) Oliver Kiddle 2018-12-10 23:51 ` Sebastian Gniazdowski 2018-12-10 23:54 ` Sebastian Gniazdowski 2018-11-11 5:38 ` X04 zle highlight tests, near-color bug Sebastian Gniazdowski 2018-11-18 15:34 ` Sebastian Gniazdowski
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).