Re: Terminal theme tool – a workaround for lack of 24-bit color in Zsh?

Re: Terminal theme tool – a workaround for lack of 24-bit color in Zsh?

[Oliver Kiddle]

On 15 Oct, Sebastian Gniazdowski wrote:
> There are 2 escape sequences, ESC]4;n;#rrggbb\a and ESC]Pnrrggbb, that
> allow to change terminal's current palette. First works on most

> I'm curious on opinions whether this really is a workaround for no
> true-color support in Zsh and maybe on ideas of what such
> terminal-theme manager should do.

That does cause the colours further back in the scrollback buffer
to change. I find 256 colours to be plenty for my needs but it would
be nice to be able to specify colours consistently such as with hex
triples. I still need to deal with 88-colour terminals and that
makes it harder to setup consistent themes. For themes, agreeing
on a particular associative array to map tokens to colours may work
well enough. Perhaps with explicit support in zsh where colours are
specified.

The patch below takes a different approach to this. Colours specified as
hex triples are matched to the nearest colour in the usual 256-colour
palette.

The patch adds a module with a hook function for retrieving the colour.
Actual true colour support wouldn't fit well in a module as it would
mostly affect the existing attribute tracking code. But this module can
act as a substitute while allowing a consistent way to specify colours.

True colour would require upping the attributes to 64-bits to make
space for 24-bits for the colour. To do that, the same questions I
posed for PM_ flags would apply. Can we just use uint64_t or is
autoconf magic required? A second int is also possible but somewhat
ugly because it means an extra parameter to some functions.

An alternative that keeps the attributes within 32-bits would be to
keep our own palette of 256 custom colours. Sebastian's approach of
redefining the terminal's palette is also an option.

The nearest colour matching code converts RGB colours to CIE L*a*b*.
There are more sophisticated methods but this does a noticably better
job than just taking distances in RGB space. It doesn't use any of the
first 16 colours, those are regularly configured and can't be relied
upon. While it is possible that someone has the whole range configured,
I don't think that's very common and, if really needed, there is an
escape sequence to get the colour values. It does compare directly
against all possible colours. I experimented with trying to optimise
it to avoid that but comparing 240 appears instant, even on my
decade old PC, and alternatives just ended up with longer code.

I'd be interested if anyone has any thoughts or views on any of this?

Oliver

PS. Thanks to Sebastian for testing out an earlier version of the
colour matching code.

diff --git a/Src/Modules/nearcolor.c b/Src/Modules/nearcolor.c
new file mode 100644
index 000000000..2a763d470
--- /dev/null
+++ b/Src/Modules/nearcolor.c
@@ -0,0 +1,178 @@
+#include "nearcolor.mdh"
+#include "nearcolor.pro"
+
+#include <math.h>
+
+struct cielab {
+    float L, a, b;
+};
+typedef struct cielab *Cielab;
+
+static float
+deltae(Cielab lab1, Cielab lab2)
+{
+    /* taking square root unnecessary as we're just comparing values */
+    return powf(lab1->L - lab2->L, 2) +
+	powf(lab1->a - lab2->a, 2) +
+	powf(lab1->b - lab2->b, 2);
+}
+
+static void
+RGBtoLAB(int red, int green, int blue, Cielab lab)
+{
+    float R = (float)red / 255.0;
+    float G = (float)green / 255.0;
+    float B = (float)blue / 255.0;
+    R = 100.0 * (R > 0.04045 ? powf((R + 0.055) / 1.055, 2.4) : R / 12.92);
+    G = 100.0 * (G > 0.04045 ? powf((G + 0.055) / 1.055, 2.4) : G / 12.92);
+    B = 100.0 * (B > 0.04045 ? powf((B + 0.055) / 1.055, 2.4) : B / 12.92);
+
+    /* Observer. = 2 degrees, Illuminant = D65 */
+    float X = (R * 0.4124 + G * 0.3576 + B * 0.1805) / 95.047;
+    float Y = (R * 0.2126 + G * 0.7152 + B * 0.0722) / 100.0;
+    float Z = (R * 0.0193 + G * 0.1192 + B * 0.9505) / 108.883;
+
+    X = (X > 0.008856) ? powf(X, 1.0/3.0) : (7.787 * X) + (16.0 / 116.0);
+    Y = (Y > 0.008856) ? powf(Y, 1.0/3.0) : (7.787 * Y) + (16.0 / 116.0);
+    Z = (Z > 0.008856) ? powf(Z, 1.0/3.0) : (7.787 * Z) + (16.0 / 116.0);
+
+    lab->L = (116.0 * Y) - 16.0;
+    lab->a = 500.0 * (X - Y);
+    lab->b = 200.0 * (Y - Z);
+}
+
+static int
+mapRGBto88(int red, int green, int blue)
+{
+    int component[] = { 0, 0x8b, 0xcd, 0xff, 0x2e, 0x5c, 0x8b, 0xa2, 0xb9, 0xd0, 0xe7 };
+    struct cielab orig, next;
+    float nextl, bestl = -1;
+    int r, g, b;
+    int comp_r = 0, comp_g = 0, comp_b = 0;
+
+    /* Get original value */
+    RGBtoLAB(red, green, blue, &orig);
+
+    /* try every one of the 72 colours */
+    for (r = 0; r < 11; r++) {
+	for (g = 0; g <= 3; g++) {
+	    for (b = 0; b <= 3; b++) {
+		if (r > 3) g = b = r; /* advance inner loops to the block of greys */
+		RGBtoLAB(component[r], component[g], component[b], &next);
+		nextl = deltae(&orig, &next);
+		if (nextl < bestl || bestl < 0) {
+		    bestl = nextl;
+		    comp_r = r;
+		    comp_g = g;
+		    comp_b = b;
+		}
+	    }
+	}
+    }
+
+    return (comp_r > 3) ? 77 + comp_r :
+        16 + (comp_r * 16) + (comp_g * 4) + comp_b;
+}
+
+/*
+ * Convert RGB to nearest colour in the 256 colour range
+ */
+static int
+mapRGBto256(int red, int green, int blue)
+{
+    int component[] = {
+	0, 0x5f, 0x87, 0xaf, 0xd7, 0xff,
+	0x8, 0x12, 0x1c, 0x26, 0x30, 0x3a, 0x44, 0x4e,
+	0x58, 0x62, 0x6c, 0x76, 0x80, 0x8a, 0x94, 0x9e,
+	0xa8, 0xb2, 0xbc, 0xc6, 0xd0, 0xda, 0xe4, 0xee
+    };
+    struct cielab orig, next;
+    float nextl, bestl = -1;
+    int r, g, b;
+    int comp_r = 0, comp_g = 0, comp_b = 0;
+
+    /* Get original value */
+    RGBtoLAB(red, green, blue, &orig);
+
+    for (r = 0; r < sizeof(component)/sizeof(*component); r++) {
+	for (g = 0; g <= 5; g++) {
+	    for (b = 0; b <= 5; b++) {
+		if (r > 5) g = b = r; /* advance inner loops to the block of greys */
+		RGBtoLAB(component[r], component[g], component[b], &next);
+		nextl = deltae(&orig, &next);
+		if (nextl < bestl || bestl < 0) {
+		    bestl = nextl;
+		    comp_r = r;
+		    comp_g = g;
+		    comp_b = b;
+		}
+	    }
+	}
+    }
+
+    return (comp_r > 5) ? 226 + comp_r :
+	16 + (comp_r * 36) + (comp_g * 6) + comp_b;
+}
+
+static int
+getnearestcolor(UNUSED(Hookdef dummy), Color_rgb col)
+{
+    if (tccolours == 256)
+	return mapRGBto256(col->red, col->green, col->blue);
+    if (tccolours == 88)
+	return mapRGBto88(col->red, col->green, col->blue);
+    return 0;
+}
+
+static struct features module_features = {
+    NULL, 0,
+    NULL, 0,
+    NULL, 0,
+    NULL, 0,
+    0
+};
+
+/**/
+int
+setup_(UNUSED(Module m))
+{
+    return 0;
+}
+
+/**/
+int
+features_(Module m, char ***features)
+{
+    *features = featuresarray(m, &module_features);
+    return 0;
+}
+
+/**/
+int
+enables_(Module m, int **enables)
+{
+    return handlefeatures(m, &module_features, enables);
+}
+
+/**/
+int
+boot_(Module m)
+{
+    addhookfunc("get_color_attr", (Hookfn) getnearestcolor);
+    return 0;
+}
+
+/**/
+int
+cleanup_(Module m)
+{
+    deletehookfunc("get_color_attr", (Hookfn) getnearestcolor);
+    return setfeatureenables(m, &module_features, NULL);
+}
+
+/**/
+int
+finish_(UNUSED(Module m))
+{
+    return 0;
+}
diff --git a/Src/Modules/nearcolor.mdd b/Src/Modules/nearcolor.mdd
new file mode 100644
index 000000000..2fcdaf04e
--- /dev/null
+++ b/Src/Modules/nearcolor.mdd
@@ -0,0 +1,5 @@
+name=zsh/nearcolor
+link=dynamic
+load=no
+
+objects="nearcolor.o"
diff --git a/Src/init.c b/Src/init.c
index cec914329..e7e62e2f7 100644
--- a/Src/init.c
+++ b/Src/init.c
@@ -94,6 +94,7 @@ mod_export struct hookdef zshhooks[] = {
     HOOKDEF("exit", NULL, HOOKF_ALL),
     HOOKDEF("before_trap", NULL, HOOKF_ALL),
     HOOKDEF("after_trap", NULL, HOOKF_ALL),
+    HOOKDEF("get_color_attr", NULL, HOOKF_ALL),
 };
 
 /* keep executing lists until EOF found */
diff --git a/Src/prompt.c b/Src/prompt.c
index 959ed8e3d..39edbdb2b 100644
--- a/Src/prompt.c
+++ b/Src/prompt.c
@@ -1621,7 +1621,24 @@ match_colour(const char **teststrp, int is_fg, int colour)
     int shft, on, named = 0, tc;
 
     if (teststrp) {
-	if ((named = ialpha(**teststrp))) {
+	if (**teststrp == '#' && isxdigit((*teststrp)[1])) {
+	    struct color_rgb color;
+	    char *end;
+	    zlong col = zstrtol(*teststrp+1, &end, 16);
+            if (end - *teststrp == 4) {
+		color.red = col >> 8 | ((col >> 8) << 4);
+		color.green = (col & 0xf0) >> 4;
+		color.green |= color.green << 4;
+		color.blue = col & 0xf;
+		color.blue |= color.blue << 4;
+	    } else if (end - *teststrp == 7) {
+		color.red = col >> 16;
+		color.green = (col & 0xff00) >> 8;
+		color.blue = col & 0xff;
+	    }
+	    *teststrp = end;
+	    colour = runhookdef(GETCOLORATTR, &color);
+	} else if ((named = ialpha(**teststrp))) {
 	    colour = match_named_colour(teststrp);
 	    if (colour == 8) {
 		/* default */
diff --git a/Src/zsh.h b/Src/zsh.h
index 894158818..68731e226 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -2707,6 +2707,12 @@ struct ttyinfo {
 #define COL_SEQ_BG	(1)
 #define COL_SEQ_COUNT	(2)
 
+struct color_rgb {
+    unsigned int red, green, blue;
+};
+
+typedef struct color_rgb *Color_rgb;
+
 /*
  * Flags to testcap() and set_colour_attribute (which currently only
  * handles TSC_PROMPT).
@@ -3203,6 +3209,7 @@ enum {
 #define EXITHOOK       (zshhooks + 0)
 #define BEFORETRAPHOOK (zshhooks + 1)
 #define AFTERTRAPHOOK  (zshhooks + 2)
+#define GETCOLORATTR   (zshhooks + 3)
 
 #ifdef MULTIBYTE_SUPPORT
 /* Final argument to mb_niceformat() */

Re: Terminal theme tool – a workaround for lack of 24-bit color in Zsh?

[dana]

On 30 Oct 2018, at 18:41, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>I'd be interested if anyone has any thoughts or views on any of this?

I don't have a need for the module (16 colours is usually sufficient for me),
but it sounds like a neat work-around for people who want their colour stuff to
be portable across different terminals

About uint64_t, i think if you use inttypes.h instead of stdint.h directly, it
should be very portable (at least, i assume nobody cares about running zsh on
weird embedded systems or enterprise UNIXes from the '90s). That's just my
impression based on reading, though; AFAIK i've never actually had to worry
about it myself

dana

PATCH: true colour support

[Oliver Kiddle]

This is a followup to the patch in 43747 to add support for generating
the escape sequences for terminals that support true colour. This works
with prompt escapes and zle_highlight/region_highlight but I'm not sure
about ZLS_COLOURS/list-colors because they use a different format.

I don't think there's any sense in trying to detect support for
true colour. termcap/terminfo can't help us and it'll be more
reliable to leave it to users to craft whatever heuristics work for
those terminals that they use. No true colour escapes will ever be
generated if you don't specify colours as hex triples anyway. And
even if they are, they should be ignored. For what it's worth,
checking for *(truecolor|24bit)* in $COLORTERM is a reasonable test
with the nearcolor module from the first patch acting as a fallback.
And, you might want to add COLORTERM in AcceptEnv/SendEnv for ssh
if it isn't there already.

The actual escape sequences appear to be quite standard and are, for
now, hard coded. We should probably support some zle_highlight fields
similar to fg_start_code to allow them to be configured but I'm unsure
of what exact form that should take.

autoconf seemingly already checks for stdint.h and we already had some
autoconf tests to check for an extra long integer type so the solution
I've applied for that looks like:

  #ifdef HAVE_STDINT_H
    typedef uint64_t zattr;
  #else
    typedef zulong zattr;
  #endif

Having the typedef makes the intent of applicable variables clearer and
makes it easier if we ever want to change it again.

Every last one of the 64-bits was needed. We previously used a signed
integer with -1 to indicate errors but I've made the error condition an
explicit flag - TXT_ERROR. Having the bytes for each of red/green/blue
for fg and bg aligned to byte boundaries allows the compiler to generate
more efficient code for extracting them. Setting TXTNO{F,B}GCOLOUR could
serve the same purpose as TXT_ERROR but for the case of specifying
"default" as the colour. And, if we ever need a bit or two, we can
probably find a way to squeeze a couple out.

Oliver

diff --git a/Src/Modules/nearcolor.c b/Src/Modules/nearcolor.c
index 2a763d470..128658e20 100644
--- a/Src/Modules/nearcolor.c
+++ b/Src/Modules/nearcolor.c
@@ -118,10 +118,12 @@ static int
 getnearestcolor(UNUSED(Hookdef dummy), Color_rgb col)
 {
     if (tccolours == 256)
-	return mapRGBto256(col->red, col->green, col->blue);
+	return mapRGBto256(col->red, col->green, col->blue) + 1;
     if (tccolours == 88)
-	return mapRGBto88(col->red, col->green, col->blue);
-    return 0;
+	return mapRGBto88(col->red, col->green, col->blue) + 1;
+    /* returning 1 indicates black rather than failure (0) so this
+     * module still serves to prevent fallback on true color */
+    return 1;
 }
 
 static struct features module_features = {
diff --git a/Src/Zle/complist.c b/Src/Zle/complist.c
index e768aee5d..429c8159f 100644
--- a/Src/Zle/complist.c
+++ b/Src/Zle/complist.c
@@ -1096,6 +1096,7 @@ compprintfmt(char *fmt, int n, int dopr, int doesc, int ml, int *stop)
 	    p += len;
 	    if (*p) {
 		int arg = 0, is_fg;
+		zattr atr;
 
 		if (idigit(*p))
 		    arg = zstrtol(p, &p, 10);
@@ -1159,13 +1160,13 @@ compprintfmt(char *fmt, int n, int dopr, int doesc, int ml, int *stop)
 		    /* colours must be ASCII */
 		    if (*p == '{') {
 			p++;
-			arg = match_colour((const char **)&p, is_fg, 0);
+			atr = match_colour((const char **)&p, is_fg, 0);
 			if (*p == '}')
 			    p++;
 		    } else
-			arg = match_colour(NULL, is_fg, arg);
-		    if (arg >= 0 && dopr)
-			set_colour_attribute(arg, is_fg ? COL_SEQ_FG :
+			atr = match_colour(NULL, is_fg, arg);
+		    if (atr != TXT_ERROR && dopr)
+			set_colour_attribute(atr, is_fg ? COL_SEQ_FG :
 					     COL_SEQ_BG, 0);
 		    break;
 		case ZWC('f'):
diff --git a/Src/Zle/zle.h b/Src/Zle/zle.h
index 8261da92b..f06c56483 100644
--- a/Src/Zle/zle.h
+++ b/Src/Zle/zle.h
@@ -430,7 +430,7 @@ enum {
  */
 struct region_highlight {
     /* Attributes turned on in the region */
-    int atr;
+    zattr atr;
     /* Start of the region */
     int start;
     /* Start of the region in metafied ZLE line */
@@ -488,7 +488,7 @@ typedef struct {
      * need the effect; 'off' attributes are only present for the
      * last character in the sequence.
      */
-    int atr;
+    zattr atr;
 } REFRESH_ELEMENT;
 
 /* A string of screen cells */
diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index d0dd1ef06..1f293845f 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -149,7 +149,7 @@ char *lpromptbuf, *rpromptbuf;
 /* Text attributes after displaying prompts */
 
 /**/
-unsigned pmpt_attr, rpmpt_attr;
+zattr pmpt_attr, rpmpt_attr;
 
 /* number of lines displayed */
 
@@ -208,7 +208,7 @@ int predisplaylen, postdisplaylen;
  * displayed on screen.
  */
 
-static int default_atr_on, special_atr_on;
+static zattr default_atr_on, special_atr_on;
 
 /*
  * Array of region highlights, no special termination.
@@ -521,7 +521,7 @@ unset_region_highlight(Param pm, int exp)
 
 
 /* The last attributes that were on. */
-static int lastatr;
+static zattr lastatr;
 
 /*
  * Clear the last attributes that we set:  used when we're going
@@ -560,7 +560,7 @@ tcoutclear(int cap)
 
 /**/
 void
-zwcputc(const REFRESH_ELEMENT *c, int *curatrp)
+zwcputc(const REFRESH_ELEMENT *c, zattr *curatrp)
 {
     /*
      * Safety: turn attributes off if last heard of turned on.
@@ -638,7 +638,7 @@ static int
 zwcwrite(const REFRESH_STRING s, size_t i)
 {
     size_t j;
-    int curatr = 0;
+    zattr curatr = 0;
 
     for (j = 0; j < i; j++)
 	zwcputc(s + j, &curatr);
@@ -891,7 +891,7 @@ snextline(Rparams rpms)
 
 /**/
 static void
-settextattributes(int atr)
+settextattributes(zattr atr)
 {
     if (txtchangeisset(atr, TXTNOBOLDFACE))
 	tsetcap(TCALLATTRSOFF, 0);
@@ -992,7 +992,7 @@ zrefresh(void)
     int tmppos;			/* t - tmpline				     */
     int tmpalloced;		/* flag to free tmpline when finished	     */
     int remetafy;		/* flag that zle line is metafied	     */
-    int txtchange;		/* attributes set after prompts              */
+    zattr txtchange;		/* attributes set after prompts              */
     int rprompt_off = 1;	/* Offset of rprompt from right of screen    */
     struct rparams rpms;
 #ifdef MULTIBYTE_SUPPORT
@@ -1212,8 +1212,9 @@ zrefresh(void)
     rpms.s = nbuf[rpms.ln = 0] + lpromptw;
     rpms.sen = *nbuf + winw;
     for (t = tmpline, tmppos = 0; tmppos < tmpll; t++, tmppos++) {
-	int base_atr_on = default_atr_on, base_atr_off = 0, ireg;
-	int all_atr_on, all_atr_off;
+	unsigned ireg;
+	zattr base_atr_on = default_atr_on, base_atr_off = 0;
+	zattr all_atr_on, all_atr_off;
 	struct region_highlight *rhp;
 	/*
 	 * Calculate attribute based on region.
@@ -1446,7 +1447,8 @@ zrefresh(void)
 	more_end = 1;
 
     if (statusline) {
-	int outll, outsz, all_atr_on, all_atr_off;
+	int outll, outsz;
+	zattr all_atr_on, all_atr_off;
 	char *statusdup = ztrdup(statusline);
 	ZLE_STRING_T outputline =
 	    stringaszleline(statusdup, 0, &outll, &outsz, NULL); 
@@ -1672,7 +1674,7 @@ zrefresh(void)
 
     /* output the right-prompt if appropriate */
 	if (put_rpmpt && !iln && !oput_rpmpt) {
-	    int attrchange;
+	    zattr attrchange;
 
 	    moveto(0, winw - rprompt_off - rpromptw);
 	    zputs(rpromptbuf, shout);
@@ -1926,7 +1928,7 @@ refreshline(int ln)
 /* 3: main display loop - write out the buffer using whatever tricks we can */
 
     for (;;) {
-	int now_off;
+	zattr now_off;
 
 #ifdef MULTIBYTE_SUPPORT
 	if ((!nl->chr || nl->chr != WEOF) && (!ol->chr || ol->chr != WEOF)) {
@@ -2506,8 +2508,9 @@ singlerefresh(ZLE_STRING_T tmpline, int tmpll, int tmpcs)
     *vp = zr_zr;
 
     for (t0 = 0; t0 < tmpll; t0++) {
-	int base_atr_on = 0, base_atr_off = 0, ireg;
-	int all_atr_on, all_atr_off;
+	unsigned ireg;
+	zattr base_atr_on = 0, base_atr_off = 0;
+	zattr all_atr_on, all_atr_off;
 	struct region_highlight *rhp;
 	/*
 	 * Calculate attribute based on region.
diff --git a/Src/Zle/zle_tricky.c b/Src/Zle/zle_tricky.c
index 22c381237..8f71d551d 100644
--- a/Src/Zle/zle_tricky.c
+++ b/Src/Zle/zle_tricky.c
@@ -2431,6 +2431,7 @@ printfmt(char *fmt, int n, int dopr, int doesc)
 	/* Handle the `%' stuff (%% == %, %n == <number of matches>). */
 	if (doesc && *p == '%') {
 	    int arg = 0, is_fg;
+	    zattr atr;
 	    if (idigit(*++p))
 		arg = zstrtol(p, &p, 10);
 	    if (*p) {
@@ -2482,13 +2483,13 @@ printfmt(char *fmt, int n, int dopr, int doesc)
 		    is_fg = (*p == 'F');
 		    if (p[1] == '{') {
 			p += 2;
-			arg = match_colour((const char **)&p, is_fg, 0);
+			atr = match_colour((const char **)&p, is_fg, 0);
 			if (*p != '}')
 			    p--;
 		    } else
-			arg = match_colour(NULL, is_fg, arg);
-		    if (arg >= 0)
-			set_colour_attribute(arg, is_fg ? COL_SEQ_FG :
+			atr = match_colour(NULL, is_fg, arg);
+		    if (atr >= 0)
+			set_colour_attribute(atr, is_fg ? COL_SEQ_FG :
 					     COL_SEQ_BG, 0);
 		    break;
 		case 'f':
diff --git a/Src/prompt.c b/Src/prompt.c
index 39edbdb2b..284c02475 100644
--- a/Src/prompt.c
+++ b/Src/prompt.c
@@ -33,7 +33,7 @@
 /* text attribute mask */
 
 /**/
-mod_export unsigned txtattrmask;
+mod_export zattr txtattrmask;
 
 /* the command stack for use with %_ in prompts */
 
@@ -168,7 +168,7 @@ promptpath(char *p, int npath, int tilde)
 
 /**/
 mod_export char *
-promptexpand(char *s, int ns, char *rs, char *Rs, unsigned int *txtchangep)
+promptexpand(char *s, int ns, char *rs, char *Rs, zattr *txtchangep)
 {
     struct buf_vars new_vars;
 
@@ -236,8 +236,8 @@ promptexpand(char *s, int ns, char *rs, char *Rs, unsigned int *txtchangep)
 }
 
 /* Parse the argument for %F and %K */
-static int
-parsecolorchar(int arg, int is_fg)
+static zattr
+parsecolorchar(zattr arg, int is_fg)
 {
     if (bv->fm[1] == '{') {
 	char *ep;
@@ -268,10 +268,11 @@ parsecolorchar(int arg, int is_fg)
 
 /**/
 static int
-putpromptchar(int doprint, int endchar, unsigned int *txtchangep)
+putpromptchar(int doprint, int endchar, zattr *txtchangep)
 {
     char *ss, *hostnam;
     int t0, arg, test, sep, j, numjobs, len;
+    zattr atr;
     struct tm *tm;
     struct timespec ts;
     time_t timet;
@@ -538,13 +539,13 @@ putpromptchar(int doprint, int endchar, unsigned int *txtchangep)
 		tsetcap(TCUNDERLINEEND, TSC_PROMPT|TSC_DIRTY);
 		break;
 	    case 'F':
-		arg = parsecolorchar(arg, 1);
-		if (arg >= 0 && !(arg & TXTNOFGCOLOUR)) {
-		    txtchangeset(txtchangep, arg & TXT_ATTR_FG_ON_MASK,
+		atr = parsecolorchar(arg, 1);
+		if (!(atr & (TXT_ERROR | TXTNOFGCOLOUR))) {
+		    txtchangeset(txtchangep, atr & TXT_ATTR_FG_ON_MASK,
 				 TXTNOFGCOLOUR | TXT_ATTR_FG_COL_MASK);
 		    txtunset(TXT_ATTR_FG_COL_MASK);
-		    txtset(arg & TXT_ATTR_FG_ON_MASK);
-		    set_colour_attribute(arg, COL_SEQ_FG, TSC_PROMPT);
+		    txtset(atr & TXT_ATTR_FG_ON_MASK);
+		    set_colour_attribute(atr, COL_SEQ_FG, TSC_PROMPT);
 		    break;
 		}
 		/* else FALLTHROUGH */
@@ -554,13 +555,13 @@ putpromptchar(int doprint, int endchar, unsigned int *txtchangep)
 		set_colour_attribute(TXTNOFGCOLOUR, COL_SEQ_FG, TSC_PROMPT);
 		break;
 	    case 'K':
-		arg = parsecolorchar(arg, 0);
-		if (arg >= 0 && !(arg & TXTNOBGCOLOUR)) {
-		    txtchangeset(txtchangep, arg & TXT_ATTR_BG_ON_MASK,
+		atr = parsecolorchar(arg, 0);
+		if (!(atr & (TXT_ERROR | TXTNOBGCOLOUR))) {
+		    txtchangeset(txtchangep, atr & TXT_ATTR_BG_ON_MASK,
 				 TXTNOBGCOLOUR | TXT_ATTR_BG_COL_MASK);
 		    txtunset(TXT_ATTR_BG_COL_MASK);
-		    txtset(arg & TXT_ATTR_BG_ON_MASK);
-		    set_colour_attribute(arg, COL_SEQ_BG, TSC_PROMPT);
+		    txtset(atr & TXT_ATTR_BG_ON_MASK);
+		    set_colour_attribute(atr, COL_SEQ_BG, TSC_PROMPT);
 		    break;
 		}
 		/* else FALLTHROUGH */
@@ -1185,7 +1186,7 @@ countprompt(char *str, int *wp, int *hp, int overf)
 /**/
 static int
 prompttrunc(int arg, int truncchar, int doprint, int endchar,
-	    unsigned int *txtchangep)
+	    zattr *txtchangep)
 {
     if (arg > 0) {
 	char ch = *bv->fm, *ptr, *truncstr;
@@ -1567,8 +1568,8 @@ static const char *ansi_colours[] = {
 /* Defines the available types of highlighting */
 struct highlight {
     const char *name;
-    int mask_on;
-    int mask_off;
+    zattr mask_on;
+    zattr mask_off;
 };
 
 static const struct highlight highlights[] = {
@@ -1615,11 +1616,21 @@ match_named_colour(const char **teststrp)
  */
 
 /**/
-mod_export int
+mod_export zattr
 match_colour(const char **teststrp, int is_fg, int colour)
 {
-    int shft, on, named = 0, tc;
+    int shft, named = 0, tc;
+    zattr on;
 
+    if (is_fg) {
+	shft = TXT_ATTR_FG_COL_SHIFT;
+	on = TXTFGCOLOUR;
+	tc = TCFGCOLOUR;
+    } else {
+	shft = TXT_ATTR_BG_COL_SHIFT;
+	on = TXTBGCOLOUR;
+	tc = TCBGCOLOUR;
+    }
     if (teststrp) {
 	if (**teststrp == '#' && isxdigit((*teststrp)[1])) {
 	    struct color_rgb color;
@@ -1637,7 +1648,12 @@ match_colour(const char **teststrp, int is_fg, int colour)
 		color.blue = col & 0xff;
 	    }
 	    *teststrp = end;
-	    colour = runhookdef(GETCOLORATTR, &color);
+	    colour = runhookdef(GETCOLORATTR, &color) - 1;
+	    if (colour < 0) { /* no hook function added, try true color (24-bit) */
+		colour = (((color.red << 8) + color.green) << 8) + color.blue;
+		return on | (is_fg ? TXT_ATTR_FG_24BIT : TXT_ATTR_BG_24BIT) |
+			(zattr)colour << shft;
+	    }
 	} else if ((named = ialpha(**teststrp))) {
 	    colour = match_named_colour(teststrp);
 	    if (colour == 8) {
@@ -1645,22 +1661,14 @@ match_colour(const char **teststrp, int is_fg, int colour)
 		return is_fg ? TXTNOFGCOLOUR : TXTNOBGCOLOUR;
 	    }
 	}
-	else
+	else {
 	    colour = (int)zstrtol(*teststrp, (char **)teststrp, 10);
-    }
-    if (colour < 0 || colour >= 256)
-	return -1;
-    if (is_fg) {
-	shft = TXT_ATTR_FG_COL_SHIFT;
-	on = TXTFGCOLOUR;
-	tc = TCFGCOLOUR;
-    } else {
-	shft = TXT_ATTR_BG_COL_SHIFT;
-	on = TXTBGCOLOUR;
-	tc = TCBGCOLOUR;
+	    if (colour < 0 || colour >= 256)
+		return TXT_ERROR;
+	}
     }
     /*
-     * Try termcap for numbered characters if posible.
+     * Try termcap for numbered characters if possible.
      * Don't for named characters, since our best bet
      * of getting the names right is with ANSI sequences.
      */
@@ -1671,7 +1679,7 @@ match_colour(const char **teststrp, int is_fg, int colour)
 	     * Can we assume ANSI colours work?
 	     */
 	    if (colour > 7)
-		return -1; /* No. */
+		return TXT_ERROR; /* No. */
 	} else {
 	    /*
 	     * We can handle termcap colours and the number
@@ -1681,7 +1689,7 @@ match_colour(const char **teststrp, int is_fg, int colour)
 		TXT_ATTR_BG_TERMCAP;
 	}
     }
-    return on | (colour << shft);
+    return on | (zattr)colour << shft;
 }
 
 /*
@@ -1691,7 +1699,7 @@ match_colour(const char **teststrp, int is_fg, int colour)
 
 /**/
 mod_export void
-match_highlight(const char *teststr, int *on_var)
+match_highlight(const char *teststr, zattr *on_var)
 {
     int found = 1;
 
@@ -1701,7 +1709,8 @@ match_highlight(const char *teststr, int *on_var)
 
 	found = 0;
 	if (strpfx("fg=", teststr) || strpfx("bg=", teststr)) {
-	    int is_fg = (teststr[0] == 'f'), atr;
+	    int is_fg = (teststr[0] == 'f');
+	    zattr atr;
 
 	    teststr += 3;
 	    atr = match_colour(&teststr, is_fg, 0);
@@ -1711,7 +1720,7 @@ match_highlight(const char *teststr, int *on_var)
 		break;
 	    found = 1;
 	    /* skip out of range colours but keep scanning attributes */
-	    if (atr >= 0)
+	    if (atr != TXT_ERROR)
 		*on_var |= atr;
 	} else {
 	    for (hl = highlights; hl->name; hl++) {
@@ -1776,7 +1785,7 @@ output_colour(int colour, int fg_bg, int use_tc, char *buf)
 
 /**/
 mod_export int
-output_highlight(int atr, char *buf)
+output_highlight(zattr atr, char *buf)
 {
     const struct highlight *hp;
     int atrlen = 0, len;
@@ -1939,7 +1948,8 @@ allocate_colour_buffer(void)
 	strlen(fg_bg_sequences[COL_SEQ_BG].end);
 
     len = lenfg > lenbg ? lenfg : lenbg;
-    colseq_buf = (char *)zalloc(len+1);
+    /* add 1 for the null and 14 for truecolor */
+    colseq_buf = (char *)zalloc(len+15);
 }
 
 /* Free the colour buffer previously allocated. */
@@ -1970,21 +1980,23 @@ free_colour_buffer(void)
 
 /**/
 mod_export void
-set_colour_attribute(int atr, int fg_bg, int flags)
+set_colour_attribute(zattr atr, int fg_bg, int flags)
 {
     char *ptr;
     int do_free, is_prompt = (flags & TSC_PROMPT) ? 1 : 0;
-    int colour, tc, def, use_termcap;
+    int colour, tc, def, use_termcap, use_truecolor;
 
     if (fg_bg == COL_SEQ_FG) {
 	colour = txtchangeget(atr, TXT_ATTR_FG_COL);
 	tc = TCFGCOLOUR;
 	def = txtchangeisset(atr, TXTNOFGCOLOUR);
+	use_truecolor = txtchangeisset(atr, TXT_ATTR_FG_24BIT);
 	use_termcap = txtchangeisset(atr, TXT_ATTR_FG_TERMCAP);
     } else {
 	colour = txtchangeget(atr, TXT_ATTR_BG_COL);
 	tc = TCBGCOLOUR;
 	def = txtchangeisset(atr, TXTNOBGCOLOUR);
+	use_truecolor = txtchangeisset(atr, TXT_ATTR_BG_24BIT);
 	use_termcap = txtchangeisset(atr, TXT_ATTR_BG_TERMCAP);
     }
 
@@ -1992,12 +2004,13 @@ set_colour_attribute(int atr, int fg_bg, int flags)
      * If we're not restoring the default, and either have a
      * colour value that is too large for ANSI, or have been told
      * to use the termcap sequence, try to use the termcap sequence.
+     * True color is not covered by termcap.
      *
      * We have already sanitised the values we allow from the
      * highlighting variables, so much of this shouldn't be
      * necessary at this point, but we might as well be safe.
      */
-    if (!def && (colour > 7 || use_termcap)) {
+    if (!def && !use_truecolor && (colour > 7 || use_termcap)) {
 	/*
 	 * We can if it's available, and either we couldn't get
 	 * the maximum number of colours, or the colour is in range.
@@ -2041,6 +2054,9 @@ set_colour_attribute(int atr, int fg_bg, int flags)
 	strcpy(ptr, fg_bg_sequences[fg_bg].def);
 	while (*ptr)
 	    ptr++;
+    } else if (use_truecolor) {
+	ptr += sprintf(ptr, "8;2;%d;%d;%d", colour >> 16,
+		(colour >> 8) & 0xff, colour & 0xff);
     } else
 	*ptr++ = colour + '0';
     strcpy(ptr, fg_bg_sequences[fg_bg].end);
diff --git a/Src/zsh.h b/Src/zsh.h
index 68731e226..10897372b 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -2633,6 +2633,12 @@ struct ttyinfo {
  * Text attributes for displaying in ZLE
  */
 
+#ifdef HAVE_STDINT_H
+  typedef uint64_t zattr;
+#else
+  typedef zulong zattr;
+#endif
+
 #define TXTBOLDFACE   0x0001
 #define TXTSTANDOUT   0x0002
 #define TXTUNDERLINE  0x0004
@@ -2664,32 +2670,41 @@ struct ttyinfo {
  */
 #define TXT_MULTIWORD_MASK  0x0400
 
+/* used when, e.g an invalid colour is specified */
+#define TXT_ERROR 0x0800
+
 /* Mask for colour to use in foreground */
-#define TXT_ATTR_FG_COL_MASK     0x000FF000
+#define TXT_ATTR_FG_COL_MASK     0x000000FFFFFF0000
 /* Bits to shift the foreground colour */
-#define TXT_ATTR_FG_COL_SHIFT    (12)
+#define TXT_ATTR_FG_COL_SHIFT    (16)
 /* Mask for colour to use in background */
-#define TXT_ATTR_BG_COL_MASK     0x0FF00000
+#define TXT_ATTR_BG_COL_MASK     0xFFFFFF0000000000
 /* Bits to shift the background colour */
-#define TXT_ATTR_BG_COL_SHIFT    (20)
+#define TXT_ATTR_BG_COL_SHIFT    (40)
 
 /* Flag to use termcap AF sequence to set colour, if available */
-#define TXT_ATTR_FG_TERMCAP      0x10000000
+#define TXT_ATTR_FG_TERMCAP      0x1000
 /* Flag to use termcap AB sequence to set colour, if available */
-#define TXT_ATTR_BG_TERMCAP      0x20000000
+#define TXT_ATTR_BG_TERMCAP      0x2000
+
+/* Flag to indicate that foreground is a 24-bit colour */
+#define TXT_ATTR_FG_24BIT        0x4000
+/* Flag to indicate that background is a 24-bit colour */
+#define TXT_ATTR_BG_24BIT        0x8000
 
 /* Things to turn on, including values for the colour elements */
 #define TXT_ATTR_ON_VALUES_MASK	\
     (TXT_ATTR_ON_MASK|TXT_ATTR_FG_COL_MASK|TXT_ATTR_BG_COL_MASK|\
-     TXT_ATTR_FG_TERMCAP|TXT_ATTR_BG_TERMCAP)
+     TXT_ATTR_FG_TERMCAP|TXT_ATTR_BG_TERMCAP|\
+     TXT_ATTR_FG_24BIT|TXT_ATTR_BG_24BIT)
 
 /* Mask out everything to do with setting a foreground colour */
 #define TXT_ATTR_FG_ON_MASK \
-    (TXTFGCOLOUR|TXT_ATTR_FG_COL_MASK|TXT_ATTR_FG_TERMCAP)
+    (TXTFGCOLOUR|TXT_ATTR_FG_COL_MASK|TXT_ATTR_FG_TERMCAP|TXT_ATTR_FG_24BIT)
 
 /* Mask out everything to do with setting a background colour */
 #define TXT_ATTR_BG_ON_MASK \
-    (TXTBGCOLOUR|TXT_ATTR_BG_COL_MASK|TXT_ATTR_BG_TERMCAP)
+    (TXTBGCOLOUR|TXT_ATTR_BG_COL_MASK|TXT_ATTR_BG_TERMCAP|TXT_ATTR_BG_24BIT)
 
 /* Mask out everything to do with activating colours */
 #define TXT_ATTR_COLOUR_ON_MASK			\
diff --git a/Src/zsh_system.h b/Src/zsh_system.h
index 8289ee97c..e7d529b6e 100644
--- a/Src/zsh_system.h
+++ b/Src/zsh_system.h
@@ -137,6 +137,10 @@ char *alloca _((size_t));
 #include <stddef.h>
 #endif
 
+#ifdef HAVE_STDINT_H
+# include <stdint.h>
+#endif
+
 #include <stdio.h>
 #include <ctype.h>
 #include <sys/stat.h>

Re: PATCH: true colour support

[Oliver Kiddle]

The following patch adds documentation for the nearcolor module and
true color support.

By the way, the code is rather generous in terms of allowing missing
closing braces or extra characters before the closing brace in %F{…}
prompt expansions. So %F{123crap} just gives you colour 123. This is
in parsecolorchar() in prompt.c. Should this be considered a bug and
changed. This behaviour is old and has nothing todo with the patches.

Oliver

diff --git a/Doc/Makefile.in b/Doc/Makefile.in
index 71ca52195..5a6a705ff 100644
--- a/Doc/Makefile.in
+++ b/Doc/Makefile.in
@@ -63,7 +63,8 @@ Zsh/mod_compctl.yo Zsh/mod_complete.yo Zsh/mod_complist.yo \
 Zsh/mod_computil.yo Zsh/mod_curses.yo \
 Zsh/mod_datetime.yo Zsh/mod_db_gdbm.yo Zsh/mod_deltochar.yo \
 Zsh/mod_example.yo Zsh/mod_files.yo Zsh/mod_langinfo.yo \
-Zsh/mod_mapfile.yo Zsh/mod_mathfunc.yo Zsh/mod_newuser.yo \
+Zsh/mod_mapfile.yo Zsh/mod_mathfunc.yo \
+Zsh/mod_nearcolor.yo Zsh/mod_newuser.yo \
 Zsh/mod_parameter.yo Zsh/mod_pcre.yo Zsh/mod_private.yo \
 Zsh/mod_regex.yo Zsh/mod_sched.yo Zsh/mod_socket.yo \
 Zsh/mod_stat.yo  Zsh/mod_system.yo Zsh/mod_tcp.yo \
diff --git a/Doc/Zsh/mod_nearcolor.yo b/Doc/Zsh/mod_nearcolor.yo
new file mode 100644
index 000000000..9694bc105
--- /dev/null
+++ b/Doc/Zsh/mod_nearcolor.yo
@@ -0,0 +1,36 @@
+COMMENT(!MOD!zsh/nearcolor
+Map colours to the nearest colour in the available palette.
+!MOD!)
+The tt(zsh/nearcolor) module replaces colours specified as hex triplets
+with the nearest colour in the 88 or 256 colour palettes that are widely
+used by terminal emulators.  By default, 24-bit true colour escape codes
+are generated when colours are specified using hex triplets.  These are
+not supported by all terminals.  The purpose of this module is to make
+it easier to define colour preferences in a form that can work across a
+range of terminal emulators.
+
+Aside from the default colour, the ANSI standard for terminal escape
+codes provides for eight colours. The bright attribute brings this to
+sixteen. These basic colours are commonly used in terminal applications
+due to being widely supported. Expanded 88 and 256 colour palettes are
+also common and, while the first sixteen colours vary somewhat between
+terminals and configurations, these add a generally consistent and
+predictable set of colours.
+
+In order to use the tt(zsh/nearcolor) module, it only needs to be
+loaded. Thereafter, whenever a colour is specified using a hex triplet,
+it will be compared against each of the available colours and the
+closest will be selected. The first sixteen colours are never matched in
+this process due to being unpredictable.
+
+It isn't possible to reliably detect support for true colour in the
+terminal emulator. It is therefore recommended to be selective in
+loading the tt(zsh/nearcolor) module. For example, the following
+checks the tt(COLORTERM) environment variable:
+
+example([[ $COLORTERM = *LPAR()24bit|truecolor+RPAR()* ]] || \
+    zmodload zsh/nearcolor)
+
+Note that some terminals accept the true color escape codes but map
+them internally to a more limited palette in a similar manner to the
+tt(zsh/nearcolor) module.
diff --git a/Doc/Zsh/zle.yo b/Doc/Zsh/zle.yo
index 895c5c570..fe4e5bd04 100644
--- a/Doc/Zsh/zle.yo
+++ b/Doc/Zsh/zle.yo
@@ -2703,8 +2703,9 @@ this to appear with other types of highlighting; it is used to override
 a default.
 )
 item(tt(fg=)var(colour))(
-The foreground colour should be set to var(colour), a decimal integer
-or the name of one of the eight most widely-supported colours.
+The foreground colour should be set to var(colour), a decimal integer,
+the name of one of the eight most widely-supported colours or as a
+`tt(#)' followed by an RGB triplet in hexadecimal format.
 
 Not all terminals support this and, of those that do, not all provide
 facilities to test the support, hence the user should decide based on the
@@ -2721,6 +2722,15 @@ Co)'; if this succeeds, it indicates a limit on the number of colours which
 will be enforced by the line editor.  The number of colours is in any case
 limited to 256 (i.e. the range 0 to 255).
 
+Some modern terminal emulators have support for 24-bit true colour (16
+million colours). In this case, the hex triplet format can be used. This
+consists of either a three or six digit hexadecimal number describing
+the red, green and blue components of the colour. Hex triplets can also
+be used with 88 and 256 colour terminals via the tt(zsh/nearcolor)
+module (see ifzman(zmanref(zshmodules))\
+ifnzman(noderef(The zsh/nearcolor Module))\
+).
+
 Colour is also known as color.
 )
 item(tt(bg=)var(colour))(
diff --git a/NEWS b/NEWS
index f3e617b9c..0b0c0e84c 100644
--- a/NEWS
+++ b/NEWS
@@ -4,6 +4,15 @@ CHANGES FROM PREVIOUS VERSIONS OF ZSH
 
 Note also the list of incompatibilities in the README file.
 
+Changes since 5.6.2
+-------------------
+
+Support for 24-bit true color terminals has been added. Hex triplets
+can be used when specifying colours for prompts and line editor
+highlighting. On 88 and 256 colour terminals, a new zsh/nearcolor module
+allows colours specified with hex triplets to be matched against the
+nearest available colour.
+
 Changes from 5.6.1 to 5.6.2
 ---------------------------
 

Re: PATCH: true colour support

[Sebastian Gniazdowski]

On Sun, 4 Nov 2018 at 02:42, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>
> This is a followup to the patch in 43747 to add support for generating
> the escape sequences for terminals that support true colour. This works
> with prompt escapes and zle_highlight/region_highlight but I'm not sure
> about ZLS_COLOURS/list-colors because they use a different format.

I've updated a F-Sy-H theme to use the color-value for
region_highlight in form of hex-triplet, e.g.:

: ${FAST_HIGHLIGHT_STYLES[x-paragon-tccommand]:=#7fa8c5}

x-paragon-tc (-tc for truecolor) is a theme name, then goes the style
name, `command', then the true-color value of the style. I've added
debug prints to F-Sy-H, #7fa8c5 is really added to region highlight,
but the array then comes up with a value: "0 4 none". What can be the
reason, what to check for?

> Oliver
>
> diff --git a/Src/Modules/nearcolor.c b/Src/Modules/nearcolor.c

Re: PATCH: true colour support

[Sebastian Gniazdowski]

On Mon, 5 Nov 2018 at 15:56, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> Is something basic like a widget that just does the following working:
>   region_highlight=( "P4 6 bg=#7fa8c5" )
>
> Oliver

Ah, I didn't use fg=/bg=, it's probably the cause.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: PATCH: true colour support

[Sebastian Gniazdowski]

On Mon, 5 Nov 2018 at 16:48, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Mon, 5 Nov 2018 at 15:56, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> > Is something basic like a widget that just does the following working:
> >   region_highlight=( "P4 6 bg=#7fa8c5" )
> >
> > Oliver
>
> Ah, I didn't use fg=/bg=, it's probably the cause.

Yes, this was the cause, so the support now works. I've compared a
F-Sy-H theme in 256-color and TrueColor, it might be a bad choice
because it's an example of a VERY successful CIELab conversion, but
still, I was able to eliminate sea-colors:

https://asciinema.org/a/MdxxsoYr1uxYao5IRRvrXJpl5

Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: PATCH: true colour support

[Sebastian Gniazdowski]

On Sun, 4 Nov 2018 at 02:42, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> (...)
> I don't think there's any sense in trying to detect support for
> true colour. termcap/terminfo can't help us and it'll be more
> reliable to leave it to users to craft whatever heuristics work for
> those terminals that they use. No true colour escapes will ever be
> generated if you don't specify colours as hex triples anyway. And
> even if they are, they should be ignored. For what it's worth,
> checking for *(truecolor|24bit)* in $COLORTERM is a reasonable test
> with the nearcolor module from the first patch acting as a fallback.
> And, you might want to add COLORTERM in AcceptEnv/SendEnv for ssh
> if it isn't there already.

I'm grepping the patches and cannot find any snippet that decides
whether to use 24-bit escape sequences or zsh/nearcolor module. How
it's decided?

You've also written: "No true colour escapes will ever be generated if
you don't specify colours as hex triples anyway. And even if they are,
they should be ignored.".This sounds like: if true-color/near-color
was not enabled, ignore hex-triplet. But I think there is no switch
(e.g. an option) to enable true-color support?

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: PATCH: true colour support

[Oliver Kiddle]

Sebastian Gniazdowski wrote:
> I'm grepping the patches and cannot find any snippet that decides
> whether to use 24-bit escape sequences or zsh/nearcolor module. How
> it's decided?

The following section of new documentation (in mod_nearcolor.yo) is
intended to cover this:

+In order to use the tt(zsh/nearcolor) module, it only needs to be
+loaded. Thereafter, whenever a colour is specified using a hex triplet,
+it will be compared against each of the available colours and the
+closest will be selected.

On loading, zsh/nearcolor registers a hook function for GETCOLORATTR.
After parsing a hex triplet, match_colour() calls:
  colour = runhookdef(GETCOLORATTR, &color) - 1.
runhookdef() returns 0 if there is no registered handler for the hook
The following line in match_colour() is:
  if (colour < 0) { /* no hook function added, try true color (24-bit) */

> You've also written: "No true colour escapes will ever be generated if
> you don't specify colours as hex triples anyway. And even if they are,
> they should be ignored.".This sounds like: if true-color/near-color
> was not enabled, ignore hex-triplet. But I think there is no switch
> (e.g. an option) to enable true-color support?

You don't need to do anything to enable true colour. It is the
default when you specify colours as a hex triplet. With the text
you quoted, I was trying to justify this decision by pointing out
that there are no backward compatibility concerns - any existing
setup that only uses the old forms - %F{blue}, %K{123} etc - will
work exactly as before (no true color escape codes are ever generated).

But before introducing colours specified as hex triplets in their
configuration, users may want to take care to either load zsh/nearcolor
or use a terminal that supports true colour.

Feel free to suggest improvements to the documentation.

Oliver

Re: PATCH: true colour support

[Sebastian Gniazdowski]

On Tue, 6 Nov 2018 at 14:37, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> (...)
> But before introducing colours specified as hex triplets in their
> configuration, users may want to take care to either load zsh/nearcolor
> or use a terminal that supports true colour.
>
> Feel free to suggest improvements to the documentation.

Ok, I now see, true color escapes are just being emitted, then user
can test $COLORTERM and decide to intercept them (the escapes) by
loading zsh/nearcolor, and emulate true-color with CIELab onto-88/256
palette translation. It's a nice solution.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Should there be a Zsh release for true-color support to adopt earlier?

[Sebastian Gniazdowski]

Hello
Could there be a release made shortly to introduce true color support
possibly early, without possible complications when other changes
arise, and without waiting a half year for it to happen?

There's a matter of testing the patches. I'm running patched Zsh with
true-color syntax-highlighting themes from yesterday. I think I could
possibly stress-test the patches for about 2 weeks before the release?
And people would run Git-built Zsh for that time, but not utilizing
the support (unless I release some 24-bit themes early in F-Sy-H, then
some number of people could be testing it too).

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Should there be a Zsh release for true-color support to adopt earlier?

[Daniel Shahaf]

Sebastian Gniazdowski wrote on Tue, 06 Nov 2018 13:22 +0100:
> Hello
> Could there be a release made shortly to introduce true color support
> possibly early, without possible complications when other changes
> arise, and without waiting a half year for it to happen?
> 

I don't see anything stopping us from doing a "development" release,
5.6.1-test-0.

I don't have an opinion on whether to issue a 5.7.  That depends on what
other changes have been made since 5.6, and I don't have time to review
'git log' currently to assess that.

I won't be available to RM.

> There's a matter of testing the patches. I'm running patched Zsh with
> true-color syntax-highlighting themes from yesterday. I think I could
> possibly stress-test the patches for about 2 weeks before the release?
> And people would run Git-built Zsh for that time, but not utilizing
> the support (unless I release some 24-bit themes early in F-Sy-H, then
> some number of people could be testing it too).

Re: Should there be a Zsh release for true-color support to adopt earlier?

[Bart Schaefer]

On Tue, Nov 6, 2018 at 2:46 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> I don't have an opinion on whether to issue a 5.7.  That depends on what
> other changes have been made since 5.6, and I don't have time to review
> 'git log' currently to assess that.

Several more attachtty/subjob/superjob fixes that are probably worth
releasing (if people have actually been running the head checkout to
try them).

Re: Terminal theme tool – a workaround for lack of 24-bit color in Zsh?

[Sebastian Gniazdowski]

On Wed, 31 Oct 2018 at 00:41, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> The nearest colour matching code converts RGB colours to CIE L*a*b*.
> There are more sophisticated methods but this does a noticably better
> job than just taking distances in RGB space. It doesn't use any of the

I have an example of CIELab vs. naive nearest-finding (I've earlier
used code from Oliver). On the page:

http://psprint.blinkenshell.org/palette.xhtml

there are two palettes, one converted by CIELab, the second one with
the naive method. The source of the original colors is this theme:
https://www.syntaxenvy.com/0063633. The point: Syntax-Envy generates
colors with the same lightness. And CIELab successfully preserves that
property. Just looking at the 6 colors reveals this, they are quite
the same bright.

Naive conversion openly breaks down, first SteelBlue3, which is quite
dark, then DarkKhaki, which is very dark, and then the repeated
LightSkyBlue3, where CIELab found a distinct candidate – CadetBlue.

To make this post little less dependent on an external web page, the
CIELab palette is: LightSkyBlue1, LightSkyBlue3, LightGoldenrod2,
CadetBlue, PaleGreen3, LightSkyBlue3, Grey11, while the naive palette
is: LightSkyBlue3, SteelBlue3, DarkKhaki, LightSkyBlue3, PaleGreen3,
LightSkyBlue3, Grey0 .

> I'd be interested if anyone has any thoughts or views on any of this?
>
> Oliver
>
> PS. Thanks to Sebastian for testing out an earlier version of the
> colour matching code.
>
> diff --git a/Src/Modules/nearcolor.c b/Src/Modules/nearcolor.c
> new file mode 100644
> index 000000000..2a763d470
> --- /dev/null
> +++ b/Src/Modules/nearcolor.c
> @@ -0,0 +1,178 @@
> +#include "nearcolor.mdh"
> +#include "nearcolor.pro"
> +
> +#include <math.h>
> +
> +struct cielab {
> +    float L, a, b;
> +};
> +typedef struct cielab *Cielab;
> +
> +static float
> +deltae(Cielab lab1, Cielab lab2)
> +{
> +    /* taking square root unnecessary as we're just comparing values */
> +    return powf(lab1->L - lab2->L, 2) +
> +       powf(lab1->a - lab2->a, 2) +
> +       powf(lab1->b - lab2->b, 2);
> +}
> +
> +static void
> +RGBtoLAB(int red, int green, int blue, Cielab lab)
> +{
> +    float R = (float)red / 255.0;
> +    float G = (float)green / 255.0;
> +    float B = (float)blue / 255.0;
> +    R = 100.0 * (R > 0.04045 ? powf((R + 0.055) / 1.055, 2.4) : R / 12.92);
> +    G = 100.0 * (G > 0.04045 ? powf((G + 0.055) / 1.055, 2.4) : G / 12.92);
> +    B = 100.0 * (B > 0.04045 ? powf((B + 0.055) / 1.055, 2.4) : B / 12.92);
> +
> +    /* Observer. = 2 degrees, Illuminant = D65 */
> +    float X = (R * 0.4124 + G * 0.3576 + B * 0.1805) / 95.047;
> +    float Y = (R * 0.2126 + G * 0.7152 + B * 0.0722) / 100.0;
> +    float Z = (R * 0.0193 + G * 0.1192 + B * 0.9505) / 108.883;
> +
> +    X = (X > 0.008856) ? powf(X, 1.0/3.0) : (7.787 * X) + (16.0 / 116.0);
> +    Y = (Y > 0.008856) ? powf(Y, 1.0/3.0) : (7.787 * Y) + (16.0 / 116.0);
> +    Z = (Z > 0.008856) ? powf(Z, 1.0/3.0) : (7.787 * Z) + (16.0 / 116.0);
> +
> +    lab->L = (116.0 * Y) - 16.0;
> +    lab->a = 500.0 * (X - Y);
> +    lab->b = 200.0 * (Y - Z);
> +}
> +
> +static int
> +mapRGBto88(int red, int green, int blue)
> +{
> +    int component[] = { 0, 0x8b, 0xcd, 0xff, 0x2e, 0x5c, 0x8b, 0xa2, 0xb9, 0xd0, 0xe7 };
> +    struct cielab orig, next;
> +    float nextl, bestl = -1;
> +    int r, g, b;
> +    int comp_r = 0, comp_g = 0, comp_b = 0;
> +
> +    /* Get original value */
> +    RGBtoLAB(red, green, blue, &orig);
> +
> +    /* try every one of the 72 colours */
> +    for (r = 0; r < 11; r++) {
> +       for (g = 0; g <= 3; g++) {
> +           for (b = 0; b <= 3; b++) {
> +               if (r > 3) g = b = r; /* advance inner loops to the block of greys */
> +               RGBtoLAB(component[r], component[g], component[b], &next);
> +               nextl = deltae(&orig, &next);
> +               if (nextl < bestl || bestl < 0) {
> +                   bestl = nextl;
> +                   comp_r = r;
> +                   comp_g = g;
> +                   comp_b = b;
> +               }
> +           }
> +       }
> +    }
> +
> +    return (comp_r > 3) ? 77 + comp_r :
> +        16 + (comp_r * 16) + (comp_g * 4) + comp_b;
> +}
> +
> +/*
> + * Convert RGB to nearest colour in the 256 colour range
> + */
> +static int
> +mapRGBto256(int red, int green, int blue)
> +{
> +    int component[] = {
> +       0, 0x5f, 0x87, 0xaf, 0xd7, 0xff,
> +       0x8, 0x12, 0x1c, 0x26, 0x30, 0x3a, 0x44, 0x4e,
> +       0x58, 0x62, 0x6c, 0x76, 0x80, 0x8a, 0x94, 0x9e,
> +       0xa8, 0xb2, 0xbc, 0xc6, 0xd0, 0xda, 0xe4, 0xee
> +    };
> +    struct cielab orig, next;
> +    float nextl, bestl = -1;
> +    int r, g, b;
> +    int comp_r = 0, comp_g = 0, comp_b = 0;
> +
> +    /* Get original value */
> +    RGBtoLAB(red, green, blue, &orig);
> +
> +    for (r = 0; r < sizeof(component)/sizeof(*component); r++) {
> +       for (g = 0; g <= 5; g++) {
> +           for (b = 0; b <= 5; b++) {
> +               if (r > 5) g = b = r; /* advance inner loops to the block of greys */
> +               RGBtoLAB(component[r], component[g], component[b], &next);
> +               nextl = deltae(&orig, &next);
> +               if (nextl < bestl || bestl < 0) {
> +                   bestl = nextl;
> +                   comp_r = r;
> +                   comp_g = g;
> +                   comp_b = b;
> +               }
> +           }
> +       }
> +    }
> +
> +    return (comp_r > 5) ? 226 + comp_r :
> +       16 + (comp_r * 36) + (comp_g * 6) + comp_b;
> +}
> +
> +static int
> +getnearestcolor(UNUSED(Hookdef dummy), Color_rgb col)
> +{
> +    if (tccolours == 256)
> +       return mapRGBto256(col->red, col->green, col->blue);
> +    if (tccolours == 88)
> +       return mapRGBto88(col->red, col->green, col->blue);
> +    return 0;
> +}
> +
> +static struct features module_features = {
> +    NULL, 0,
> +    NULL, 0,
> +    NULL, 0,
> +    NULL, 0,
> +    0
> +};
> +
> +/**/
> +int
> +setup_(UNUSED(Module m))
> +{
> +    return 0;
> +}
> +
> +/**/
> +int
> +features_(Module m, char ***features)
> +{
> +    *features = featuresarray(m, &module_features);
> +    return 0;
> +}
> +
> +/**/
> +int
> +enables_(Module m, int **enables)
> +{
> +    return handlefeatures(m, &module_features, enables);
> +}
> +
> +/**/
> +int
> +boot_(Module m)
> +{
> +    addhookfunc("get_color_attr", (Hookfn) getnearestcolor);
> +    return 0;
> +}
> +
> +/**/
> +int
> +cleanup_(Module m)
> +{
> +    deletehookfunc("get_color_attr", (Hookfn) getnearestcolor);
> +    return setfeatureenables(m, &module_features, NULL);
> +}
> +
> +/**/
> +int
> +finish_(UNUSED(Module m))
> +{
> +    return 0;
> +}
> diff --git a/Src/Modules/nearcolor.mdd b/Src/Modules/nearcolor.mdd
> new file mode 100644
> index 000000000..2fcdaf04e
> --- /dev/null
> +++ b/Src/Modules/nearcolor.mdd
> @@ -0,0 +1,5 @@
> +name=zsh/nearcolor
> +link=dynamic
> +load=no
> +
> +objects="nearcolor.o"
> diff --git a/Src/init.c b/Src/init.c
> index cec914329..e7e62e2f7 100644
> --- a/Src/init.c
> +++ b/Src/init.c
> @@ -94,6 +94,7 @@ mod_export struct hookdef zshhooks[] = {
>      HOOKDEF("exit", NULL, HOOKF_ALL),
>      HOOKDEF("before_trap", NULL, HOOKF_ALL),
>      HOOKDEF("after_trap", NULL, HOOKF_ALL),
> +    HOOKDEF("get_color_attr", NULL, HOOKF_ALL),
>  };
>
>  /* keep executing lists until EOF found */
> diff --git a/Src/prompt.c b/Src/prompt.c
> index 959ed8e3d..39edbdb2b 100644
> --- a/Src/prompt.c
> +++ b/Src/prompt.c
> @@ -1621,7 +1621,24 @@ match_colour(const char **teststrp, int is_fg, int colour)
>      int shft, on, named = 0, tc;
>
>      if (teststrp) {
> -       if ((named = ialpha(**teststrp))) {
> +       if (**teststrp == '#' && isxdigit((*teststrp)[1])) {
> +           struct color_rgb color;
> +           char *end;
> +           zlong col = zstrtol(*teststrp+1, &end, 16);
> +            if (end - *teststrp == 4) {
> +               color.red = col >> 8 | ((col >> 8) << 4);
> +               color.green = (col & 0xf0) >> 4;
> +               color.green |= color.green << 4;
> +               color.blue = col & 0xf;
> +               color.blue |= color.blue << 4;
> +           } else if (end - *teststrp == 7) {
> +               color.red = col >> 16;
> +               color.green = (col & 0xff00) >> 8;
> +               color.blue = col & 0xff;
> +           }
> +           *teststrp = end;
> +           colour = runhookdef(GETCOLORATTR, &color);
> +       } else if ((named = ialpha(**teststrp))) {
>             colour = match_named_colour(teststrp);
>             if (colour == 8) {
>                 /* default */
> diff --git a/Src/zsh.h b/Src/zsh.h
> index 894158818..68731e226 100644
> --- a/Src/zsh.h
> +++ b/Src/zsh.h
> @@ -2707,6 +2707,12 @@ struct ttyinfo {
>  #define COL_SEQ_BG     (1)
>  #define COL_SEQ_COUNT  (2)
>
> +struct color_rgb {
> +    unsigned int red, green, blue;
> +};
> +
> +typedef struct color_rgb *Color_rgb;
> +
>  /*
>   * Flags to testcap() and set_colour_attribute (which currently only
>   * handles TSC_PROMPT).
> @@ -3203,6 +3209,7 @@ enum {
>  #define EXITHOOK       (zshhooks + 0)
>  #define BEFORETRAPHOOK (zshhooks + 1)
>  #define AFTERTRAPHOOK  (zshhooks + 2)
> +#define GETCOLORATTR   (zshhooks + 3)
>
>  #ifdef MULTIBYTE_SUPPORT
>  /* Final argument to mb_niceformat() */



-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Terminal theme tool – a workaround for lack of 24-bit color in Zsh?

[Jun T]

In nearcolor.c, function RGBtoLAB(), floating point data are saved in
float (not double) variables; for example:

> +     float R = (float)red / 255.0;
(snip)
> +     R = 100.0 * (R > 0.04045 ? powf((R + 0.055) / 1.055, 2.4) : R / 12.92);

Is this to improve performance?
If so, it would be better to explicitly use float literals, such as 255.0f.
Otherwise, since 255.0 is double, all the calculations are done in double
(after promoting all values to double), and the final result is truncated
back to float.

I believe there is virtually no performance difference (on most CPUs).
The reason I'm writing this is I'm getting lots of warnings like:

nearcolor.c:55:18: warning: implicit conversion increases floating-point precision: 'float' to 'double' [-Wdouble-promotion]
    R = 100.0 * (R > 0.04045 ? powf((R + 0.055) / 1.055, 2.4) : R / 12.92);
                 ^ ~

-Wdouble-promotion is OFF by default, but I still want to remove these
warnings.

Re: Terminal theme tool – a workaround for lack of 24-bit color in Zsh?

[Peter Stephenson]

On Thu, 2018-11-15 at 20:50 +0900, Jun T wrote:
> In nearcolor.c, function RGBtoLAB(), floating point data are saved in
> float (not double) variables; for example:
> 
> > 
> > +     float R = (float)red / 255.0;
> (snip)
> > 
> > +     R = 100.0 * (R > 0.04045 ? powf((R + 0.055) / 1.055, 2.4) : R / 12.92);
> Is this to improve performance?
> If so, it would be better to explicitly use float literals, such as 255.0f.
> Otherwise, since 255.0 is double, all the calculations are done in double
> (after promoting all values to double), and the final result is truncated
> back to float.
> 
> I believe there is virtually no performance difference (on most CPUs).

The majority of standardish Unixy systems I know about are designed to
assume most floating point calculations are going to be done as double
--- I see Linux supplies float and long double versions of library
functions as a supplement.  But I doubt Oliver was thinking that much
about optimisation here --- terminal IO is slow enough I can't see how
it could be a big deal.

pws

Re: Terminal theme tool – a workaround for lack of 24-bit color in Zsh?

[Jun T]

Is it OK to replace all floats by doubles?

What we get is just to silence the (non-standard) warnings.


diff --git a/Src/Modules/nearcolor.c b/Src/Modules/nearcolor.c
index 0b9877bf4..b49ee9afb 100644
--- a/Src/Modules/nearcolor.c
+++ b/Src/Modules/nearcolor.c
@@ -33,37 +33,37 @@
 #include <math.h>
 
 struct cielab {
-    float L, a, b;
+    double L, a, b;
 };
 typedef struct cielab *Cielab;
 
-static float
+static double
 deltae(Cielab lab1, Cielab lab2)
 {
     /* taking square root unnecessary as we're just comparing values */
-    return powf(lab1->L - lab2->L, 2) +
-	powf(lab1->a - lab2->a, 2) +
-	powf(lab1->b - lab2->b, 2);
+    return pow(lab1->L - lab2->L, 2) +
+	pow(lab1->a - lab2->a, 2) +
+	pow(lab1->b - lab2->b, 2);
 }
 
 static void
 RGBtoLAB(int red, int green, int blue, Cielab lab)
 {
-    float R = (float)red / 255.0;
-    float G = (float)green / 255.0;
-    float B = (float)blue / 255.0;
-    R = 100.0 * (R > 0.04045 ? powf((R + 0.055) / 1.055, 2.4) : R / 12.92);
-    G = 100.0 * (G > 0.04045 ? powf((G + 0.055) / 1.055, 2.4) : G / 12.92);
-    B = 100.0 * (B > 0.04045 ? powf((B + 0.055) / 1.055, 2.4) : B / 12.92);
+    double R = red / 255.0;
+    double G = green / 255.0;
+    double B = blue / 255.0;
+    R = 100.0 * (R > 0.04045 ? pow((R + 0.055) / 1.055, 2.4) : R / 12.92);
+    G = 100.0 * (G > 0.04045 ? pow((G + 0.055) / 1.055, 2.4) : G / 12.92);
+    B = 100.0 * (B > 0.04045 ? pow((B + 0.055) / 1.055, 2.4) : B / 12.92);
 
     /* Observer. = 2 degrees, Illuminant = D65 */
-    float X = (R * 0.4124 + G * 0.3576 + B * 0.1805) / 95.047;
-    float Y = (R * 0.2126 + G * 0.7152 + B * 0.0722) / 100.0;
-    float Z = (R * 0.0193 + G * 0.1192 + B * 0.9505) / 108.883;
+    double X = (R * 0.4124 + G * 0.3576 + B * 0.1805) / 95.047;
+    double Y = (R * 0.2126 + G * 0.7152 + B * 0.0722) / 100.0;
+    double Z = (R * 0.0193 + G * 0.1192 + B * 0.9505) / 108.883;
 
-    X = (X > 0.008856) ? powf(X, 1.0/3.0) : (7.787 * X) + (16.0 / 116.0);
-    Y = (Y > 0.008856) ? powf(Y, 1.0/3.0) : (7.787 * Y) + (16.0 / 116.0);
-    Z = (Z > 0.008856) ? powf(Z, 1.0/3.0) : (7.787 * Z) + (16.0 / 116.0);
+    X = (X > 0.008856) ? pow(X, 1.0/3.0) : (7.787 * X) + (16.0 / 116.0);
+    Y = (Y > 0.008856) ? pow(Y, 1.0/3.0) : (7.787 * Y) + (16.0 / 116.0);
+    Z = (Z > 0.008856) ? pow(Z, 1.0/3.0) : (7.787 * Z) + (16.0 / 116.0);
 
     lab->L = (116.0 * Y) - 16.0;
     lab->a = 500.0 * (X - Y);
@@ -75,7 +75,7 @@ mapRGBto88(int red, int green, int blue)
 {
     int component[] = { 0, 0x8b, 0xcd, 0xff, 0x2e, 0x5c, 0x8b, 0xa2, 0xb9, 0xd0, 0xe7 };
     struct cielab orig, next;
-    float nextl, bestl = -1;
+    double nextl, bestl = -1;
     int r, g, b;
     int comp_r = 0, comp_g = 0, comp_b = 0;
 
@@ -116,7 +116,7 @@ mapRGBto256(int red, int green, int blue)
 	0xa8, 0xb2, 0xbc, 0xc6, 0xd0, 0xda, 0xe4, 0xee
     };
     struct cielab orig, next;
-    float nextl, bestl = -1;
+    double nextl, bestl = -1;
     int r, g, b;
     int comp_r = 0, comp_g = 0, comp_b = 0;
 

Re: Re: Terminal theme tool – a workaround for lack of 24-bit color in Zsh?

[Oliver Kiddle]

On 16 Nov, Jun T wrote:
> Is it OK to replace all floats by doubles?

It should be.   Thanks.
Either this or putting f at the end of all the literals to force them to
be float should be fine.

I was initially unsure whether I'd need to be keeping a cache of
struct cielabs. For 256 triples, float over double would be a
memory saving.  In practice it was quick enough not to warrant a
cache. On most modern hardware, there's little to no performance
difference between float and double. It may make a difference on
the sort of old hardware where it actually matters like something that
only does soft-float. My one remaining device in that category died
recently.

Oliver