Fun redraw issue with double-width characters

Fun redraw issue with double-width characters

[Mikael Magnusson]

I lied about the "fun".

I got a random thought, "what if i enter five doublewidth characters,
then go back to the start and enter a bunch of spaces". Turns out
doing that causes the line to clear and cursor to move up one line.
Doesn't happen with four or less. Long story short, it happens only if
ich1 is defined in the terminfo for the current $TERM, which it
happens to be for rxvt-unicode, but also for rxvt. It doesn't matter
if you're using xterm or rxvt or urxvt, just set TERM=rxvt, enter five
ま and go back to the start and hold down space. I don't really know
where to begin debugging this. I checked strace output and these lines
are what zsh does when there are two, one and zero cells remaining on
the line, respectively (note that the redraw issue happens not on
wrap, but the step before).

write(10, " \343\201\276\343\201\276\343\201\276\343\201\276\343\201\276\33[10D",
21) = 21
write(10, "\33[@ \33[10C\33[K\33[10D", 17) = 17
write(10, "\33[@ \33[9C \r\33[K\33[A\33[84C", 21) = 21

If i remove the ich1 entry from terminfo (it's the \33[@ bit), i don't
have the problem.

write(10, " \343\201\276\343\201\276\343\201\276\343\201\276\343\201\276\33[10D",
21) = 21
write(10, " \343\201\276\343\201\276\343\201\276\343\201\276\343\201\276\33[10D",
21) = 21
write(10, " \343\201\276\343\201\276\343\201\276\343\201\276\343\201\276\33[K\33[10D",
24) = 24

-- 
Mikael Magnusson

Re: Fun redraw issue with double-width characters

[Peter Stephenson]

On Sun, 24 Oct 2010 14:11:08 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> I got a random thought, "what if i enter five doublewidth characters,
> then go back to the start and enter a bunch of spaces". Turns out
> doing that causes the line to clear and cursor to move up one line.
> Doesn't happen with four or less.

I think that may depend on what you're inserting: if you're inserting
stuff such that the character you're inserting changes the line near
the start, it may happen early.   But this may be a wild goose chase.

> Long story short, it happens only if
> ich1 is defined in the terminfo for the current $TERM, which it
> happens to be for rxvt-unicode, but also for rxvt. It doesn't matter
> if you're using xterm or rxvt or urxvt, just set TERM=rxvt, enter five
> ま and go back to the start and hold down space. I don't really know
> where to begin debugging this.

Actually, there's only one use of insert in the entire source, which is
in a quite short chunk in zle_refresh.c, so localising it was the easy
bit.

After staring enough I think I've worked out what's going on, but you
might want to stare at it, too.  Much of this is reformatting for
clarity, the important bit is near the end where I've moved an extra
chunk inside an #ifdef MULTIBYTE_SUPPORT with a new comment.

Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.81
diff -p -u -r1.81 zle_refresh.c
--- Src/Zle/zle_refresh.c	17 Oct 2010 17:59:27 -0000	1.81
+++ Src/Zle/zle_refresh.c	24 Oct 2010 18:06:22 -0000
@@ -2000,7 +2000,7 @@ refreshline(int ln)
 		   newline: foobar	 / characters, then we have six matches */
 		if (tccan(TCDEL)) {
 		    int first = 1;
-		    for (i = 1; ol[i].chr; i++)
+		    for (i = 1; ol[i].chr; i++) {
 			if (tcdelcost(i) < wpfxlen(ol + i, nl)) {
 			    /*
 			     * Some terminals will output the current
@@ -2023,15 +2023,19 @@ refreshline(int ln)
 			    i = 0;
 			    break;
 			}
+		    }
 		    if (!i)
 			continue;
 		}
-		/* inserting characters - characters pushed off the right should be
-		   annihilated, but we don't do this if we're on the last line lest
-		   undesired scrolling occurs due to `illegal' characters on screen */
-
-		if (tccan(TCINS) && (vln != lines - 1)) {	/* not on last line */
-		    for (i = 1; nl[i].chr; i++)
+		/*
+		 * inserting characters - characters pushed off the right
+		 * should be annihilated, but we don't do this if we're on the
+		 * last line lest undesired scrolling occurs due to `illegal'
+		 * characters on screen
+		 */ 
+		if (tccan(TCINS) && (vln != lines - 1)) {
+		    /* not on last line */
+		    for (i = 1; nl[i].chr; i++) {
 			if (tcinscost(i) < wpfxlen(ol, nl + i)) {
 			    tc_inschars(i);
 			    zwrite(nl, i);
@@ -2044,19 +2048,37 @@ refreshline(int ln)
 #endif
 			    char_ins += i;
 			    ccs = (vcs += i);
-			    /* if we've pushed off the right, truncate oldline */
-			    for (i = 0; ol[i].chr && i < winw - ccs; i++);
+			    /*
+			     * if we've pushed off the right, truncate
+			     * oldline
+			     */
+			    for (i = 0; ol[i].chr && i < winw - ccs; i++)
+				;
 #ifdef MULTIBYTE_SUPPORT
 			    while (ol[i].chr == WEOF)
 				i++;
-#endif
 			    if (i >= winw - ccs) {
+				/*
+				 * Yes, we're over the right.
+				 * Make sure we truncate at the real
+				 * character, not a WEOF added to
+				 * make up the width.
+				 */
+				while (ol[i-1].chr == WEOF)
+				    i--;
 				ol[i] = zr_zr;
 				ins_last = 1;
 			    }
+#else
+			    if (i >= winw - ccs) {
+				ol[i] = zr_zr;
+				ins_last = 1;
+			    }
+#endif
 			    i = 0;
 			    break;
 			}
+		    }
 		    if (!i)
 			continue;
 		}

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Fun redraw issue with double-width characters

[Mikael Magnusson]

On 24 October 2010 20:12, Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On Sun, 24 Oct 2010 14:11:08 +0200
> Mikael Magnusson <mikachu@gmail.com> wrote:
>> I got a random thought, "what if i enter five doublewidth characters,
>> then go back to the start and enter a bunch of spaces". Turns out
>> doing that causes the line to clear and cursor to move up one line.
>> Doesn't happen with four or less.
>
> I think that may depend on what you're inserting: if you're inserting
> stuff such that the character you're inserting changes the line near
> the start, it may happen early.   But this may be a wild goose chase.
>
>> Long story short, it happens only if
>> ich1 is defined in the terminfo for the current $TERM, which it
>> happens to be for rxvt-unicode, but also for rxvt. It doesn't matter
>> if you're using xterm or rxvt or urxvt, just set TERM=rxvt, enter five
>> ま and go back to the start and hold down space. I don't really know
>> where to begin debugging this.
>
> Actually, there's only one use of insert in the entire source, which is
> in a quite short chunk in zle_refresh.c, so localising it was the easy
> bit.

Well, easy if you know ich1 means TCINS :).

> After staring enough I think I've worked out what's going on, but you
> might want to stare at it, too.  Much of this is reformatting for
> clarity, the important bit is near the end where I've moved an extra
> chunk inside an #ifdef MULTIBYTE_SUPPORT with a new comment.

I stared at it for a while, but nothing of interest happened. I did
test it though, and it works.


I am asking this only out of curiosity, how come updates on the first
line are done like this
write(10, " \343\201\276\343\201\276\343\201\276\343\201\276\343\201\276\33[10D",
21) = 21
while updates on the second line (ie after wrapping everything) are
done shorter, like this
write(10, "\33[@ \33[10C \33[11D", 15)  = 15
? (when inserting spaces in front of our five friends)

Also noticed this didn't seem to be documented. I usually have special
set to bold, but when looking at strace output in the first mail i did
unset zle_highlight and noticed this. (what i should have done was
zle_highlight=(none) ).

diff --git a/Doc/Zsh/zle.yo b/Doc/Zsh/zle.yo
index 92b5f20..b9abaac 100644
--- a/Doc/Zsh/zle.yo
+++ b/Doc/Zsh/zle.yo
@@ -2350,6 +2350,11 @@ Not all systems support this: for it to work,
the system's representation of
 wide characters must be code values from the Universal Character Set,
 as defined by IS0 10646 (also known as Unicode).
 )
+item(Wrapped double-width characters)(
+When a double-width character appears in the final column of a line, it
+is instead shown on the next line. The empty space left in the original
+position is highlighted as a special character.
+)
 enditem()

 If tt(zle_highlight) is not set or no value applies to a particular

-- 
Mikael Magnusson

Re: Fun redraw issue with double-width characters

[Bart Schaefer]

On Oct 25,  6:14am, Mikael Magnusson wrote:
}
} I am asking this only out of curiosity, how come updates on the first
} line are done like [...]
} while updates on the second line (ie after wrapping everything) are
} done shorter, like [...]

There's a complicated algorithm based on querying terminfo/curses for
the "expense" involved in doing any given screen update, which is used
to decide whether it's more efficent to send terminal controls to move
the screen image around, or instead to simply overwrite what's there.

It's possible that the computation of "expense" is wrong when some of
the characters to be moved are multibyte, but it also might be that what
constitutes "expense" doesn't map directly to the number of bytes that
pass through write().

Re: Fun redraw issue with double-width characters

[Mikael Magnusson]

Any complaints about this one?

On 25 October 2010 06:14, Mikael Magnusson <mikachu@gmail.com> wrote:
> Also noticed this didn't seem to be documented. I usually have special
> set to bold, but when looking at strace output in the first mail i did
> unset zle_highlight and noticed this. (what i should have done was
> zle_highlight=(none) ).
>
> diff --git a/Doc/Zsh/zle.yo b/Doc/Zsh/zle.yo
> index 92b5f20..b9abaac 100644
> --- a/Doc/Zsh/zle.yo
> +++ b/Doc/Zsh/zle.yo
> @@ -2350,6 +2350,11 @@ Not all systems support this: for it to work,
> the system's representation of
>  wide characters must be code values from the Universal Character Set,
>  as defined by IS0 10646 (also known as Unicode).
>  )
> +item(Wrapped double-width characters)(
> +When a double-width character appears in the final column of a line, it
> +is instead shown on the next line. The empty space left in the original
> +position is highlighted as a special character.
> +)
>  enditem()
>
>  If tt(zle_highlight) is not set or no value applies to a particular
>
> --
> Mikael Magnusson
>



-- 
Mikael Magnusson

Re: Fun redraw issue with double-width characters

[Peter Stephenson]

On Fri, 27 May 2011 03:28:59 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:

> Any complaints about this one?
> > +item(Wrapped double-width characters)(
> > +When a double-width character appears in the final column of a
> > line, it +is instead shown on the next line. The empty space left
> > in the original +position is highlighted as a special character.
> > +)

Looks fine.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog