How can Zle -U interfere with zsh-syntax-highlighting?

How can Zle -U interfere with zsh-syntax-highlighting?

[Sebastian Gniazdowski]

Hello
Below is a widget that outputs long sophisticated string with zle -U:

testwidg() {
  zle -U 'a=; a=${(r:100000::_:)a}; len=${#a}; zshstyle() { repeat
100; do b=${a[-5000,-1]}; done }; zshstyle2() { repeat 100; do
b=${a[-5000,len]}; done }; shstyle() { repeat 100; do b=${a:
-5000:5000}; done }; time ( zshstyle ); time ( zshstyle2 ); time (
shstyle )'
}
zle -N testwidg
bindkey '^t' testwidg

With z-sy-h enabled, it takes 4 seconds to appear after pressing
Ctrl-T. Pasting the same text is instant and correct (i.e.
highlighting works). What can be happening? Is there some alternative
to Zle -U?

PS. Recently, forks have been optimized out from z-sy-h. No fork for
type -w is done. Without the opt it's not 4 seconds but 10.

Best regards,
Sebastian Gniazdowski

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Sebastian Gniazdowski]

It's interesting that pressing e.g. space right after Ctrl-T produces
instant, correct reaction

Best regards,
Sebastian Gniazdowski

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Bart Schaefer]

On Sat, May 14, 2016 at 2:07 PM, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> Below is a widget that outputs long sophisticated string with zle -U:
>
[...]
>
> With z-sy-h enabled, it takes 4 seconds to appear after pressing
> Ctrl-T. Pasting the same text is instant and correct (i.e.
> highlighting works). What can be happening?

This (line 73-74 in current z-sy-h.zsh):

# Do not highlight if there are pending inputs (copy/paste).
[[ $PENDING -gt 0 ]] && return $ret

"zle -U" feeds the input back to zle one character at a time but does
not set PENDING, so I suspect the highlight is being recaculated after
every character.  I suppose it might make sense for "zle -U" to
pretend to be actual input even to the extent of PENDING, but it
currently does not.

This also explains why immediately typing something else speeds things
up:  Highlighting pauses until that character is consumed.

> Is there some alternative to Zle -U?

Other than just LBUFFER+="long text here", no.  If you want the other
effects of zle -U (i.e., that each character is interpreted as if it
had been typed) then there is no alternative.

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Sebastian Gniazdowski]

On 15 May 2016 at 13:13, Bart Schaefer <schaefer@brasslantern.com> wrote:
>> Is there some alternative to Zle -U?
>
> Other than just LBUFFER+="long text here", no.  If you want the other
> effects of zle -U (i.e., that each character is interpreted as if it
> had been typed) then there is no alternative.

Forgot about LBUFFER, thanks! I have two tools, and it is interesting
that in one (n-history) doing LBUFFER+="$var_with_long_text" causes
highlighting to correctly work (and also solves the lag issue) whie in
other tool (n-kill) it doesn't trigger highlighting. I made the two
responsible code blocks identical (I do zle redisplay and zle
kill-buffer before zle -U / LBUFFER+=), even hardcoded the text:

LBUFFER+='a=; a=${(r:100000::_:)a}...'

and it still behaves this way. Both tools use emulate -L zsh, also
tried emulate -LR zsh. Luckily, I need LBUFFER+= only in the one tool
that works.

Best regards,
Sebastian Gniazdowski

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Sebastian Gniazdowski]

PS. Also, just noting that print -zr doesn't do highlighting

Best regards,
Sebastian Gniazdowski

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Mikael Magnusson]

On Sun, May 15, 2016 at 5:24 PM, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> PS. Also, just noting that print -zr doesn't do highlighting

When the stack pops, the code that calls the pre-redraw hook isn't
reached. I tried putting it there but it caused crashes. I asked if
anyone was interested in looking into that when I posted the patches
but IIRC there were no responses to that. Those mails probably have
some info about what areas of the code to look at though. If you do
zle -M "$BUFFER" in your zle-line-init hook, you will see that it
already contains the text, so doing highlight from there _should_
work, but doesn't. I guess the hook is called before the code that
unsets all the region_highlight arrays.

-- 
Mikael Magnusson

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Sebastian Gniazdowski]

On 15 May 2016 at 17:24, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
> Luckily, I need LBUFFER+= only in the one tool that works.

To early success call. Suspecting some degree of randomness in this, I
did test on FreeBSD box and LBUFFER+= doesn't trigger highlighting
there also in my first tool n-history. :/ Any thoughts where the
slight cause of "highlight or not" can be located?

Best regards,
Sebastian Gniazdowski

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Sebastian Gniazdowski]

On 15 May 2016 at 17:41, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
> On 15 May 2016 at 17:24, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
>> Luckily, I need LBUFFER+= only in the one tool that works.
>
> To early success call. Suspecting some degree of randomness in this, I
> did test on FreeBSD box and LBUFFER+= doesn't trigger highlighting
> there also in my first tool n-history. :/ Any thoughts where the
> slight cause of "highlight or not" can be located?

Turned out this depends on z-sy-h version. Starting from commit
487b122 doing LBUFFER+= triggers highlighting. Verified on FreeBSD,
Ubuntu, Arch Linux, OS X. However one RHEL/Fedora user still reports
that highlighting isn't triggered.

Best regards,
Sebastian Gniazdowski

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Daniel Shahaf]

Bart Schaefer wrote on Sun, May 15, 2016 at 04:13:21 -0700:
> I suppose it might make sense for "zle -U" to pretend to be actual
> input even to the extent of PENDING, but it currently does not.

Like this?

[[[
diff --git a/Src/Zle/zle_params.c b/Src/Zle/zle_params.c
index c6387bf..d5b1e87 100644
--- a/Src/Zle/zle_params.c
+++ b/Src/Zle/zle_params.c
@@ -500,7 +500,7 @@ get_bufferlines(UNUSED(Param pm))
 static zlong
 get_pending(UNUSED(Param pm))
 {
-    return noquery(0);
+    return noquery(0) + kungetct;
 }
 
 /**/
]]]

It doesn't cause any obvious problem, but I couldn't reproduce the
original slowdown so I'm not sure whether this patch fixes it.

In any case, I won't push it yet since I won't be able to commit
followups timely in case it breaks something — but if someone else wants
to push it before I get back, feel free.

Cheers,

Daniel

Re: How can Zle -U interfere with zsh-syntax-highlighting?

[Sebastian Gniazdowski]

On 19 May 2016 at 23:22, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Bart Schaefer wrote on Sun, May 15, 2016 at 04:13:21 -0700:
>> I suppose it might make sense for "zle -U" to pretend to be actual
>> input even to the extent of PENDING, but it currently does not.
>
> Like this?
>
> [[[
> diff --git a/Src/Zle/zle_params.c b/Src/Zle/zle_params.c
> ...
> It doesn't cause any obvious problem, but I couldn't reproduce the
> original slowdown so I'm not sure whether this patch fixes it.

I've tested the patch and it still has delay

https://asciinema.org/a/4amtz0ie97ec3zjtg6oj0ywvr

Best regards,
Sebastian Gniazdowski