ZLE Vi-mode: visual mode selection doesn't match the characters highlighted

ZLE Vi-mode: visual mode selection doesn't match the characters highlighted

[Doron Behar]

Hey all,

I'm having a small inconvenience with the way characters are highlighted
in visual mode.

Say my $BUFFER has the following text, current $KEYMAP is "vicmd", and
the cursor is located where ^ is:

    ZSH is the best shell
                  ^

If I press `v`, and then `F<space>`, the cursor is located as expected
here:

    ZSH is the best shell
              ^
              -----            (selected characters)

Naturally, the selection now includes all characters from `t` to the
space (inclusive), as marked with the hyphens.

The problem is, that only the 3 characters ("bes") are highlighted so
it's not clear what characters exactly are selected.

In comparison, when moving _forward_ in visual mode, the characters
highlighted include all letters selected.

This behaviour is not like in Vim (with it's default settings) and is
confusing IMO. My expectation is that every character selected will be
highlighted, no matter whether the cursor is at the start or the end of
the selection.

Is there a way to tweak this behavior?

Doron.

Re: ZLE Vi-mode: visual mode selection doesn't match the characters highlighted

[Oliver Kiddle]

Doron Behar wrote:
> I'm having a small inconvenience with the way characters are highlighted
> in visual mode.
>
> Say my $BUFFER has the following text, current $KEYMAP is "vicmd", and
> the cursor is located where ^ is:

>     ZSH is the best shell
>               ^
>               -----            (selected characters)

> The problem is, that only the 3 characters ("bes") are highlighted so
> it's not clear what characters exactly are selected.

I can't reproduce this. Could you perhaps try with your setup/plugins
disabled by starting from `zsh -df'. It may also be worth trying a
different terminal emulator. If you use a vi command such as x or c, is
the full region affected?

If the position of the actual cursor is not highlighted, I would
suspect that your terminal emulator uses standout (reverse video)
for the cursor which combined with the default zsh use of standout
for the region reverses it back to the original state. What do you
have in the zle_highlight array? Try configuring the region, i.e.
region:bg=<something> It might also be worth configuring how your
terminal emulator displays the cursor.

The last character not being highlighted (the `t' of best) is curious.
This seems to perhaps reflect the emacs mode style where you wouldn't
expect `t' to be included in the selection because the original cursor
can logically be viewed as being situated between the `s' and the `t'.

This would imply a problem with the two lines in zle_refresh.c that
read:

        } else if (invicmdmode())
            INCPOS(region_highlights[0].end);

invicmdmode() merely checks !strcmp(curkeymapname, "vicmd")

If you're able to check curkeymapname at this point in a debugger, that
might help. Note that it should be `vicmd' and not `visual' because
`visual' is only used as a local keymap. If you've got some plugin that
has replaced 'vicmd' then I'd expect that to have broken a whole lot
more of vi mode.

Oliver

Re: ZLE Vi-mode: visual mode selection doesn't match the characters highlighted

[Doron Behar]

On Wed, Oct 16, 2019 at 09:37:50PM +0200, Oliver Kiddle wrote:
> On 13 Oct, Doron Behar wrote:
> > Thanks for suggesting that. Indeed this is a combination of terminal
> > emulators' settings / differences in behaviour along with a certain
> > plugin I use: https://github.com/zsh-users/zsh-syntax-highlighting
> 
> If I run zsh -df and then enable just that plugin by doing
>   . zsh-syntax-highlighting.zsh
> then I still can't reproduce the problem with highlighting of the t.
>
> It doesn't appear to create any new keymaps so I'm unsure why you
> don't get the `t' highlighted. Are you loading it with a plugin
> manager? It'd be useful to isolate just the minimum required to
> cause the problem.

I'm glad Oliver that you are attending _this_ issue as well but I'd like
to deal with it at that GitHub thread
(https://github.com/zsh-users/zsh-syntax-highlighting/issues/637), as I
don't feel it's related and focus here on the _cursor's_ character's
highlighting.

Never the less, I'll emphasize that this issue appears only when the
syntax highlighter colors the text of `t` in a certain foreground color.
If normal foreground color is used, `t` is highlighted properly.

> Zsh does highlight the cursor position. It is down to the terminal
> emulator how that appears. What terminal emulator do you use? What is
> $TERM set to and what OS are you on? I use a green solid block as my
> cursor and I know that the cursor position is included in the
> selection.  It appears much the same in vim and gvim.

I'm using NixOS and our `urxvt` is compiled according to this "recipe":

https://github.com/NixOS/nixpkgs/blob/7885eeaffdbd02ff6ce5483239fba1a063b314a9/pkgs/applications/misc/rxvt_unicode/default.nix

My `$TERM` is `rxvt-unicode-256color`.

I've tested as well: 

- Urxvt with an empty `.Xresources` (I ran `xrdb merge .Xresources`)
- Gnome-Terminal
- Konsole

I've demonstrated the problematic behaviour with `zsh -df` (without
syntax highlighting) here:

https://gist.github.com/doronbehar/cba65dbac52b5859e8e967a6bc71ea91#gistcomment-3057815

As you can see, if the cursor's color is not blinking, **all** tested
terminal emulators don't highlight the cursor's character.

I dearly hope the issue is clear now.

> You may be able to configure the cursor with escape sequences. For
> example, for a vertical bar (good for vi insert mode):
>   print -n $'\e[5 q'

I tried using an underline cursor, and it somewhat improved my
situation: The letter the cursor is standing at is highlighted so it's a
good-enough workaround but I hope this won't be considered as a solution
because in general, I prefer the block cursor.

> We could perhaps use some hooks for when visual mode is invoked and
> finished.

Yea I can use `zle-keymap-select` in the meantime...

> > Oliver, do you think perhaps ZSH could 'double reverse' the cursor
> > in visual mode? I think this will emulate Vim's behaviour better
> > because this way, with a non blinking cursor in visual mode, the
> > character the cursor stands upon will standout as well.
> 
> That could make it hard to see which end of the selection the cursor
> is at.

I can imagine, but if the cursor is a block and it's not blinking, no
sane user would expect to see on which end of the selection the cursor's
located since the cursor is a block. Vim can't help you distinguish this
as well.

With the current behaviour and a non blinking, block cursor, it's even
more confusing if the character the cursor is located at is not
highlighted at all but it _is_ selected!

> Whatever highlighting is chosen, there will be some terminal
> setup for which it is not ideal and somebody who complains.

I hope I've demonstrated good enough that this is not a terminal
emulator issue, it's an issue of a non blinking cursor. Hence, I think
my complain is legitimate.

> Vim's defaults result in a barely visible selection if you chose a
> near-white default background colour.

I'm not sure I agree. I tested Vim in Konsole and Gnome terminal and the
cursor was surely visible - a black block on a white or whitish
background instead of the other way around.

> Copying Vim might be tempting but Vim is not the only editor - zsh
> uses the term region for the selection in many cases because the
> feature first appeared in an Emacs form.

I'm not sure how emacs feels in that sense, nor ZSH's region selection
when it's emulating Emacs. Anyhow, since there are only 2 editors we
emulate, I think Vim should be emulated if `bindkey -v` is set, else
Emacs should be emulated.

> Feel free to lobby on zsh-workers if you want the defaults changed.
> Personally I was happy to just tweak zle_highlight and the terminal
> emulator.

I'd love to lobby there but I'm not sure what defaults can I change.
Given the GIF demonstrations I've made. It seems to me the underlying
code needs a change, and that the defaults are OK.

I'd thank you a lot Oliver if you'll at least verify that with your
version of ZSH and the terminals I tested you get the same behaviour
with a non blinking cursor.

Thanks, Doron.

Re: ZLE Vi-mode: visual mode selection doesn't match the characters highlighted

[Oliver Kiddle]

Doron Behar wrote:
> I'm glad Oliver that you are attending _this_ issue as well but I'd like
> to deal with it at that GitHub thread
> (https://github.com/zsh-users/zsh-syntax-highlighting/issues/637), as I
> don't feel it's related and focus here on the _cursor's_ character's
> highlighting.

Pull request #638 there addresses that issue. I was able to reproduce
that problem once I tried with a double-quoted (and hence highlighted)
string. I had missed that detail before. Would be good if you could give
that some testing - I don't use the plugin.

Seems ugly that they have to reproduce the builtin highlighting. The
region_highlight feature could do with a way to layer entries and to
tell which belong to which plugin.

> > We could perhaps use some hooks for when visual mode is invoked and
> > finished.
>
> Yea I can use `zle-keymap-select` in the meantime...

Except that doesn't trigger for local keymaps like visual.

> > Whatever highlighting is chosen, there will be some terminal
> > setup for which it is not ideal and somebody who complains.
>
> I hope I've demonstrated good enough that this is not a terminal
> emulator issue, it's an issue of a non blinking cursor. Hence, I think
> my complain is legitimate.

Yes, I agree that it is legitimate. But I don't see what you'd change in
the code other than the default colours. The effect as I understand it
is just as I originally described: the cursor reversing the highlighting
to give you back the defaults. I think vim's defaults amount to
zle_highlight=( region:bg=white,fg=black )

Oliver

Re: ZLE Vi-mode: visual mode selection doesn't match the characters highlighted

[Doron Behar]

On Wed, Oct 16, 2019 at 09:37:50PM +0200, Oliver Kiddle wrote:
> Replying off-list because your message was off-list. Feel free to quote
> me from this message if you reply back on-list.
> 
> On 13 Oct, Doron Behar wrote:
> > Thanks for suggesting that. Indeed this is a combination of terminal
> > emulators' settings / differences in behaviour along with a certain
> > plugin I use: https://github.com/zsh-users/zsh-syntax-highlighting
> 
> If I run zsh -df and then enable just that plugin by doing
>   . zsh-syntax-highlighting.zsh
> then I still can't reproduce the problem with highlighting of the t.
> 
> It doesn't appear to create any new keymaps so I'm unsure why you
> don't get the `t' highlighted. Are you loading it with a plugin
> manager. It'd be useful to isolate just the minimum required to
> cause the problem.

Oliver I appreciate your attendance to this issue yet I believe it's not
related to the cursor's character not getting highlighted. I'd like to
focus here on the cursor's character's highlighting and tend to the `t`
not highlighted at the GitHub thread:

https://github.com/zsh-users/zsh-syntax-highlighting/issues/637

I'll emphasize though, that _that_ plugin's issue, making the `t` not
highlighted, appears only when the plugin colors the text selected, as
best explained and demonstrated here:

https://github.com/zsh-users/zsh-syntax-highlighting/issues/637#issuecomment-541623625

> Zsh does highlight the cursor position. It is down to the terminal
> emulator how that appears. What terminal emulator do you use? What is
> $TERM set to and what OS are you on? I use a green solid block as my
> cursor and I know that the cursor position is included in the
> selection.  It appears much the same in vim and gvim.

I'm using NixOS with URxvt. Our urxvt is compiled according to this
"recipe":

https://github.com/NixOS/nixpkgs/blob/7885eeaffdbd02ff6ce5483239fba1a063b314a9/pkgs/applications/misc/rxvt_unicode/default.nix

My `$TERM` using `Urxvt` is `rxvt-unicode-256color`.

I tested Urxvt with a blank .Xresources (I ran `xrdb merge .Xresources`)
and other terminal emulators:

- Gnome Terminal
- Konsole

Using a **non blinking cursor**, **all terminal emulators** have this
issue.

I made a demonstration of the other terminals here:

https://gist.github.com/doronbehar/cba65dbac52b5859e8e967a6bc71ea91#gistcomment-3057815

I dearly hope the issue is clear now.

-------

> You may be able to configure the cursor with escape sequences. For
> example, for a vertical bar (good for vi insert mode):
>   print -n $'\e[5 q'

I tried using an underline cursor and it's a good enough workaround for
now - The character the cursor is located at is highlighted and I can
even see the cursor's underline at the same color of the background
right below that character. I wouldn't consider this as a solution
because I tend to prefer the block cursor.

> We could perhaps use some hooks for when visual mode is invoked and
> finished.

Yea I can use zle-keymap-select to further improve the experience but I
still prefer using a unchanging block cursor.

> > Oliver, do you think perhaps ZSH could 'double reverse' the cursor in
> > visual mode? I think this will emulate Vim's behaviour better because
> > this way, with a non blinking cursor in visual mode, the character the cursor
> > stands upon will standout as well.
> 
> That could make it hard to see which end of the selection the cursor
> is at.

I agree, but the current behavior is even worse: With a non blinking
cursor, it's still impossible to see what end of the selection the
cursor is at. On top of that, the character the cursor is at is not
highlighted at all - misleading me into thinking it's an exclusive
selection.

> Whatever highlighting is chosen, there will be some terminal
> setup for which it is not ideal and somebody who complains.

I tend to disagree, as my GIF demonstrations show that probably all
terminal emulators, with whatever colors and configured to not the
cursor don't highlight characters properly.

> Vim's defaults result in a barely visible selection if you chose a
> near-white default background colour.

I tested the terminal emulators listed above with white or whitish
backgrounds, and the same misleading highlighting behaviour was there,
only the colors were reversed. Again, only with a non blinking cursor.

> Copying Vim might be tempting but Vim is not the only editor - zsh
> uses the term region for the selection in many cases because the
> feature first appeared in an Emacs form.

I don't know how Emacs feels in that sense, nor ZSH when trying to
emulate Emacs. But since ZSH emulates no more then 2 editors - I don't
think it would be too much to emulate Vim better, if `bindkey -v` was
set.

> Feel free to lobby on zsh-workers if you want the defaults changed.
> Personally I was happy to just tweak zle_highlight and the terminal
> emulator.

Thanks for the suggestion, yet I'm not sure what defaults I can change
as it seems to me the underlying code needs a change. I'd thank you a
lot Oliver if you'll merely tell me whether you get a sense of the issue
thanks to the GIF demonstrations. Hoping you'll be able to advice me how
to describe it better for zsh-workers.

Doron.