unicode in prompt breaks command search

unicode in prompt breaks command search

[Nathan Sidwell]

For reasons, I put a multibyte unicode char in $PROMPT, this ends up 
messing with zsh's idea of the current column, as it doesn't understand 
the display width of $PROMPT.

For example reverse search, here's an example using <f0><9f><a1><85>

set PROMPT="🡅"

press C-r, see cursor gets indented another 3 spaces -- we're now at 
column 5, which would be correct if each byte of $PROMPT was a separate 
display char.

A slightly modified example is to enter some text before pressing C-r, 
then the first 3 chars get retained on the display (but not the input 
buffer at least).

Is this supposed to work?

This is ZSH 5.2 on a fedora 30 system, and an XFCE terminal.  Let me 
know if you need more info (I'm not on the list).

nathan
-- 
Nathan Sidwell

Re: unicode in prompt breaks command search

[dana]

On 16 Jan 2020, at 15:43, Nathan Sidwell <nathan@acm.org> wrote:
> For reasons, I put a multibyte unicode char in $PROMPT, this ends up messing
> with zsh's idea of the current column, as it doesn't understand the display
> width of $PROMPT.

You need to tell it what it is with the %{/%} and/or %G escape sequences. For
example, all of these are roughly equivalent and let zsh know that アイウエオ
takes up ten columns:

  PROMPT='%10{アイウエオ%}'
  PROMPT='%{アイウエオ%10G%}'
  PROMPT='%{%2Gア%2Gイ%2Gウ%2Gエ%2Gオ%}'

%{...%} without a leading digit or embedded %G is zero-width (useful for
invisible escape sequences).

http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html#Visual-effects

(I don't think the behaviour of these sequences has changed since 5.2, but you
can check your zshmisc(1) to be sure)

dana

Re: unicode in prompt breaks command search

[Roman Perepelitsa]

Zsh can figure out how many columns Unicode characters take provided
that you configure locale correctly. The most common reason why Zsh
might think that every byte is one character is because the encoding
in the current locale is not UTF-8.

There might be other reasons for the behavior you describe. This entry
from powerlevel10k FAQ can be helpful in diagnosing the problem:
https://github.com/romkatv/powerlevel10k/blob/master/README.md#why-is-my-cursor-in-the-wrong-place.
It's mostly not specific to powerlevel10k.

Roman.

Re: unicode in prompt breaks command search

[Nathan Sidwell]

On 1/16/20 8:25 PM, dana wrote:
> On 16 Jan 2020, at 15:43, Nathan Sidwell <nathan@acm.org> wrote:
>> For reasons, I put a multibyte unicode char in $PROMPT, this ends up messing
>> with zsh's idea of the current column, as it doesn't understand the display
>> width of $PROMPT.
> 
> You need to tell it what it is with the %{/%} and/or %G escape sequences. For
> example, all of these are roughly equivalent and let zsh know that アイウエオ
> takes up ten columns:
> 
>    PROMPT='%10{アイウエオ%}'
>    PROMPT='%{アイウエオ%10G%}'
>    PROMPT='%{%2Gア%2Gイ%2Gウ%2Gエ%2Gオ%}'
> 
> %{...%} without a leading digit or embedded %G is zero-width (useful for
> invisible escape sequences).
> 
> http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html#Visual-effects
> 
> (I don't think the behaviour of these sequences has changed since 5.2, but you
> can check your zshmisc(1) to be sure)

On 1/17/20 3:23 AM, Roman Perepelitsa wrote:
> Zsh can figure out how many columns Unicode characters take provided
> that you configure locale correctly. The most common reason why Zsh
> might think that every byte is one character is because the encoding
> in the current locale is not UTF-8.

thanks for the pointers!

nathan
-- 
Nathan Sidwell