Disable UTF8-escaping diacritics

Disable UTF8-escaping diacritics

[Avid Seeker]

Currently, it is not possible to input Arabic Diacritics in a zsh prompt.

E.g: pasting this in zsh:

$ echo السَّلامُ

Results in

$ echo الس<0651><064e>لام<064f>

While this is often desirable when dealing with some Unicode characters,
especially when dealing with hidden characters like ZWNJ, it is not 
helpful when inputting Arabic text.

See: https://en.wikipedia.org/wiki/Arabic_diacritics for the Unicode entries
and more examples.

Re: Disable UTF8-escaping diacritics

[Bart Schaefer]

On Tue, Feb 20, 2024 at 11:26 AM Avid Seeker <avidseeker7@protonmail.com> wrote:
>
> Currently, it is not possible to input Arabic Diacritics in a zsh prompt.

Just to clarify, this means entering diacritics as part of a command
line, rather than including them in e.g. the PS1 string.

> $ echo الس<0651><064e>لام<064f>

I can reproduce this in the stock zsh 5.7.1 on MacOS Catalina, but
cannot reproduce it with a rebuilt 5.7.1 or later using the current
Xcode tools.

The more recent builds do not get this entirely right -- the cursor
placement is messed up, which leads me to believe that e.g. wcwidth()
is returning wrong values for the Arabic glyphs -- but there is no
conversion of the diacritics to hexadecimal.

Unfortunately there's not much zsh can do about this without help from
libraries that describe the characters being used.  Do you have the
appropriate Arabic locale installed?

https://help.gooddata.com/cloudconnect/manual/locale.html

Is your LANG environment etc. set to the correct value from that list?
 I don't currently have any ar_* locales available with which to test,
which may be the reason for the cursor placement problems I observe.

You might try using "zmodload zsh/langinfo" to investigate further.

Re: Disable UTF8-escaping diacritics

[Mikael Magnusson]

On 2/20/24, Avid Seeker <avidseeker7@protonmail.com> wrote:
> Currently, it is not possible to input Arabic Diacritics in a zsh prompt.
>
> E.g: pasting this in zsh:
>
> $ echo السَّلامُ
>
> Results in
>
> $ echo الس<0651><064e>لام<064f>
>
> While this is often desirable when dealing with some Unicode characters,
> especially when dealing with hidden characters like ZWNJ, it is not
> helpful when inputting Arabic text.
>
> See: https://en.wikipedia.org/wiki/Arabic_diacritics for the Unicode
> entries
> and more examples.

Try with setopt COMBINING_CHARS, but arabic is never going to look
very readable in a cell-based system like a terminal.

-- 
Mikael Magnusson