zsh-users
 help / color / mirror / code / Atom feed
* problem with multi-byte character on command line (zsh v 5.8)
@ 2022-05-18 17:30 Pier Paolo Grassi
  2022-05-18 18:54 ` Bart Schaefer
  2022-05-19  8:59 ` Roman Perepelitsa
  0 siblings, 2 replies; 13+ messages in thread
From: Pier Paolo Grassi @ 2022-05-18 17:30 UTC (permalink / raw)
  To: Zsh-Users List

[-- Attachment #1: Type: text/plain, Size: 460 bytes --]

Hello, I have just found than having this character on command line:
$'\xf0\x9f\x8d\xbe' (that would be a 🍾)
makes the cursor unable to advance when inserting characters (it seems that
each character is overwriting the previously entered one, but when checking
the history the characters where alla added correctly, although in the
wrong position)
Hope that was clear enough, maybe this is already fixed in 5.8.1 or 5.9?
thanks

Pier Paolo Grassi

[-- Attachment #2: Type: text/html, Size: 729 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-18 17:30 problem with multi-byte character on command line (zsh v 5.8) Pier Paolo Grassi
@ 2022-05-18 18:54 ` Bart Schaefer
  2022-05-18 19:40   ` Pier Paolo Grassi
  2022-05-19  8:59 ` Roman Perepelitsa
  1 sibling, 1 reply; 13+ messages in thread
From: Bart Schaefer @ 2022-05-18 18:54 UTC (permalink / raw)
  To: Pier Paolo Grassi; +Cc: Zsh-Users List

On Wed, May 18, 2022 at 10:30 AM Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
>
> Hello, I have just found than having this character on command line:
> $'\xf0\x9f\x8d\xbe' (that would be a 🍾)
> makes the cursor unable to advance when inserting characters

I'm unable to reproduce this in either 5.8 or 5.9.  Is it possibly
specific to a terminal emulator or locale setting?


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-18 18:54 ` Bart Schaefer
@ 2022-05-18 19:40   ` Pier Paolo Grassi
  2022-05-18 19:55     ` Bart Schaefer
  0 siblings, 1 reply; 13+ messages in thread
From: Pier Paolo Grassi @ 2022-05-18 19:40 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Zsh-Users List

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

Hello Bart, I normally use terminator as my terminal emulator. I have tried
launching xterm and the result is the same.
I use zsh 5.8 (x86_64-pc-linux-musl) (the hermetic version provided by
Roman: https://github.com/romkatv/zsh-bin)
which seems to be the culprit, since if I switch to the zsh 5.1.1 provided
by the os of the vm where I am testing the problem does not manifests.
By the way the problem happens only when the cursor is placed before the
4-bytes character.
regards

Pier Paolo Grassi

Il giorno mer 18 mag 2022 alle ore 20:55 Bart Schaefer <
schaefer@brasslantern.com> ha scritto:

> On Wed, May 18, 2022 at 10:30 AM Pier Paolo Grassi <pierpaolog@gmail.com>
> wrote:
> >
> > Hello, I have just found than having this character on command line:
> > $'\xf0\x9f\x8d\xbe' (that would be a 🍾)
> > makes the cursor unable to advance when inserting characters
>
> I'm unable to reproduce this in either 5.8 or 5.9.  Is it possibly
> specific to a terminal emulator or locale setting?
>

[-- Attachment #2: Type: text/html, Size: 1706 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-18 19:40   ` Pier Paolo Grassi
@ 2022-05-18 19:55     ` Bart Schaefer
  2022-05-18 20:10       ` Pier Paolo Grassi
  0 siblings, 1 reply; 13+ messages in thread
From: Bart Schaefer @ 2022-05-18 19:55 UTC (permalink / raw)
  To: Pier Paolo Grassi; +Cc: Zsh-Users List

On Wed, May 18, 2022 at 12:40 PM Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
>
> I use zsh 5.8 (x86_64-pc-linux-musl)

Ahh.  I wonder if this is a side-effect of musl's "special" C locale
(see workers/50235):

>> Starting with version 1.1.11, musl provides a special C locale where bytes
>> 0x80-0xff are treated as abstract single-byte-character units with no actual
>> character identity (they’re mapped into wchar_t values that occupy the
>> Unicode surrogates range).

> By the way the problem happens only when the cursor is placed before the 4-bytes character.

Still not reproducible on Ubuntu 20.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-18 19:55     ` Bart Schaefer
@ 2022-05-18 20:10       ` Pier Paolo Grassi
  2022-05-18 20:12         ` Pier Paolo Grassi
  0 siblings, 1 reply; 13+ messages in thread
From: Pier Paolo Grassi @ 2022-05-18 20:10 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Zsh-Users List

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

Just tried an ubuntu 18.04 with zsh 5.4.2 (x86_64-ubuntu-linux-gnu)
and I am able to reproduce.
I tried also with gnome-terminal, same result.
I have tried en_US.utf8 and it_IT.utf8, can you share yours?
thanks


Pier Paolo Grassi


Il giorno mer 18 mag 2022 alle ore 21:55 Bart Schaefer <
schaefer@brasslantern.com> ha scritto:

> On Wed, May 18, 2022 at 12:40 PM Pier Paolo Grassi <pierpaolog@gmail.com>
> wrote:
> >
> > I use zsh 5.8 (x86_64-pc-linux-musl)
>
> Ahh.  I wonder if this is a side-effect of musl's "special" C locale
> (see workers/50235):
>
> >> Starting with version 1.1.11, musl provides a special C locale where
> bytes
> >> 0x80-0xff are treated as abstract single-byte-character units with no
> actual
> >> character identity (they’re mapped into wchar_t values that occupy the
> >> Unicode surrogates range).
>
> > By the way the problem happens only when the cursor is placed before the
> 4-bytes character.
>
> Still not reproducible on Ubuntu 20.
>

[-- Attachment #2: Type: text/html, Size: 1676 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-18 20:10       ` Pier Paolo Grassi
@ 2022-05-18 20:12         ` Pier Paolo Grassi
  2022-05-19  4:05           ` Bart Schaefer
  0 siblings, 1 reply; 13+ messages in thread
From: Pier Paolo Grassi @ 2022-05-18 20:12 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Zsh-Users List

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

> I have tried en_US.utf8 and it_IT.utf8, can you share yours?
sorry, should have been:

For the locale I have tried en_US.utf8 and it_IT.utf8, can you share yours?

Pier Paolo Grassi


Il giorno mer 18 mag 2022 alle ore 22:10 Pier Paolo Grassi <
pierpaolog@gmail.com> ha scritto:

> Just tried an ubuntu 18.04 with zsh 5.4.2 (x86_64-ubuntu-linux-gnu)
> and I am able to reproduce.
> I tried also with gnome-terminal, same result.
> I have tried en_US.utf8 and it_IT.utf8, can you share yours?
> thanks
>
>
> Pier Paolo Grassi
>
>
> Il giorno mer 18 mag 2022 alle ore 21:55 Bart Schaefer <
> schaefer@brasslantern.com> ha scritto:
>
>> On Wed, May 18, 2022 at 12:40 PM Pier Paolo Grassi <pierpaolog@gmail.com>
>> wrote:
>> >
>> > I use zsh 5.8 (x86_64-pc-linux-musl)
>>
>> Ahh.  I wonder if this is a side-effect of musl's "special" C locale
>> (see workers/50235):
>>
>> >> Starting with version 1.1.11, musl provides a special C locale where
>> bytes
>> >> 0x80-0xff are treated as abstract single-byte-character units with no
>> actual
>> >> character identity (they’re mapped into wchar_t values that occupy the
>> >> Unicode surrogates range).
>>
>> > By the way the problem happens only when the cursor is placed before
>> the 4-bytes character.
>>
>> Still not reproducible on Ubuntu 20.
>>
>

[-- Attachment #2: Type: text/html, Size: 2438 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-18 20:12         ` Pier Paolo Grassi
@ 2022-05-19  4:05           ` Bart Schaefer
  0 siblings, 0 replies; 13+ messages in thread
From: Bart Schaefer @ 2022-05-19  4:05 UTC (permalink / raw)
  To: Pier Paolo Grassi; +Cc: Zsh-Users List

On Wed, May 18, 2022 at 1:11 PM Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
>
> Just tried an ubuntu 18.04 with zsh 5.4.2 (x86_64-ubuntu-linux-gnu)
> and I am able to reproduce.

At least that rules out musl libc.

On Wed, May 18, 2022 at 1:12 PM Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
>
> For the locale I have tried en_US.utf8 and it_IT.utf8, can you share yours?

I'm using LANG=en_US.utf8 ... I don't have Italian locales installed.
I have not tried changing the locale variables other than LANG, should
I?  Or perhaps you need to provide more specific instructions.

I've tried Gnome terminal and xterm.  xterm is not able to display the
glyph at all (I just get a 2-position-wide box) but I am able to move
the cursor back and forth past it and insert single-byte characters
anywhere around it.

This is with the stock Ubuntu zsh 5.8 and with 5.9 built from git
clone.  I've also tried inserting the character into the line editor
both with copy-paste from your original email and with `print -z
$'\xf0\x9f\x8d\xbe'`


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-18 17:30 problem with multi-byte character on command line (zsh v 5.8) Pier Paolo Grassi
  2022-05-18 18:54 ` Bart Schaefer
@ 2022-05-19  8:59 ` Roman Perepelitsa
  2022-05-19 14:01   ` Pier Paolo Grassi
  1 sibling, 1 reply; 13+ messages in thread
From: Roman Perepelitsa @ 2022-05-19  8:59 UTC (permalink / raw)
  To: Pier Paolo Grassi; +Cc: Zsh-Users List

On Wed, May 18, 2022 at 7:31 PM Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
>
> Hello, I have just found than having this character on command line:
> $'\xf0\x9f\x8d\xbe' (that would be a 🍾)
> makes the cursor unable to advance when inserting characters

Can you try running `cat`, pasting 🍾 and then typing something after
it? Does this also cause cursor issues for you?

Roman.

P.S.

I cannot reproduce the issue you've described on Ubuntu 21.10 with zsh
5.8, LC_ALL=C.UTF-8 and xterm or gnome-terminal.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-19  8:59 ` Roman Perepelitsa
@ 2022-05-19 14:01   ` Pier Paolo Grassi
  2022-05-19 14:11     ` Roman Perepelitsa
  0 siblings, 1 reply; 13+ messages in thread
From: Pier Paolo Grassi @ 2022-05-19 14:01 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh-Users List

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]

Hi Roman, just to clarify, the problem happens only when the cursor appears
before the character.
I tried your suggestion and there were no issues

Bart: this is the output of locale:
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=it_IT.UTF-8
LC_TIME=it_IT.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=it_IT.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=it_IT.UTF-8
LC_NAME=it_IT.UTF-8
LC_ADDRESS=it_IT.UTF-8
LC_TELEPHONE=it_IT.UTF-8
LC_MEASUREMENT=it_IT.UTF-8
LC_IDENTIFICATION=it_IT.UTF-8
LC_ALL=

I tried with
LC_ALL=C.UTF-8 zsh
locale
sed: -e expression #1, char 18: Invalid collation character
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=C.UTF-8

the problem still appears

Pier Paolo Grassi


Il giorno gio 19 mag 2022 alle ore 11:00 Roman Perepelitsa <
roman.perepelitsa@gmail.com> ha scritto:

> On Wed, May 18, 2022 at 7:31 PM Pier Paolo Grassi <pierpaolog@gmail.com>
> wrote:
> >
> > Hello, I have just found than having this character on command line:
> > $'\xf0\x9f\x8d\xbe' (that would be a 🍾)
> > makes the cursor unable to advance when inserting characters
>
> Can you try running `cat`, pasting 🍾 and then typing something after
> it? Does this also cause cursor issues for you?
>
> Roman.
>
> P.S.
>
> I cannot reproduce the issue you've described on Ubuntu 21.10 with zsh
> 5.8, LC_ALL=C.UTF-8 and xterm or gnome-terminal.
>

[-- Attachment #2: Type: text/html, Size: 2600 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-19 14:01   ` Pier Paolo Grassi
@ 2022-05-19 14:11     ` Roman Perepelitsa
  2022-05-19 14:12       ` Pier Paolo Grassi
  2022-05-19 14:23       ` Ray Andrews
  0 siblings, 2 replies; 13+ messages in thread
From: Roman Perepelitsa @ 2022-05-19 14:11 UTC (permalink / raw)
  To: Pier Paolo Grassi; +Cc: Zsh-Users List

On Thu, May 19, 2022 at 4:02 PM Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
>
> Hi Roman, just to clarify, the problem happens only when the cursor appears before the character.

Have you tried it in any program other than zsh that also allows you
to move the cursor?

Roman.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-19 14:11     ` Roman Perepelitsa
@ 2022-05-19 14:12       ` Pier Paolo Grassi
  2022-05-19 14:23       ` Ray Andrews
  1 sibling, 0 replies; 13+ messages in thread
From: Pier Paolo Grassi @ 2022-05-19 14:12 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh-Users List

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

I have now, with mysql command line client. No problem here

Pier Paolo Grassi


Il giorno gio 19 mag 2022 alle ore 16:11 Roman Perepelitsa <
roman.perepelitsa@gmail.com> ha scritto:

> On Thu, May 19, 2022 at 4:02 PM Pier Paolo Grassi <pierpaolog@gmail.com>
> wrote:
> >
> > Hi Roman, just to clarify, the problem happens only when the cursor
> appears before the character.
>
> Have you tried it in any program other than zsh that also allows you
> to move the cursor?
>
> Roman.
>

[-- Attachment #2: Type: text/html, Size: 1067 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-19 14:11     ` Roman Perepelitsa
  2022-05-19 14:12       ` Pier Paolo Grassi
@ 2022-05-19 14:23       ` Ray Andrews
  2022-05-19 14:26         ` Roman Perepelitsa
  1 sibling, 1 reply; 13+ messages in thread
From: Ray Andrews @ 2022-05-19 14:23 UTC (permalink / raw)
  To: zsh-users

On 2022-05-19 07:11, Roman Perepelitsa wrote:
> On Thu, May 19, 2022 at 4:02 PM Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
>> Hi Roman, just to clarify, the problem happens only when the cursor appears before the character.
> Have you tried it in any program other than zsh that also allows you
> to move the cursor?
>
> Roman.
>
Hey Roman, are we going to have a pre-build of the latest?  I'm 
helplessly lazy now :-)



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: problem with multi-byte character on command line (zsh v 5.8)
  2022-05-19 14:23       ` Ray Andrews
@ 2022-05-19 14:26         ` Roman Perepelitsa
  0 siblings, 0 replies; 13+ messages in thread
From: Roman Perepelitsa @ 2022-05-19 14:26 UTC (permalink / raw)
  To: Ray Andrews; +Cc: Zsh Users

On Thu, May 19, 2022 at 4:24 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> Hey Roman, are we going to have a pre-build of the latest?

You mean https://github.com/romkatv/zsh-bin, right? I'll build it
eventually but there is no ETA. Sorry! There are no changes in 5.9
that affect me personally, so updating zsh-bin goes into my altruistic
TODO list, which is rather long.

Roman.


^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2022-05-19 14:28 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-18 17:30 problem with multi-byte character on command line (zsh v 5.8) Pier Paolo Grassi
2022-05-18 18:54 ` Bart Schaefer
2022-05-18 19:40   ` Pier Paolo Grassi
2022-05-18 19:55     ` Bart Schaefer
2022-05-18 20:10       ` Pier Paolo Grassi
2022-05-18 20:12         ` Pier Paolo Grassi
2022-05-19  4:05           ` Bart Schaefer
2022-05-19  8:59 ` Roman Perepelitsa
2022-05-19 14:01   ` Pier Paolo Grassi
2022-05-19 14:11     ` Roman Perepelitsa
2022-05-19 14:12       ` Pier Paolo Grassi
2022-05-19 14:23       ` Ray Andrews
2022-05-19 14:26         ` Roman Perepelitsa

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).