From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham autolearn_force=no version=3.4.4 Received: from zero.zsh.org (zero.zsh.org [IPv6:2a02:898:31:0:48:4558:7a:7368]) by inbox.vuxu.org (Postfix) with ESMTP id D17BE221F3 for ; Fri, 10 May 2024 11:38:26 +0200 (CEST) ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1715333906; b=BrOzWKJ+IoKjdt5u79gd6t/SzxAVY1maSdlB5TAcNE/UJFG/WE8MVT5wcK5yNobmpF+vOVGWxS heRxyJe+NSlJgiJ4jgy4iKUxeWkI/ztNGZlVMPiVCso8daNx1Najp7eYBNZ19vb4Q0U6XRNFqd hekjVTQVzzw5DczRmcjftsod/7V1ybevmsWFHnqty0JJj7BsTwIGCBcSNN/auO1kzVm6l9oCLe gZWTu7WZf1D1IjdHxe72Cj2AEAYYBH9auVp07ARPgohNlDGbEQBmD8am6yOyYYtcyPKedJg59t sl0DfPOlSDJnONuazNPEE9Nd9kOtiiaWa61vvz7drorQ2Q==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (mail-ej1-f53.google.com) smtp.remote-ip=209.85.218.53; dkim=pass header.d=gmail.com header.s=20230601 header.a=rsa-sha256; dmarc=pass header.from=gmail.com; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1715333906; bh=sI1Q8P/SaAzMsYHzod/JVtlfAiw+wQ1iGGO9KlJRmYw=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:Content-Transfer-Encoding:Content-Type:Cc:To:Subject: Message-ID:Date:From:In-Reply-To:References:MIME-Version:DKIM-Signature: DKIM-Signature; b=diUaUeRjGo36LhCkMw9earSfIuJ33Grx8fpLG0HONUaE2De+Y7wuw7NMKRXJ3Wcm+31gp49Hxo oWAUA3QJrWIausheX/HP/mmcWUOpTmeFEgZrdPiKWFRZ7d0NYD0onAVoas67GBczqWrZFTH9XS d3jDH0WIIbebQd1Rhc+06+dz3ctCQqsgHT9ICF/+vN6XtIJII0dDhg6PPdpzvKBUrXhombGEDH ZDXbA2yQUpLVJjNE/LieLViNBJiSL3z7lnbKUZECeiZ8uy2wRH7Goy/Bp+wL2i6OIes5oHOD50 dxR624KjiLb+Xm9paJaDdiAGUQN5LzxBuPjC9XFBl/Wfiw==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:Content-Transfer-Encoding: Content-Type:Cc:To:Subject:Message-ID:Date:From:In-Reply-To:References: MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=sI1Q8P/SaAzMsYHzod/JVtlfAiw+wQ1iGGO9KlJRmYw=; b=KwAmKiyntqeVitg8htG4UCcMuR qkZFAtSlxnylXqPPmbFc4cONba0W/u8Z8tp6Q7AeaVCAjPGtnUsQjrmwBLTdQrnPm574WYArWHJgW V+RqzZ2Vy3bzeCAI6ZdKcj02PKnE0l8q1fQcJrfVAVk3422PxWXYq3VBK9OIZ8TyvA0I84ZAMNGJ+ zAMg1wzmdtB278JCiDhnr9ie2rGAckdiaKV0l4if06Lc6q/7iPFMZP4bB8Dsd19hEa8x7infLX/wV W5q9FXdOD353oguCMOjOOKY6mO2kNGDq6XMRpJJaHAv28J8AcxVbfHvng1dzxuCYGnMIp9a8CvB7h 4AQYGpyA==; Received: by zero.zsh.org with local id 1s5Mht-0004Iq-KT; Fri, 10 May 2024 09:38:25 +0000 Authentication-Results: zsh.org; iprev=pass (mail-ej1-f53.google.com) smtp.remote-ip=209.85.218.53; dkim=pass header.d=gmail.com header.s=20230601 header.a=rsa-sha256; dmarc=pass header.from=gmail.com; arc=none Received: from mail-ej1-f53.google.com ([209.85.218.53]:58790) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_128_GCM_SHA256:128) id 1s5MhZ-000406-LT; Fri, 10 May 2024 09:38:06 +0000 Received: by mail-ej1-f53.google.com with SMTP id a640c23a62f3a-a5a1054cf61so451499266b.1 for ; Fri, 10 May 2024 02:38:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715333885; x=1715938685; darn=zsh.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=sI1Q8P/SaAzMsYHzod/JVtlfAiw+wQ1iGGO9KlJRmYw=; b=iQewz7DOgld0sJJOBGrm1TSdEsDfdi64ZvlT+2J34o5GBbBcaUh/BxzHD8/plhdW2+ TAuZMmM0mm44fn4NfeFxclngZzzDBBLYRZOERky9eS4NLnyaGQNvx+Vqh9gBJ3wXkT/u VhmGy78+89sxtj5HS6azXSmhMZ4woC0mhA/y/z7PYqhfS2UVfr9ZoxUaA6fQHW0OV7TC V0qABt6iysrflOaea4kHJyXI3nX0Db2oLcACg0h++TWBPTGQRxxJqiGYyDXJJNk+ixQP FU3tJq2rcDt8Ydf3CUw7TA28wJr7DPxQnUmNtRVECHqF4cpxX8riepz2NOC4OAHqOSpz K7AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715333885; x=1715938685; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sI1Q8P/SaAzMsYHzod/JVtlfAiw+wQ1iGGO9KlJRmYw=; b=DEunXeOfKSQw79tFAOFdxvYtIceJWUoLayonPXgqdIt0NYTW7BEKOYugn0TqzZoHpi DDoeqeETLIgpKgejYj97y57unPZm8uVJYNw7atAMMhr6yUG7svXxv6rDBRmGTndZvwEu kmxL1kngbj5nCYMLDMC6UzxqM2xJgTDQP+Ip2etq5Soq43qMNzzOmFF9LLerSDVZakLB cGnsalNU7NL3YBM5228bXAdvsaKQhRgDh3EPE6DrddpiHfeSO7GRsjeEJdVFBmFp7SzM TpbtVBxuO0lxzFzRQeDjTTopGELJAgOYONESe5R05noJj4b1yHPbrW04DE23Zc3meDRq vcGQ== X-Gm-Message-State: AOJu0YyPQDLJSaZ22y/PuUmFNLWYOl6Z4a+clf1e15KSdnJPRhKRGo+7 swO/k9IcFF9TLRkyuZJ+iS/QjAm0UoOibR1Xk8d/Nb9ZK97NjbgD1dqGk2aDtjJDN/sJFEasPnq eCvQm6Z3hUBh5Hj+Y7LPPuaYLlq0= X-Google-Smtp-Source: AGHT+IFq17RtaSJ3KT5KjaWpnBwf5X/x4+GP9G4rFtRGX35sy8ngDGaRL9EwUxfCnJIyLOjJJP/3rUJPUDAU2inKZA8= X-Received: by 2002:a17:906:2b06:b0:a59:a091:ec73 with SMTP id a640c23a62f3a-a5a2d572c25mr198130866b.23.1715333884706; Fri, 10 May 2024 02:38:04 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Mikael Magnusson Date: Fri, 10 May 2024 11:37:50 +0200 Message-ID: Subject: Re: [BUG] ZLE character width with emoji presentation variation selectors in Unicode To: Advait Maybhate Cc: zsh-workers@zsh.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Seq: 52922 Archived-At: X-Loop: zsh-workers@zsh.org Errors-To: zsh-workers-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-workers-request@zsh.org X-no-archive: yes List-Id: List-Help: , List-Subscribe: , List-Unsubscribe: , List-Post: List-Owner: List-Archive: On Thu, May 9, 2024 at 4:46=E2=80=AFPM Advait Maybhate wr= ote: > > Hey folks! > > > Wanted to file a bug report/get a discussion going on the best way to han= dle emoji variation selectors with Unicode characters. > > > Metadata: > > Zsh version: zsh 5.9 (x86_64-apple-darwin23.0), OS version: macOS Sonoma = 14.3.1 > > Terminal: tested across Warp, Kitty, default Mac terminal, Alacritty, iTe= rm 2 > > > ZLE incorrectly treats characters with the emoji variation selector as 1 = character instead of 2 characters, causing off-by-one cursor movement issue= s in terminals that (correctly) treat it as 2 characters. > > > This is most easily reproduced in Kitty (v0.34), which renders and calcul= ates these emojis as 2 cells (most terminal emulators seem to incorrectly h= andle this case of Unicode). > > > To repro: > > Paste in the command =E2=80=9Cecho =E2=98=81=EF=B8=8F=E2=80=9D into Kitty= (the last character is \0x2601 followed by \0xFE0F). Note that this result= s in bracketed paste mode in Zsh. > > > Expected behavior: > > ZLE contains =E2=80=9Cecho =E2=98=81=EF=B8=8F=E2=80=9D. > > > Actual behavior: > > ZLE contains =E2=80=9Ceecho =E2=98=81=EF=B8=8F=E2=80=9D (note the additio= nal =E2=80=9Ce=E2=80=9D at the beginning here - inverted colors from the br= acketed paste). Confirmed that this is due to an off-by-one on the cursor i= nstruction, from the PTY recording. > > > Screenshot: link > > > I=E2=80=99d love to discuss how to fix this for terminals that do respect= variation selectors. One way to do this could be via a new `terminfo` entr= y, but I=E2=80=99d love to know what ZSH devs think! I=E2=80=99m an enginee= r building the Warp terminal, so I=E2=80=99d be happy to work on any termin= al-side changes of this with `terminfo` (we actually use bracketed paste mo= de for all commands, to best support multiline commands with Warp's input e= ditor)! > > > Notably, Fish 3.6 seems to calculate the width correctly as 2 cells (this= is what originally prompted my investigation, due to the Starship prompt -= see fish-shell/issues/10461), along with Bash (using bracketed paste with = Bash 5.2). > > > I=E2=80=99ve seen 2017/msg00432 which is related to this, but deals with = 0xFE0E not 0xFE0F. Generally speaking it is impossible to handle combining emoji, since the specification allows the rendering to either combine or not combine the glyphs, it is not possible for zsh to know how much space they will take up. Of course, your problem isn't even about combining emoji, but as far as I can see the same conceptual problem applies here; there is no way for zsh to know what "render as an image" implies for glyph width, all we can do is call wcwidth. I took a quick look at some unicode emoji standards pages and none of them even mention the word width. If you can find an authorative part of the standard talking about emoji width, feel free to link it... In my terminal your example renders as 1 glyph wide which agrees with zsh's guess, and I don't get any display errors. --=20 Mikael Magnusson