From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/12378 Path: news.gmane.org!.POSTED!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: Updating Unicode support Date: Tue, 23 Jan 2018 18:38:57 -0500 Message-ID: <20180123233857.GW1627@brightrain.aerifal.cx> References: <20180123015446.vera7ocpvgaqvkss@sinister.lan.codevat.com> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: blaine.gmane.org 1516750640 30142 195.159.176.226 (23 Jan 2018 23:37:20 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 23 Jan 2018 23:37:20 +0000 (UTC) User-Agent: Mutt/1.5.21 (2010-09-15) To: musl@lists.openwall.com Original-X-From: musl-return-12394-gllmg-musl=m.gmane.org@lists.openwall.com Wed Jan 24 00:37:16 2018 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1ee87q-0007Cr-RA for gllmg-musl@m.gmane.org; Wed, 24 Jan 2018 00:37:10 +0100 Original-Received: (qmail 22063 invoked by uid 550); 23 Jan 2018 23:39:11 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 22042 invoked from network); 23 Jan 2018 23:39:10 -0000 Content-Disposition: inline In-Reply-To: <20180123015446.vera7ocpvgaqvkss@sinister.lan.codevat.com> Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:12378 Archived-At: On Mon, Jan 22, 2018 at 05:54:49PM -0800, Eric Pruitt wrote: > NOTE: When I first started writing this email, I didn't realize musl's > Unicode property table had recently been updated, but I noticed > when I was > looking up commit IDs to cite. I'm leaving most of the verbiage below > unchanged since I think it adds useful context. OK. With this in mind, I hope you're also aware that musl's Unicode tables are all highly optimized for size and (aside from case mapping) very good speed relative to their size, and are generated mechanically from the UCD files via some ugly code here: https://github.com/richfelker/musl-chartable-tools If someone wants to make local changes or upgrade to newer Unicode before it's upstream in musl, these tools generally provide the best way to do it. > The Unicode property data used by musl has not been updated in quite > some time, and due to changes introduced in recent publications of the > Unicode standard, musl's width data is incorrect for many symbols -- > notably emoji. If you mean that emoji should be considered double-width, I agree with that in principle, but everything has to *agree* upon widths in order for them to work. If not, terminal contents just get corrupted when programs or systems that disagree try to communicate. It would take a coordinated effort with glibc, third-party libraries, and programs like screen that ship their own wcwidth-equivalent tables to redefine them as double-width, and ideally there should probably be some Unicode recommendation to document the change. Note that musl assumes all characters that aren't already defined as control or nonspacing are single-width, except in the extended CJK planes where it assumes they're double-width, so lack of support for latest Unicode is only a problem when new nonspacing characters are added or when wide characters are added outside the CJK planes. > This can lead to rendering glitches in terminals when > some applications are not built with musl; for example, my terminal > emulator is dynamically linked against a version of GNU libc that > supports Unicode 9 (released June 21, 2016) whereas musl's table was > lasted updated in 2011 or 2012 (commit 1b0ce9a). Do you have an example of characters that caused the problem? I'd like to better understand how it came up. Maybe glibc is already doing something different than what I think they're doing. > To resolve this problem, I wrote a drop-in replacement for musl's > wcwidth(3) implementation that uses utf8proc > (https://github.com/JuliaLang/utf8proc) as the source of truth. You can > find the code for this at > . > I am wondering if the musl developers would consider accepting a patch > that implements optional / configurable support for utf8proc. The > utf8proc-wcwidth.c file I linked to includes some additional code > unrelated to musl making it possible to use the file as an LD_PRELOAD > library. The LD_PRELOAD stuff would **not** be include in the proposed > patch. I'm also investigating implementing the Unicode Collation > Algorithm (https://unicode.org/reports/tr10/) for wcscoll(3); would that > be of interest? Thanks for pointing out this library -- it looks like something we might should add to the wiki as a recommended lib, and seems to implement a lot of Unicode functionality that's otherwise only available in gigantic bloated libraries like ICU. I'd like to take a closer look at it when I get time. Of course it's possible to drop it in to musl's tree locally like you did as a hack, but this isn't something musl can really do due to both namespace considerations (wcwidth depending on symbols not in reserved namespace) and policy about not introducing config switches. But if the table contents in utf8proc do differ from musl, you can always use the chartable tools package to generate matching tables to drop into musl. Rich