Re: Updating Unicode support

[Rich Felker <dalias@libc.org>]

On Mon, Jan 22, 2018 at 05:54:49PM -0800, Eric Pruitt wrote:
> NOTE: When I first started writing this email, I didn't realize musl's
> Unicode property table had recently been updated, but I noticed
> <https://git.musl-libc.org/cgit/musl/commit/?id=c72c1c5> when I was
> looking up commit IDs to cite. I'm leaving most of the verbiage below
> unchanged since I think it adds useful context.

OK. With this in mind, I hope you're also aware that musl's Unicode
tables are all highly optimized for size and (aside from case mapping)
very good speed relative to their size, and are generated mechanically
from the UCD files via some ugly code here:

https://github.com/richfelker/musl-chartable-tools

If someone wants to make local changes or upgrade to newer Unicode
before it's upstream in musl, these tools generally provide the best
way to do it.

> The Unicode property data used by musl has not been updated in quite
> some time, and due to changes introduced in recent publications of the
> Unicode standard, musl's width data is incorrect for many symbols --
> notably emoji.

If you mean that emoji should be considered double-width, I agree with
that in principle, but everything has to *agree* upon widths in order
for them to work. If not, terminal contents just get corrupted when
programs or systems that disagree try to communicate. It would take a
coordinated effort with glibc, third-party libraries, and programs
like screen that ship their own wcwidth-equivalent tables to redefine
them as double-width, and ideally there should probably be some
Unicode recommendation to document the change.

Note that musl assumes all characters that aren't already defined as
control or nonspacing are single-width, except in the extended CJK
planes where it assumes they're double-width, so lack of support for
latest Unicode is only a problem when new nonspacing characters are
added or when wide characters are added outside the CJK planes.

> This can lead to rendering glitches in terminals when
> some applications are not built with musl; for example, my terminal
> emulator is dynamically linked against a version of GNU libc that
> supports Unicode 9 (released June 21, 2016) whereas musl's table was
> lasted updated in 2011 or 2012 (commit 1b0ce9a).

Do you have an example of characters that caused the problem? I'd like
to better understand how it came up. Maybe glibc is already doing
something different than what I think they're doing.

> To resolve this problem, I wrote a drop-in replacement for musl's
> wcwidth(3) implementation that uses utf8proc
> (https://github.com/JuliaLang/utf8proc) as the source of truth. You can
> find the code for this at
> <https://github.com/ericpruitt/static-unix-userland/blob/42cbdbb/utf8proc-wcwidth/utf8proc-wcwidth.c>.
> I am wondering if the musl developers would consider accepting a patch
> that implements optional / configurable support for utf8proc. The
> utf8proc-wcwidth.c file I linked to includes some additional code
> unrelated to musl making it possible to use the file as an LD_PRELOAD
> library. The LD_PRELOAD stuff would **not** be include in the proposed
> patch. I'm also investigating implementing the Unicode Collation
> Algorithm (https://unicode.org/reports/tr10/) for wcscoll(3); would that
> be of interest?

Thanks for pointing out this library -- it looks like something we
might should add to the wiki as a recommended lib, and seems to
implement a lot of Unicode functionality that's otherwise only
available in gigantic bloated libraries like ICU. I'd like to take a
closer look at it when I get time.

Of course it's possible to drop it in to musl's tree locally like you
did as a hack, but this isn't something musl can really do due to both
namespace considerations (wcwidth depending on symbols not in reserved
namespace) and policy about not introducing config switches. But if
the table contents in utf8proc do differ from musl, you can always use
the chartable tools package to generate matching tables to drop into
musl.

Rich