Re: [9front] [PATCH] introduce code points above BMP to /lib/unicode

[Jacob Moody <moody@mail.posixcafe.org>]

Some additional discussion I'd like to get some input on is if we
should just include the entirety of UnicodeData.txt.  There are some
fields in there, notably decomposition mappings, that would be quite
useful.  It would also be nice to generate the ranges used in things
like runetype(2) from the upstream documents so that we can more
easily keep up to date.

On this topic, I have been considering what should be done about
compositional runes in general, as we currently do nothing with them.
For some quick background, these are runes typically used for
diacritic or tonal markings(but not exclusively) in unicode that are
meant to be combined with another base rune.  For various reasons many
combinations have specific precomposed runes they map to.  Currently
our fonts support only these precomposed variants.

One way we could get better is to put in some unicode normalization,
specifically I am looking at NFC, in someplace like libdraw.  Checking
for normalization is cheap, and fixing up strings under the hood would
be an easy way to make (better) use of the bitmaps in our fonts
already.  NFC canonically decomposes then recomposes the runes to
consistently fully precompose the string before handing it off
to the fonts.

It is worth pointing out also that we can't precompose everything,
there are ranges in unicode where you have no option but to implement
shaping yourself.  This won't address those, and would be nice to not
get in the way of that down the road.  Realistically this would allow
us to support a large majority of decomposed latin, decomposed Korean,
and some other decomposed edge cases that do provide precomposed
variants.

This matters if keyboard maps provide these combinational runes, which
as I understand it is not uncommon.  With this change, the
combinational runes would essentially become zero width codepoints to
the perspective of libdraw users.  Which means backspacing (without
any changes) would require two(or more) hits to fully strike out the
rune, progressively unwinding the modifications.  This makes sense to
me, but I cant make assumptions about how others use these runes.

A bit of a ramble, but I wanted to write out what I've been thinking
so someone else can pick it apart if they'd like.

Thanks,
moody