Help establishing wctype character classes

[Rich Felker <dalias@aerifal.cx>]

Hi,

I'm looking for some help establishing the proper definitions (in
terms of Unicode) for the different wctype character classes (isw*()
functions) in musl. Some are relatively easy and already done:

- Spaces: Unicode White_Space property, minus non-breaking spaces
  (which lack "space" semantics; that's their whole point) and
  script-specific, non-blank-glyph characters that Unicode oddly calls
  "spaces".

- Control: The traditional C0/C1 control codes, plus some bidi and
  embedding control stuff. (Basically, anything that can't be printed
  literally in plain text and expected to work.)

- Printable: Any valid codepoint that's not control. This broad
  definition is to avoid the situation where an application refuses to
  print valid text in the end user's language because it contains
  characters added to Unicode after the libc/app was published. It's
  assumed that the vast majority of, if not all, new characters added
  to Unicode will be non-control characters and non-problematic to
  print.

- Graphic: Any printable character that's not a space. This definition
  is provided by the C standard.

- Digit: Exactly the ASCII decimal digits '0'...'9' and nothing else.
  This definition is provided by the C standard.

If anyone has objections to the above definitions, I'd be happy to
hear them, but what I'm most concerned about is the remaining classes,
specifically alphabetic and punctuation. Here are some tentative
partial definitions:

- Alphabetic: Derived property Alphabetic from Unicode's
  DerivedCoreProperties.txt. Not yet in musl.

- Punctutation: Currently defined in musl as Unicode character classes
  P* and Sm, plus Pattern_Syntax from PropList.txt.

However, these definitions omit a number of characters which should
probably be in one or the other of these two classes, in particular at
least:

- All Unicode class N* (numeric) characters, like superscripts, vulgar
  fractions, Roman numerals, etc.

- Combining diacritic characters that are not themselves letters.

and I'm unclear on what should be done with them. Especially for
numeric/digit-like characters, they cannot be classified as "digits"
(per ISO C rules), and calling them letters may make more sense; glibc
on the other hand considers them punctuation. As for combining
diacritics, considering them letters if their intended usage is to
combine onto letters would be convenient for handling NFD text with
regular expressions (/[[:alpha:]]+/ will match full words even if the
diacritics are decomposed), but I suspect it will also make some cases
behave worse.

Before anybody suggests just dumping what glibc does and copying it,
glibc is horribly incorrect on a number of characters in ways that
break internationalized usage. For instance, all Tibetan subjoined
letter characters are wrongly classified as punctuation rather than as
letters; this will break usage of regular expressions like
/[[:alpha:]]+/ with affected scripts/languages.

Rich