Help establishing wctype character classes

Help establishing wctype character classes

[Rich Felker]

Hi,

I'm looking for some help establishing the proper definitions (in
terms of Unicode) for the different wctype character classes (isw*()
functions) in musl. Some are relatively easy and already done:

- Spaces: Unicode White_Space property, minus non-breaking spaces
  (which lack "space" semantics; that's their whole point) and
  script-specific, non-blank-glyph characters that Unicode oddly calls
  "spaces".

- Control: The traditional C0/C1 control codes, plus some bidi and
  embedding control stuff. (Basically, anything that can't be printed
  literally in plain text and expected to work.)

- Printable: Any valid codepoint that's not control. This broad
  definition is to avoid the situation where an application refuses to
  print valid text in the end user's language because it contains
  characters added to Unicode after the libc/app was published. It's
  assumed that the vast majority of, if not all, new characters added
  to Unicode will be non-control characters and non-problematic to
  print.

- Graphic: Any printable character that's not a space. This definition
  is provided by the C standard.

- Digit: Exactly the ASCII decimal digits '0'...'9' and nothing else.
  This definition is provided by the C standard.

If anyone has objections to the above definitions, I'd be happy to
hear them, but what I'm most concerned about is the remaining classes,
specifically alphabetic and punctuation. Here are some tentative
partial definitions:

- Alphabetic: Derived property Alphabetic from Unicode's
  DerivedCoreProperties.txt. Not yet in musl.

- Punctutation: Currently defined in musl as Unicode character classes
  P* and Sm, plus Pattern_Syntax from PropList.txt.

However, these definitions omit a number of characters which should
probably be in one or the other of these two classes, in particular at
least:

- All Unicode class N* (numeric) characters, like superscripts, vulgar
  fractions, Roman numerals, etc.

- Combining diacritic characters that are not themselves letters.

and I'm unclear on what should be done with them. Especially for
numeric/digit-like characters, they cannot be classified as "digits"
(per ISO C rules), and calling them letters may make more sense; glibc
on the other hand considers them punctuation. As for combining
diacritics, considering them letters if their intended usage is to
combine onto letters would be convenient for handling NFD text with
regular expressions (/[[:alpha:]]+/ will match full words even if the
diacritics are decomposed), but I suspect it will also make some cases
behave worse.

Before anybody suggests just dumping what glibc does and copying it,
glibc is horribly incorrect on a number of characters in ways that
break internationalized usage. For instance, all Tibetan subjoined
letter characters are wrongly classified as punctuation rather than as
letters; this will break usage of regular expressions like
/[[:alpha:]]+/ with affected scripts/languages.

Rich

Re: Help establishing wctype character classes

[Rich Felker]

It seems glibc defines these via localedata/gen-unicode-ctype.c in the
following ways:

- Alphabetic: It has complex special-casing for some particular
  characters based on reported errors in Unicode, but basically it
  amounts to all of categories L*, Nl, Nd, and members of category So
  which have "LETTER" in their name.

- Blank: Tab and all of category Zs without <noBreak>.

- Space: The ASCII space class, plus all of Zs, Zl, and Zp without
  <noBreak>.

- Control: Anything with <control> as its name or category Zl/Zp.

- Graphic: Any non-control, non-space. 

- Printable: Any non-control.

- Punctuation: Any non-alphanumeric graphic. They cite this as "the
  traditional POSIX definition of punctuation", so I'm inclined to
  think they have a good idea here.

Source:
http://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/gen-unicode-ctype.c

Note that wherever I've said "any", it's actually quantified only over
defined characters; thus any characters added to Unicode later than
the version glibc is sync'd with are reported as non-printable but
also non-control. This seems highly undesirable to me; it's the reason
"less" refuses to show new characters and instead prints <U+xxxx>.

I would rather have every valid _codepoint_ be either control or
printable, and all the non-space printable codepoints be graphic, but
then among the graphic codepoints, only define alphanumeric or
punctuation class for those codepoints assigned to characters in
Unicode. That is, the hierarchy would break down as:

1. All valid codepoints are either control or printable.
2. All printable codepoints are either ASCII space or graphic.
3. All graphic codepoints are either assigned or unassigned.
4. All assigned graphic codepoints (graphic _characters_) are either
alphanumeric or punctuation.

It seems the only arbitrary decision left for us to make is how to
divide the graphic characters between alphanumeric and punctuation.
And this can be done by an explicit definition for either one, in
terms of which the other will be implicitly defined.

Here's a possible definition for alphanumeric:

- All characters with Unicode Alphabetic property (includes L*, Nl,
  and special cases (Other_Alphabetic) defined in PropList.txt).
- All characters with category Nd (digit).

And possibly also:

- Some or all characters with category No (other numeric - this
  includes things like superscripts, vulgar fractions, and
  script-specific numerical notations that are anything other than a
  direct copy of the ten decimal digits). Note that most of these in
  the Latin blocks are traditionally considered punctuation on Unixy
  systems.
- Some or all characters with category So (other symbol) with LETTER
  in their names (just U+2129 turned Greek small letter iota and a
  bunch of useless circled/parenthesized letters).
- Excluding a few special cases like glibc does (2 Thai characters
  that are actually not letters but punctuation, according to
  Theppitak Karoonboonyanan).

Does this sound like a reasonable plan? Any tweaks needed?

Rich

Re: Help establishing wctype character classes

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

And here's a quick implementation of the iswalnum set generator. At
this point it just generates a dumb lookup table without doing any
multi-level table optimizations or compression, and dumps the table to
stdout in a visual form. I need to run some analysis on the table to
determine the best way to get it down to a reasonable size.

Rich

[-- Attachment #2: gen_ctype_alpha.c --]
[-- Type: text/plain, Size: 1046 bytes --]

#include <stdio.h>

int main()
{
	char *set = calloc(0x110000,1);
	char buf[128], dummy;
	int a, b;
	FILE *f;

	/* Alphabetic property */
	f = fopen("DerivedCoreProperties.txt", "rb");
	while (fgets(buf, sizeof buf, f)) {
		if (sscanf(buf, "%x..%x ; Alphabetic%c", &a, &b, &dummy)==3)
			for (; a<=b; a++) set[a]=1;
		else if (sscanf(buf, "%x ; Alphabetic%c", &a, &dummy)==2)
			set[a] = 1;
	}
	fclose(f);

	/* Plus Nd category */
	f = fopen("UnicodeData.txt", "rb");
	while (fgets(buf, sizeof buf, f)) {
		if (sscanf(buf, "%x;%*[^;];Nd%c", &a, &dummy)==2)
			set[a] = 1;
	}
	fclose(f);

	/* Fix misclassified Thai characters */
	set[0xe2f] = set[0xe46] = 0;

	/* Fill in elided CJK ranges */
	for (a=0x3400; a<=0x4db5; a++) set[a]=1;
	for (a=0x4e00; a<=0x9fcc; a++) set[a]=1;
	for (a=0xac00; a<=0xd7a3; a++) set[a]=1;
	for (a=0x20000; a<=0x2a6d6; a++) set[a]=1;
	for (a=0x2a700; a<=0x2b734; a++) set[a]=1;
	for (a=0x2b740; a<=0x2b81d; a++) set[a]=1;

	for (a=0; a<0x110000; a++) {
		putchar(set[a]?'*':'.');
		if (!(a+1&63)) putchar('\n');
	}
}