* [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47%
@ 2026-01-23 13:20 Xan Phung
2026-01-23 13:40 ` Xan Phung
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Xan Phung @ 2026-01-23 13:20 UTC (permalink / raw)
To: musl; +Cc: xan.phung
Currently iswalpha and iswpunct have total text data size of over 8kb.
A more efficient encoding has reduced the total size to 4.2kb, a 47%
reduction.
This encoding also features a speed improvement, with lookups for
approx 220 of the 256 BMP code blocks done with just a single memory
load.
The remaining Unicode blocks are niche uses, and are accessed with
three memory loads. Compared to the old musl encoding (which used a
two level table) the topmost level remains much the same, providing
offsets into code block units (256 codepoint granularity). The second
level data uses fixed sizes, of one 32 bit word per codepage (where
each 2 bit pair in word identifies a block of 16 codepoints as all 0,
all 1, or mixed). The third level is a variable length series of
extension bytes, indexed by the popcount of set high bits within the
second level's 32 bit word. Popcount is used as not all 16 codepoint
blocks need an extension byte. This popcount is calculated with
nearly same latency as a 32 bit multiply (so it is comparable with
the indexing speed of accessing a 2D array of non power of 2 size),
but has the advantage of greater compactness for sparse data.
Results have been tested against first 0x20000 codepoints, and match
that returned by the pre-existing musl implementation.
A new char table tool to replace the existing gen_ctype is provided
separately. The new tool has similar command line use:
gen_ctype [a|p] -- generates iswalpha & iswpunct table.h files
gen_ctype [A|P] -- generates iswalpha & iswpunct dict.h files
Signed-off-by: Xan Phung <xan.phung@gmail.com>
---
src/ctype/alpha.h | 172 -----------------------------
src/ctype/iswalpha.c | 62 +++++++++--
src/ctype/iswalpha_dict.h | 15 +++
src/ctype/iswalpha_table.h | 217 +++++++++++++++++++++++++++++++++++++
src/ctype/iswpunct.c | 60 ++++++++--
src/ctype/iswpunct_dict.h | 19 ++++
src/ctype/iswpunct_table.h | 211 ++++++++++++++++++++++++++++++++++++
src/ctype/punct.h | 141 ------------------------
8 files changed, 568 insertions(+), 329 deletions(-)
delete mode 100644 src/ctype/alpha.h
create mode 100644 src/ctype/iswalpha_dict.h
create mode 100644 src/ctype/iswalpha_table.h
create mode 100644 src/ctype/iswpunct_dict.h
create mode 100644 src/ctype/iswpunct_table.h
delete mode 100644 src/ctype/punct.h
diff --git a/src/ctype/alpha.h b/src/ctype/alpha.h
deleted file mode 100644
index 4167f38..0000000
--- a/src/ctype/alpha.h
+++ /dev/null
@@ -1,172 +0,0 @@
-18,17,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,17,34,35,36,17,37,38,39,40,
-41,42,43,44,17,45,46,47,16,16,48,16,16,16,16,16,16,16,49,50,51,16,52,53,16,16,
-17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,54,
-17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
-17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
-17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
-17,17,17,55,17,17,17,17,56,17,57,58,59,60,61,62,17,17,17,17,17,17,17,17,17,17,
-17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
-17,17,17,17,17,17,17,63,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,64,65,17,66,67,
-68,69,70,71,72,73,74,17,75,76,77,78,79,80,81,16,82,83,84,85,86,87,88,89,90,91,
-92,93,16,94,95,96,16,17,17,17,97,98,99,16,16,16,16,16,16,16,16,16,16,17,17,17,
-17,100,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,17,101,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,17,17,102,103,16,16,104,105,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
-17,17,17,17,17,17,17,17,17,106,17,17,107,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,
-108,109,16,16,16,16,16,16,16,16,16,110,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,111,112,113,114,16,16,16,16,16,16,16,16,115,116,
-117,16,16,16,16,16,118,119,16,16,16,16,120,16,16,121,16,16,16,16,16,16,16,16,
-16,16,16,16,16,
-16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,254,255,255,7,254,
-255,255,7,0,0,0,0,0,4,32,4,255,255,127,255,255,255,127,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,195,255,3,0,31,80,0,0,0,0,0,0,0,0,0,0,32,0,0,0,0,0,223,188,64,215,255,255,
-251,255,255,255,255,255,255,255,255,255,191,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,3,252,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,254,255,255,255,127,2,255,255,255,
-255,255,1,0,0,0,0,255,191,182,0,255,255,255,135,7,0,0,0,255,7,255,255,255,255,
-255,255,255,254,255,195,255,255,255,255,255,255,255,255,255,255,255,255,239,
-31,254,225,255,
-159,0,0,255,255,255,255,255,255,0,224,255,255,255,255,255,255,255,255,255,255,
-255,255,3,0,255,255,255,255,255,7,48,4,255,255,255,252,255,31,0,0,255,255,255,
-1,255,7,0,0,0,0,0,0,255,255,223,63,0,0,240,255,248,3,255,255,255,255,255,255,
-255,255,255,239,255,223,225,255,207,255,254,255,239,159,249,255,255,253,197,
-227,159,89,128,176,207,255,3,16,238,135,249,255,255,253,109,195,135,25,2,94,
-192,255,63,0,238,191,251,255,255,253,237,227,191,27,1,0,207,255,0,30,238,159,
-249,255,255,253,237,227,159,25,192,176,207,255,2,0,236,199,61,214,24,199,255,
-195,199,29,129,0,192,255,0,0,239,223,253,255,255,253,255,227,223,29,96,7,207,
-255,0,0,239,223,253,255,255,253,239,227,223,29,96,64,207,255,6,0,239,223,253,
-255,255,255,255,231,223,93,240,128,207,255,0,252,236,255,127,252,255,255,251,
-47,127,128,95,255,192,255,12,0,254,255,255,255,255,127,255,7,63,32,255,3,0,0,
-0,0,214,247,255,255,175,255,255,59,95,32,255,243,0,0,0,
-0,1,0,0,0,255,3,0,0,255,254,255,255,255,31,254,255,3,255,255,254,255,255,255,
-31,0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,249,255,3,255,255,255,255,255,
-255,255,255,255,63,255,255,255,255,191,32,255,255,255,255,255,247,255,255,255,
-255,255,255,255,255,255,61,127,61,255,255,255,255,255,61,255,255,255,255,61,
-127,61,255,127,255,255,255,255,255,255,255,61,255,255,255,255,255,255,255,255,
-7,0,0,0,0,255,255,0,0,255,255,255,255,255,255,255,255,255,255,63,63,254,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,159,255,255,254,255,255,7,255,255,255,255,255,255,255,255,
-255,199,255,1,255,223,15,0,255,255,15,0,255,255,15,0,255,223,13,0,255,255,255,
-255,255,255,207,255,255,1,128,16,255,3,0,0,0,0,255,3,255,255,255,255,255,255,
-255,255,255,255,255,1,255,255,255,255,255,7,255,255,255,255,255,255,255,255,
-63,
-0,255,255,255,127,255,15,255,1,192,255,255,255,255,63,31,0,255,255,255,255,
-255,15,255,255,255,3,255,3,0,0,0,0,255,255,255,15,255,255,255,255,255,255,255,
-127,254,255,31,0,255,3,255,3,128,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
-255,239,255,239,15,255,3,0,0,0,0,255,255,255,255,255,243,255,255,255,255,255,
-255,191,255,3,0,255,255,255,255,255,255,127,0,255,227,255,255,255,255,255,63,
-255,1,255,255,255,255,255,231,0,0,0,0,0,222,111,4,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,
-128,255,31,0,255,255,63,63,255,255,255,255,63,63,255,170,255,255,255,63,255,
-255,255,255,255,255,223,95,220,31,207,15,255,31,220,31,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,2,128,0,0,255,31,0,0,0,0,0,0,0,0,0,0,0,0,132,252,47,62,80,189,255,243,
-224,67,0,0,255,255,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,192,255,255,255,255,255,255,3,0,
-0,255,255,255,255,255,127,255,255,255,255,255,127,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,31,120,12,0,255,255,255,255,191,32,255,
-255,255,255,255,255,255,128,0,0,255,255,127,0,127,127,127,127,127,127,127,127,
-255,255,255,255,0,0,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,224,0,0,0,254,3,62,31,254,255,255,255,255,255,255,255,255,255,127,224,254,
-255,255,255,255,255,255,255,255,255,255,247,224,255,255,255,255,255,254,255,
-255,255,255,255,255,255,255,255,255,127,0,0,255,255,255,7,0,0,0,0,0,0,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,63,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,
-0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,31,0,0,
-0,0,0,0,0,0,255,255,255,255,255,63,255,31,255,255,255,15,0,0,255,255,255,255,
-255,127,240,143,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,
-0,128,255,252,255,255,255,255,255,255,255,255,255,255,255,255,249,255,255,255,
-255,255,255,124,0,0,0,0,0,128,255,191,255,255,255,255,0,0,0,255,255,255,255,
-255,255,15,0,255,255,255,255,255,255,255,255,47,0,255,3,0,0,252,232,255,255,
-255,255,255,7,255,255,255,255,7,0,255,255,255,31,255,255,255,255,255,255,247,
-255,0,128,255,3,255,255,255,127,255,255,255,255,255,255,127,0,255,63,255,3,
-255,255,127,252,255,255,255,255,255,255,255,127,5,0,0,56,255,255,60,0,126,126,
-126,0,127,127,255,255,255,255,255,247,255,0,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,7,255,3,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,15,0,255,255,127,248,255,255,255,255,
-255,
-15,255,255,255,255,255,255,255,255,255,255,255,255,255,63,255,255,255,255,255,
-255,255,255,255,255,255,255,255,3,0,0,0,0,127,0,248,224,255,253,127,95,219,
-255,255,255,255,255,255,255,255,255,255,255,255,255,3,0,0,0,248,255,255,255,
-255,255,255,255,255,255,255,255,255,63,0,0,255,255,255,255,255,255,255,255,
-252,255,255,255,255,255,255,0,0,0,0,0,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,223,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,31,0,0,255,3,
-254,255,255,7,254,255,255,7,192,255,255,255,255,255,255,255,255,255,255,127,
-252,252,252,28,0,0,0,0,255,239,255,255,127,255,255,183,255,63,255,63,0,0,0,0,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,7,0,0,0,0,0,0,0,0,
-255,255,255,255,255,255,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,255,255,255,31,255,255,255,255,255,255,1,0,0,0,0,
-0,255,255,255,255,0,224,255,255,255,7,255,255,255,255,255,7,255,255,255,63,
-255,255,255,255,15,255,62,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,63,255,3,255,255,255,255,15,255,255,255,
-255,15,255,255,255,255,255,0,255,255,255,255,255,255,15,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,0,255,255,63,0,255,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,63,253,255,255,255,255,191,145,255,255,63,0,255,255,
-127,0,255,255,255,127,0,0,0,0,0,0,0,0,255,255,55,0,255,255,63,0,255,255,255,3,
-0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,192,0,0,0,0,0,0,0,0,111,240,239,
-254,255,255,63,0,0,0,0,0,255,255,255,31,255,255,255,31,0,0,0,0,255,254,255,
-255,31,0,0,0,255,255,255,255,255,255,63,0,255,255,63,0,255,255,7,0,255,255,3,
-0,0,0,0,0,0,0,0,0,0,0,0,
-0,255,255,255,255,255,255,255,255,255,1,0,0,0,0,0,0,255,255,255,255,255,255,7,
-0,255,255,255,255,255,255,7,0,255,255,255,255,255,0,255,3,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,31,128,0,255,255,63,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,255,255,127,0,255,255,255,255,255,255,255,255,63,0,0,0,
-192,255,0,0,252,255,255,255,255,255,255,1,0,0,255,255,255,1,255,3,255,255,255,
-255,255,255,199,255,112,0,255,255,255,255,71,0,255,255,255,255,255,255,255,
-255,30,0,255,23,0,0,0,0,255,255,251,255,255,255,159,64,0,0,0,0,0,0,0,0,127,
-189,255,191,255,1,255,255,255,255,255,255,255,1,255,3,239,159,249,255,255,253,
-237,227,159,25,129,224,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,
-255,255,255,255,255,187,7,255,131,0,0,0,0,255,255,255,255,255,255,255,255,179,
-0,255,3,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,63,127,0,0,0,63,0,0,
-0,0,255,255,255,255,255,255,255,127,17,0,255,3,0,0,0,0,255,255,255,255,255,
-255,63,1,255,3,0,0,0,0,0,0,255,255,255,231,255,7,255,3,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,
-0,255,255,255,255,255,255,255,255,255,3,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,255,252,255,255,255,255,255,252,26,0,0,0,255,255,255,255,255,255,231,
-127,0,0,255,255,255,255,255,255,255,255,255,32,0,0,0,0,255,255,255,255,255,
-255,255,1,255,253,255,255,255,255,127,127,1,0,255,3,0,0,252,255,255,255,252,
-255,255,254,127,0,0,0,0,0,0,0,0,0,127,251,255,255,255,255,127,180,203,0,255,3,
-191,253,255,255,255,127,123,1,255,3,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,127,0,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,3,0,0,
-0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,255,255,127,0,
-0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
-255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
-255,255,255,255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,
-255,255,255,255,255,255,1,255,255,255,127,255,3,0,0,0,0,0,0,0,0,0,0,0,0,255,
-255,255,63,0,0,255,255,255,255,255,255,0,0,15,0,255,3,248,255,255,224,255,255,
-0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,255,255,255,255,255,255,255,255,255,135,255,255,255,255,255,255,255,128,
-255,255,0,0,0,0,0,0,0,0,11,0,0,0,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,7,0,255,255,255,127,0,0,0,0,0,
-0,7,0,240,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,15,255,255,255,255,
-255,255,255,255,255,255,255,255,255,7,255,31,255,1,255,67,0,0,0,0,0,0,0,0,0,0,
-0,0,255,255,255,255,255,255,255,255,255,255,223,255,255,255,255,255,255,255,
-255,223,100,222,255,235,239,255,255,255,255,255,255,
-255,191,231,223,223,255,255,255,123,95,252,253,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,63,255,255,255,
-253,255,255,247,255,255,255,247,255,255,223,255,255,255,223,255,255,127,255,
-255,255,127,255,255,255,253,255,255,255,253,255,255,247,207,255,255,255,255,
-255,255,127,255,255,249,219,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,255,255,255,255,255,31,128,63,255,67,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
-15,255,3,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,31,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,
-143,8,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,239,255,255,255,150,254,247,10,132,234,150,170,150,247,247,94,255,251,255,
-15,238,251,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,3,255,255,255,3,255,
-255,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
diff --git a/src/ctype/iswalpha.c b/src/ctype/iswalpha.c
index 1c5485d..d92c2e3 100644
--- a/src/ctype/iswalpha.c
+++ b/src/ctype/iswalpha.c
@@ -1,16 +1,60 @@
#include <wctype.h>
+#include <endian.h>
-static const unsigned char table[] = {
-#include "alpha.h"
+#define BYTE(x,y) ((x) >> (y^(__BYTE_ORDER == __BIG_ENDIAN ? 0:24)) & 255)
+#define BYTES(x) BYTE(x,24),BYTE(x,16),BYTE(x,8),BYTE(x,0)
+#define PAGE_SH 8
+#define PAGE_MAX (1u << PAGE_SH)
+#define PAGES (0x20000 / PAGE_MAX)
+#define PAGEH (0x1000/8 + PAGES) /* Direct map 0x1000 codepts */
+
+const static union {
+ unsigned char b[PAGEH + 184*4];
+ unsigned int w[PAGEH/4 + 184];
+} table = {{
+#include "iswalpha_table.h"
+}};
+
+const static unsigned short dict[87] = {
+#include "iswalpha_dict.h"
};
-int iswalpha(wint_t wc)
-{
- if (wc<0x20000U)
- return (table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1;
- if (wc<0x2fffeU)
- return 1;
- return 0;
+int iswalpha(wint_t wc) {
+ unsigned direct, page, bmap, shfr, lane, base, rev;
+ unsigned target, huff, type, popc, fast;
+ signed char ext;
+ if ((unsigned)wc >= 0x20000)
+ return (unsigned)wc < 0x2fffe;
+
+ /* Direct path used in 220 of the 256 BMP code pages */
+ direct = wc < (PAGEH-PAGES)*8;
+ page = (wc >> PAGE_SH);
+ bmap = (wc >> 3) + PAGES;
+ base = table.b[direct ? bmap:page];
+ if (base <= direct*256 + 1)
+ return base >> (wc & -direct & 7) & 1;
+
+ /* 2nd & 3rd level arrays: final level idx=popc^rev_direction */
+ target = wc & (PAGE_MAX-1);
+ shfr = (target & 15);
+ lane = (target >> 4);
+ huff = table.w[base += PAGEH/4 - 2]; /* base 0/1 reserved */
+ type = (huff >> (2 * lane)) & 3;
+ popc = (huff << (31 - 2 * lane));
+ popc = (popc & 0x11111111) + ((popc & 0x44444444) >> 2);
+ popc = (popc * 0x11111111) >> 28;
+ base+= (rev = -(page & 1)) + 1;
+ ext = (table.b + base*4)[(int)(popc^rev)];
+
+ /* Dictionary lookup is only 1% of codepoints */
+ fast = (type != 2 | ext & 1);
+ if (fast) {
+ shfr = (shfr + 5) & -(type >= 2);
+ shfr = (shfr - (6 & -(type == 2)) + type);
+ return (ext << 8 | 0xfe) >> shfr & 1;
+ } else {
+ return dict[ext >> 1 & 0x7f] >> shfr & 1;
+ }
}
int __iswalpha_l(wint_t c, locale_t l)
diff --git a/src/ctype/iswalpha_dict.h b/src/ctype/iswalpha_dict.h
new file mode 100644
index 0000000..ba3f1e0
--- /dev/null
+++ b/src/ctype/iswalpha_dict.h
@@ -0,0 +1,15 @@
+/* success: all odd pages matched to even < 0x1e2 */
+/* success: all odd pages matched to even < 0x1e8 */
+/* success: all odd pages matched to even < 0x1ee */
+/* wctype: dictionary 87 entries */
+63871,8383,15743,32573,65341,65407,16191,40959,
+4224,32767,128,4079,56832,1135,43775,24543,
+8156,4047,32770,64644,15919,48464,17376,30751,
+33023,32639,32768,224,1022,7998,57471,36848,
+59644,64639,14336,32382,63615,57592,24447,64764,
+7420,47103,57344,65295,64831,37311,61551,65263,
+16543,48511,49151,40943,58349,6559,57473,1979,
+33791,179,32575,16128,319,32743,64383,46207,
+203,64959,379,34815,240,17407,56932,59327,
+57311,31743,64607,65343,53239,2011,16256,2191,
+65174,2807,60036,43670,63382,24311,64494,
diff --git a/src/ctype/iswalpha_table.h b/src/ctype/iswalpha_table.h
new file mode 100644
index 0000000..3018209
--- /dev/null
+++ b/src/ctype/iswalpha_table.h
@@ -0,0 +1,217 @@
+/* success: all odd pages matched to even < 0x1e2 */
+/* success: all odd pages matched to even < 0x1e8 */
+/* success: all odd pages matched to even < 0x1ee */
+/* wctype: table 512 x 256 codepoints */
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 2, 1, 6, 5, 12, 1, 16, 11, 21, 30, 25, 40, 31, 43, 1, 36,
+ 37, 58, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 44, 15, 47, 0,
+ 51, 24, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 54, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 46,
+ 1, 1, 1, 1, 55, 1, 59, 94, 63, 20, 68, 50, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1,121, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 73,136, 1, 62, 76,100,
+ 79,103, 83, 67, 86, 75, 1,139, 90,142, 95, 72,101, 78, 0, 89,
+104,108,109,146,113, 82,118,150,122,153,125, 0,128,117,133, 0,
+ 1, 1, 1,112,137,124, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 1, 1, 1, 1,140, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 1, 1,143, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,147,132, 0, 0,151,156,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1,127, 1, 1,154, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 1,159,157, 0, 0, 0, 0, 0, 0, 0, 0, 0,160, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0,165,176,169,164, 0, 0, 0, 0, 0, 0, 0, 0,
+173,168,177, 0, 0, 0, 0, 0,179, 85, 0, 0, 0, 0,181, 0,
+ 0,172, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,254,255,255, 7,254,255,255, 7,
+ 0, 0, 0, 0, 0, 4, 32, 4,255,255,127,255,255,255,127,255,
+255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
+255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
+255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
+255,255,255,255,255,255,255,255,195,255, 3, 0, 31, 80, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0,223,188,
+ 64,215,255,255,251,255,255,255,255,255,255,255,255,255,191,255,
+255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
+ 3,252,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
+255,255,255,255,255,255,254,255,255,255,127, 2,255,255,255,255,
+255, 1, 0, 0, 0, 0,255,191,182, 0,255,255,255,135, 7, 0,
+ 0, 0,255, 7,255,255,255,255,255,255,255,254,255,195,255,255,
+255,255,255,255,255,255,255,255,255,255,239, 31,254,225,255,159,
+ 0, 0,255,255,255,255,255,255, 0,224,255,255,255,255,255,255,
+255,255,255,255,255,255, 3, 0,255,255,255,255,255, 7, 48, 4,
+255,255,255,252,255, 31, 0, 0,255,255,255, 1,255, 7, 0, 0,
+ 0, 0, 0, 0,255,255,223, 63, 0, 0,240,255,248, 3,255,255,
+255,255,255,255,255,255,255,239,255,223,225,255,207,255,254,255,
+239,159,249,255,255,253,197,227,159, 89,128,176,207,255, 3, 16,
+238,135,249,255,255,253,109,195,135, 25, 2, 94,192,255, 63, 0,
+238,191,251,255,255,253,237,227,191, 27, 1, 0,207,255, 0, 30,
+238,159,249,255,255,253,237,227,159, 25,192,176,207,255, 2, 0,
+236,199, 61,214, 24,199,255,195,199, 29,129, 0,192,255, 0, 0,
+239,223,253,255,255,253,255,227,223, 29, 96, 7,207,255, 0, 0,
+239,223,253,255,255,253,239,227,223, 29, 96, 64,207,255, 6, 0,
+239,223,253,255,255,255,255,231,223, 93,240,128,207,255, 0,252,
+236,255,127,252,255,255,251, 47,127,128, 95,255,192,255, 12, 0,
+254,255,255,255,255,127,255, 7, 63, 32,255, 3, 0, 0, 0, 0,
+214,247,255,255,175,255,255, 59, 95, 32,255,243, 0, 0, 0, 0,
+ 1, 0, 0, 0,255, 3, 0, 0,255,254,255,255,255, 31,254,255,
+ 3,255,255,254,255,255,255, 31, 0, 0, 0, 0, 0, 0, 0, 0,
+
+/* wctype: pages U+01000,01300 [words -1-2] */
+BYTES(0x96595695),0,7,127,2,239,12,15,8,BYTES(0x95510959),
+
+/* wctype: pages U+01200,01700 [words 3-8] */
+BYTES(0x5a965a55),123,4,123,6,8,10,7,16,3,207,13,191,
+15,15,15,191,BYTES(0x2ad5edde),
+
+/* wctype: pages U+01400,02d00 [words 9-12] */
+BYTES(0x55555557),254,50,50,50,50,127,48,2,BYTES(0x5aad2565),
+
+/* wctype: pages U+01600,0a900 [words 13-17] */
+BYTES(0xa55b6555),14,254,15,143,3,18,7,52,247,63,7,15,
+BYTES(0x9ad59d65),
+
+/* wctype: pages U+01800,03100 [words 18-21] */
+BYTES(0xd5659558),7,3,15,63,15,18,254,224,BYTES(0x409255d7),
+
+/* wctype: pages U+01a00,01900 [words 22-27] */
+BYTES(0x002af959),31,18,254,31,7,7,20,7,7,31,31,127,
+192,3,31,18,BYTES(0x0a65e7a9),
+
+/* wctype: pages U+01c00,01f00 [words 28-33] */
+BYTES(0xa09696d5),127,199,127,3,207,24,26,32,63,34,32,30,
+127,28,12,12,BYTES(0xaa959a59),
+
+/* wctype: pages U+02000,01b00 [words 34-37] */
+BYTES(0x00088000),36,63,3,191,231,7,22,239,BYTES(0xf5650ad5),
+
+/* wctype: pages U+02400,01d00 [words 38-40] */
+BYTES(0x25c00000),192,7,31,128,BYTES(0xf0555555),
+
+/* wctype: pages U+02c00,09f00 [words 41-43] */
+BYTES(0xe5555965),18,18,46,12,BYTES(0x15555555),
+
+/* wctype: pages U+02e00,0ab00 [words 44-47] */
+BYTES(0x00000020),52,7,15,1,239,50,126,70,BYTES(0xa555696e),
+
+/* wctype: pages U+03000,04d00 [words 48-51] */
+BYTES(0x957957a2),54,56,58,254,60,254,239,63,BYTES(0x00d55555),
+
+/* wctype: pages U+0a400,02100 [words 52-55] */
+BYTES(0x94025555),63,127,3,44,231,42,40,38,BYTES(0x000252aa),
+
+/* wctype: pages U+0a600,0fd00 [words 56-59] */
+BYTES(0x1555a526),63,31,18,62,31,1,252,127,BYTES(0x825d5495),
+
+/* wctype: pages U+0a800,10300 [words 60-64] */
+BYTES(0x8b55d527),191,1,15,47,7,64,62,86,127,15,15,84,
+BYTES(0x0e599665),
+
+/* wctype: pages U+0aa00,10b00 [words 65-69] */
+BYTES(0xdb959ad5),127,127,7,66,18,5,68,60,3,7,63,63,
+BYTES(0x000dddd5),
+
+/* wctype: pages U+0fa00,10500 [words 70-72] */
+BYTES(0x09556555),127,7,15,1,BYTES(0x00003565),
+
+/* wctype: pages U+0fe00,10d00 [words 73-75] */
+BYTES(0x9555c000),223,63,7,1,BYTES(0x000000a5),
+
+/* wctype: pages U+10000,11500 [words 76-79] */
+BYTES(0x95550aa6),223,10,82,127,127,15,118,116,BYTES(0x08950000),
+
+/* wctype: pages U+10200,1e900 [words 80-82] */
+BYTES(0x0d590000),63,1,7,158,BYTES(0x00000a55),
+
+/* wctype: pages U+10400,10f00 [words 83-86] */
+BYTES(0x99695555),127,7,86,31,127,63,20,63,BYTES(0xd0000369),
+
+/* wctype: pages U+10800,0a700 [words 87-91] */
+BYTES(0xd009dd96),88,90,63,127,18,55,0,128,124,243,252,128,
+BYTES(0xc356557c),
+
+/* wctype: pages U+10a00,0ff00 [words 92-97] */
+BYTES(0x360990da),92,94,63,63,63,253,31,80,78,18,192,15,
+254,15,254,7,BYTES(0x0a957bb8),
+
+/* wctype: pages U+10c00,10100 [words 98-100] */
+BYTES(0xd5d50255),3,7,7,31,BYTES(0x0000d500),
+
+/* wctype: pages U+11000,11100 [words 101-105] */
+BYTES(0xa4973355),63,192,252,3,3,7,0,47,30,71,112,199,
+BYTES(0x0b55d7d5),
+
+/* wctype: pages U+11200,12300 [words 106-109] */
+BYTES(0xa56a009d),251,96,98,100,3,3,7,7,BYTES(0x00095555),
+
+/* wctype: pages U+11400,11d00 [words 110-114] */
+BYTES(0x0a550a55),110,112,114,7,7,132,18,130,7,128,126,124,
+BYTES(0x002a6a96),
+
+/* wctype: pages U+11600,0d700 [words 115-118] */
+BYTES(0x02950b95),18,17,7,120,7,31,72,15,BYTES(0x96755555),
+
+/* wctype: pages U+11800,12500 [words 119-121] */
+BYTES(0xa5500095),3,7,52,15,BYTES(0x00000355),
+
+/* wctype: pages U+11a00,18700 [words 122-124] */
+BYTES(0x95095495),122,65,3,1,BYTES(0x95555555),
+
+/* wctype: pages U+11c00,16b00 [words 125-129] */
+BYTES(0x00edcb96),251,50,1,7,252,252,253,127,193,248,7,15,
+BYTES(0x0001bb15),
+
+/* wctype: pages U+11e00,0fb00 [words 130-133] */
+BYTES(0xd0000000),127,248,3,219,76,251,74,127,BYTES(0x5cd557ab),
+
+/* wctype: pages U+12400,10700 [words 134-136] */
+BYTES(0x55552555),18,1,63,127,BYTES(0x00002dd5),
+
+/* wctype: pages U+13400,10900 [words 137-139] */
+BYTES(0x00000025),18,129,7,63,BYTES(0x0095009d),
+
+/* wctype: pages U+14600,11300 [words 140-143] */
+BYTES(0x00000355),127,15,108,106,104,251,249,102,BYTES(0x00003aae),
+
+/* wctype: pages U+16a00,11700 [words 144-147] */
+BYTES(0x24002995),3,18,7,127,0,7,15,207,BYTES(0x000000a9),
+
+/* wctype: pages U+16e00,11900 [words 148-150] */
+BYTES(0x00005500),0,26,249,249,BYTES(0x39600000),
+
+/* wctype: pages U+18a00,16f00 [words 151-153] */
+BYTES(0xd5555555),7,11,48,134,BYTES(0x30065655),
+
+/* wctype: pages U+1b200,1b100 [words 154-156] */
+BYTES(0x95555555),31,136,7,18,BYTES(0x55556c09),
+
+/* wctype: pages U+1bc00,1d700 [words 157-161] */
+BYTES(0x000aa555),15,63,3,138,0,152,251,251,18,18,223,223,
+BYTES(0x566666dd),
+
+/* wctype: pages U+1d400,1e100 [words 162-165] */
+BYTES(0x57a95d55),223,191,140,215,239,138,156,63,BYTES(0x000002a5),
+
+/* wctype: pages U+1d600,1f100 [words 166-169] */
+BYTES(0x9b655555),150,253,239,239,0,7,7,7,BYTES(0x00026640),
+
+/* wctype: pages U+1e000,1d500 [words 170-173] */
+BYTES(0x0000002a),10,243,154,253,148,146,144,142,BYTES(0x55555e9a),
+
+/* wctype: pages U+1e200,1e800 [words 174-176] */
+BYTES(0xa5000000),31,7,0,0,
+/* wctype: pages U+1e800,1ee00 [words 176-178] */
+BYTES(0x03555555),31,0,0,0,
+/* wctype: pages U+1ee00,1ee00 [words 178-183] */
+BYTES(0x00aaaaa7),239,160,162,164,166,168,170,247,31,172,31,0,
+0,0,0,0,
\ No newline at end of file
diff --git a/src/ctype/iswpunct.c b/src/ctype/iswpunct.c
index f0b9ea0..cff793b 100644
--- a/src/ctype/iswpunct.c
+++ b/src/ctype/iswpunct.c
@@ -1,14 +1,60 @@
#include <wctype.h>
+#include <endian.h>
-static const unsigned char table[] = {
-#include "punct.h"
+#define BYTE(x,y) ((x) >> (y^(__BYTE_ORDER == __BIG_ENDIAN ? 0:24)) & 255)
+#define BYTES(x) BYTE(x,24),BYTE(x,16),BYTE(x,8),BYTE(x,0)
+#define PAGE_SH 8
+#define PAGE_MAX (1u << PAGE_SH)
+#define PAGES (0x20000 / PAGE_MAX)
+#define PAGEH (0x800/8 + PAGES) /* Direct map 0x800 codepts */
+
+const static union {
+ unsigned char b[PAGEH + 196*4];
+ unsigned int w[PAGEH/4 + 196];
+} table = {{
+#include "iswpunct_table.h"
+}};
+
+const static unsigned short dict[74] = {
+#include "iswpunct_dict.h"
};
-int iswpunct(wint_t wc)
-{
- if (wc<0x20000U)
- return (table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1;
- return 0;
+int iswpunct(wint_t wc) {
+ unsigned direct, page, bmap, shfr, lane, base, rev;
+ unsigned target, huff, type, popc, fast;
+ signed char ext;
+ if ((unsigned)wc >= 0x20000)
+ return 0;
+
+ /* Direct path used in 212 of the 256 BMP code pages */
+ direct = wc < (PAGEH-PAGES)*8;
+ page = (wc >> PAGE_SH);
+ bmap = (wc >> 3) + PAGES;
+ base = table.b[direct ? bmap:page];
+ if (base <= direct*256 + 1)
+ return base >> (wc & -direct & 7) & 1;
+
+ /* 2nd & 3rd level arrays: final level idx=popc^rev_direction */
+ target = wc & (PAGE_MAX-1);
+ shfr = (target & 15);
+ lane = (target >> 4);
+ huff = table.w[base += PAGEH/4 - 2]; /* base 0/1 reserved */
+ type = (huff >> (2 * lane)) & 3;
+ popc = (huff << (31 - 2 * lane));
+ popc = (popc & 0x11111111) + ((popc & 0x44444444) >> 2);
+ popc = (popc * 0x11111111) >> 28;
+ base+= (rev = -(page & 1)) + 1;
+ ext = (table.b + base*4)[(int)(popc^rev)];
+
+ /* Dictionary lookup is only 1% of codepoints */
+ fast = (type != 2 | (ext & 1) == 0);
+ if (fast) {
+ shfr = (shfr + 6) & -(type >= 2);
+ shfr = (shfr - (8 & -(type == 2)) + (type^1));
+ return (ext << 8 | 0x01) >> shfr & 1;
+ } else {
+ return dict[ext >> 1 & 0x7f] >> shfr & 1;
+ }
}
int __iswpunct_l(wint_t c, locale_t l)
diff --git a/src/ctype/iswpunct_dict.h b/src/ctype/iswpunct_dict.h
new file mode 100644
index 0000000..41da56d
--- /dev/null
+++ b/src/ctype/iswpunct_dict.h
@@ -0,0 +1,19 @@
+/* success: all odd pages matched to even < 0x1e8 */
+/* success: all odd pages matched to even < 0x1ec */
+/* success: all odd pages matched to even < 0x1ee */
+/* success: all odd pages matched to even < 0x1f0 */
+/* success: all odd pages matched to even < 0x1f2 */
+/* success: all odd pages matched to even < 0x1f6 */
+/* success: all odd pages matched to even < 0x1f8 */
+/* success: all odd pages matched to even < 0x1fa */
+/* wctype: dictionary 74 entries */
+32767,19968,64519,28668,57347,253,2047,40960,
+32512,1023,32768,57280,252,57343,1664,8191,
+24576,12159,40928,16255,61452,63616,255,8703,
+912,64480,64767,32753,891,49616,17071,48159,
+65343,34784,65027,32769,64511,4095,65310,64513,
+57537,28687,240,49168,5891,32776,16383,896,
+3967,32639,36800,34560,511,63584,16320,8195,
+16353,16224,8128,63556,26624,192,32792,128,
+384,32771,45056,65151,1904,61439,64635,51175,
+59391,1807,
diff --git a/src/ctype/iswpunct_table.h b/src/ctype/iswpunct_table.h
new file mode 100644
index 0000000..8631fe2
--- /dev/null
+++ b/src/ctype/iswpunct_table.h
@@ -0,0 +1,211 @@
+ 0, 0, 0, 0, 0, 0, 0, 0, 2, 25, 7, 48, 12, 6, 17, 11,
+ 21, 0, 0, 28, 26, 0, 29, 16, 32, 34, 35, 39, 40, 20, 0, 60,
+ 44, 56, 1, 1, 49, 1, 1, 1, 1, 1, 1, 64, 53,117, 57, 70,
+ 61,120, 65, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 68, 0, 71, 43, 75,124, 79, 78, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 83, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 67, 0, 82, 90,133,
+ 0, 86, 94, 89, 0,163, 0, 0, 98, 52,102,167,107, 0,111, 74,
+114, 97,118, 93,121,128,125,137,129,170,134, 0,138,141,142,173,
+ 0, 0, 0, 0,145, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0,148, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,151,106, 0, 0,154,176,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,157, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+161,144,164,147, 0, 0,168,110, 1, 1,171, 0, 0, 0, 0, 0,
+ 0,150,174, 0, 0, 0, 0, 0,177,153, 0, 0,179,156,181, 0,
+183,160,186, 1, 1, 1,189,113,191,101,194, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0,254,255, 0,252, 1, 0, 0,248, 1, 0, 0,120,
+ 0, 0, 0, 0,255,251,223,251, 0, 0,128, 0, 0, 0,128, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 60, 0,252,255,224,175,255,255,
+255,255,255,255,255,255,255,255,223,255,255,255,255,255, 32, 64,
+176, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+252, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,252, 0, 0, 0, 0,
+ 0,230,254,255,255,255, 0, 64, 73, 0, 0, 0, 0, 0, 24, 0,
+255,255, 0,216, 0, 0, 0, 0, 0, 0, 0, 1, 0, 60, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16,224, 1, 30, 0, 96,
+255,191, 0, 0, 0, 0, 0, 0,255, 7, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,248,207,227,
+
+/* wctype: pages U+00800,00d00 [words -1-3] */
+BYTES(0x2c0008a8),6,64,1,3,8,5,16,8,19,17,15,48,
+BYTES(0xc2008a80),
+
+/* wctype: pages U+00a00,00f00 [words 4-8] */
+BYTES(0x8280c280),32,64,64,32,64,9,13,27,128,25,248,254,
+BYTES(0x0a820067),
+
+/* wctype: pages U+00c00,01700 [words 9-13] */
+BYTES(0x0283c203),16,64,128,16,32,64,19,35,252,48,112,16,
+BYTES(0x8ac000cc),
+
+/* wctype: pages U+00e00,01d00 [words 14-17] */
+BYTES(0x02800aa0),21,21,23,24,8,62,51,127,BYTES(0xb5000000),
+
+/* wctype: pages U+01000,00900 [words 18-22] */
+BYTES(0x80080280),29,248,128,16,7,64,32,1,48,30,64,32,
+BYTES(0x8280fe80),
+
+/* wctype: pages U+01400,01300 [words 23-25] */
+BYTES(0x00000003),1,19,31,192,BYTES(0x00089800),
+
+/* wctype: pages U+01600,04d00 [words 26-28] */
+BYTES(0x200b2000),33,1,48,112,BYTES(0x55000000),
+
+/* wctype: pages U+01800,01900 [words 29-31] */
+BYTES(0x00000002),1,136,49,28,BYTES(0x58000380),
+
+/* wctype: pages U+01a00,01b00 [words 32-36] */
+BYTES(0x00a0b008),128,1,37,39,1,41,64,24,31,248,16,16,
+BYTES(0xb0209bc0),
+
+/* wctype: pages U+01c00,0a700 [words 37-40] */
+BYTES(0xa6008080),43,128,45,47,49,12,3,127,BYTES(0x0002003d),
+
+/* wctype: pages U+02000,00b00 [words 41-45] */
+BYTES(0xd452b966),43,53,1,223,55,1,1,13,64,11,64,32,
+BYTES(0x82008280),
+
+/* wctype: pages U+02400,10900 [words 46-49] */
+BYTES(0x60d55235),127,13,63,248,252,96,21,101,BYTES(0x5d800088),
+
+/* wctype: pages U+02c00,02100 [words 50-53] */
+BYTES(0xa0000000),67,69,28,63,24,61,59,57,BYTES(0x555606aa),
+
+/* wctype: pages U+02e00,01f00 [words 54-57] */
+BYTES(0xd5590165),1,73,15,33,192,192,9,15,BYTES(0xaa800000),
+
+/* wctype: pages U+03000,02b00 [words 58-61] */
+BYTES(0x803800a6),77,79,81,60,1,16,65,207,BYTES(0x5559d555),
+
+/* wctype: pages U+03200,0fb00 [words 62-64] */
+BYTES(0x55555559),1,3,252,4,BYTES(0x03c00020),
+
+/* wctype: pages U+0a400,02f00 [words 65-67] */
+BYTES(0x83540000),127,128,75,63,BYTES(0x8d555555),
+
+/* wctype: pages U+0a600,10f00 [words 68-71] */
+BYTES(0x8000a002),192,21,83,45,19,192,127,192,BYTES(0x00000b38),
+
+/* wctype: pages U+0a800,0ab00 [words 72-75] */
+BYTES(0x920080a3),64,30,19,85,87,89,112,16,BYTES(0x20000800),
+
+/* wctype: pages U+0aa00,0fd00 [words 76-79] */
+BYTES(0xcb808800),224,95,21,2,128,67,96,128,BYTES(0x80000080),
+
+/* wctype: pages U+0e000,10100 [words 80-83] */
+BYTES(0x00000003),1,93,1,75,1,224,143,135,BYTES(0x943ac0d7),
+
+/* wctype: pages U+0f800,10300 [words 84-86] */
+BYTES(0x80000000),21,1,21,15,BYTES(0x0c080030),
+
+/* wctype: pages U+0fe00,11300 [words 87-90] */
+BYTES(0x80002d59),19,247,97,21,31,117,64,48,BYTES(0x0000e280),
+
+/* wctype: pages U+10200,11100 [words 91-94] */
+BYTES(0x90000000),75,31,254,208,113,56,15,24,BYTES(0xfa00c3c0),
+
+/* wctype: pages U+10800,1f900 [words 95-98] */
+BYTES(0x8030cc00),128,128,128,240,145,143,141,139,BYTES(0x56659556),
+
+/* wctype: pages U+10a00,16b00 [words 99-103] */
+BYTES(0xe2088a80),103,105,105,192,192,2,107,127,0,3,240,48,
+BYTES(0x00003b40),
+
+/* wctype: pages U+10c00,1d700 [words 104-107] */
+BYTES(0x80000000),248,8,4,4,21,21,32,32,BYTES(0x032222cc),
+
+/* wctype: pages U+10e00,1f700 [words 108-110] */
+BYTES(0x00009000),1,75,105,15,BYTES(0x2955d555),
+
+/* wctype: pages U+11000,02d00 [words 111-114] */
+BYTES(0x0283be00),109,252,63,21,3,252,111,71,BYTES(0x00008000),
+
+/* wctype: pages U+11200,03100 [words 115-117] */
+BYTES(0x20200080),115,4,12,15,BYTES(0x35040000),
+
+/* wctype: pages U+11400,0a900 [words 118-121] */
+BYTES(0x03000a00),119,121,76,128,93,8,91,240,BYTES(0x0ac00820),
+
+/* wctype: pages U+11600,11500 [words 122-125] */
+BYTES(0x00802380),21,14,31,123,0,0,45,21,BYTES(0x09800000),
+
+/* wctype: pages U+11800,0ff00 [words 126-130] */
+BYTES(0xe0000080),28,248,7,96,99,63,240,1,240,1,248,254,
+BYTES(0xa0003bbb),
+
+/* wctype: pages U+11a00,11700 [words 131-134] */
+BYTES(0x00380280),125,45,190,7,0,0,248,16,BYTES(0x000000a0),
+
+/* wctype: pages U+11c00,11d00 [words 135-138] */
+BYTES(0x0000eb80),21,62,248,31,3,0,127,52,BYTES(0x00080300),
+
+/* wctype: pages U+11e00,1d100 [words 139-141] */
+BYTES(0x80000000),129,0,105,135,BYTES(0x25555565),
+
+/* wctype: pages U+12400,1d300 [words 142-144] */
+BYTES(0x0000c000),31,0,105,127,BYTES(0x00009d55),
+
+/* wctype: pages U+13400,1e100 [words 145-147] */
+BYTES(0x00000080),105,0,21,127,BYTES(0x000002c0),
+
+/* wctype: pages U+16a00,1e900 [words 148-150] */
+BYTES(0xc0002000),128,63,128,137,BYTES(0x00000a00),
+
+/* wctype: pages U+16e00,1ed00 [words 151-153] */
+BYTES(0x00090000),13,0,93,254,BYTES(0x00000097),
+
+/* wctype: pages U+1bc00,1f100 [words 154-157] */
+BYTES(0x00380000),133,15,192,31,248,56,248,31,BYTES(0x70262216),
+
+/* wctype: pages U+1d000,10500 [words 158-160] */
+BYTES(0xd5555555),63,0,0,21,BYTES(0x00002000),
+
+/* wctype: pages U+1d200,10b00 [words 161-164] */
+BYTES(0xd0000355),63,15,0,252,60,254,254,252,BYTES(0x00288880),
+
+/* wctype: pages U+1d600,11900 [words 165-167] */
+BYTES(0x8b000000),2,16,16,5,BYTES(0x30000000),
+
+/* wctype: pages U+1da00,11f00 [words 168-170] */
+BYTES(0x003a5555),75,240,254,131,BYTES(0x95000000),
+
+/* wctype: pages U+1e200,16f00 [words 171-173] */
+BYTES(0xa0000000),224,21,0,4,BYTES(0x30000000),
+
+/* wctype: pages U+1e800,1ec00 [words 174-176] */
+BYTES(0x0f000000),128,127,0,0,
+/* wctype: pages U+1ec00,1ee00 [words 176-178] */
+BYTES(0x00d5c000),254,31,0,0,
+/* wctype: pages U+1ee00,1f000 [words 178-180] */
+BYTES(0xc0000000),3,0,0,0,
+/* wctype: pages U+1f000,1f200 [words 180-183] */
+BYTES(0xdfed5565),75,15,1,254,254,254,63,0,
+/* wctype: pages U+1f200,1f600 [words 183-186] */
+BYTES(0x00003e97),7,75,105,3,63,0,0,0,
+/* wctype: pages U+1f600,1f800 [words 186-188] */
+BYTES(0xad555555),63,31,13,0,
+/* wctype: pages U+1f800,1fa00 [words 188-191] */
+BYTES(0x00265a56),75,45,19,45,93,0,0,0,
+/* wctype: pages U+1fa00,1fa00 [words 191-195] */
+BYTES(0x000fad55),15,93,147,7,63,0,0,0,0,0,0,0,
diff --git a/src/ctype/punct.h b/src/ctype/punct.h
deleted file mode 100644
index 6792947..0000000
--- a/src/ctype/punct.h
+++ /dev/null
@@ -1,141 +0,0 @@
-18,16,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,16,16,34,35,16,36,37,38,39,
-40,41,42,43,16,44,45,46,17,17,47,17,17,17,17,17,17,48,49,50,51,52,53,54,55,17,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,56,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,57,16,58,59,60,61,62,63,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,64,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,65,16,16,66,16,67,68,
-69,16,70,71,72,16,73,16,16,74,75,76,77,78,16,79,80,81,82,83,84,85,86,87,88,89,
-90,91,16,92,93,94,95,16,16,16,16,96,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,97,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,98,99,16,16,100,101,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,16,16,16,16,16,102,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
-16,16,16,103,104,105,106,16,16,107,108,17,17,109,16,16,16,16,16,16,110,111,16,
-16,16,16,16,112,113,16,16,114,115,116,16,117,118,119,17,17,17,120,121,122,123,
-124,16,16,16,16,
-16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,254,255,0,252,1,0,0,248,1,
-0,0,120,0,0,0,0,255,251,223,251,0,0,128,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,60,0,252,255,224,175,255,255,255,255,255,255,255,255,
-255,255,223,255,255,255,255,255,32,64,176,0,0,0,0,0,0,0,0,0,0,0,0,0,64,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,252,0,0,0,0,0,230,254,255,255,255,0,64,73,0,0,0,0,0,24,0,255,255,0,216,
-0,0,0,0,0,0,0,1,0,60,0,0,0,0,0,0,0,0,0,0,0,0,16,224,1,30,0,
-96,255,191,0,0,0,0,0,0,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,248,207,
-227,0,0,0,3,0,32,255,127,0,0,0,78,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,0,7,252,0,0,0,
-0,0,0,0,0,0,16,0,32,30,0,48,0,1,0,0,0,0,0,0,0,0,16,0,32,0,0,0,0,252,111,0,0,0,
-0,0,0,0,16,0,32,0,0,0,0,64,0,0,0,0,0,0,0,0,16,0,32,0,0,0,0,3,224,0,0,0,0,0,0,
-0,16,0,32,0,0,0,0,253,0,0,0,0,0,0,0,0,0,0,32,0,0,0,0,255,7,16,0,0,0,0,0,0,0,0,
-32,0,0,0,0,128,255,16,0,0,0,0,0,0,16,0,32,0,0,0,0,0,0,0,0,0,0,0,0,0,24,0,160,
-0,127,0,0,255,3,0,0,0,0,0,0,0,0,0,4,0,0,0,0,16,0,0,0,0,0,0,128,0,128,192,223,
-0,12,0,0,0,0,0,0,0,0,0,0,0,4,0,31,0,0,0,0,0,
-0,254,255,255,255,0,252,255,255,0,0,0,0,0,0,0,0,252,0,0,0,0,0,0,192,255,223,
-255,7,0,0,0,0,0,0,0,0,0,0,128,6,0,252,0,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,0,
-0,0,8,0,0,0,0,0,0,0,0,0,0,0,224,255,255,255,31,0,0,255,3,0,0,0,0,0,0,0,0,0,0,
-0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,96,0,0,1,0,0,24,0,0,0,0,0,0,0,0,0,56,0,0,0,0,16,0,0,0,112,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,48,0,0,254,127,47,0,0,255,3,255,127,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,14,49,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,196,255,255,255,
-255,0,0,0,192,0,0,0,0,0,0,0,0,1,0,224,159,0,0,0,0,127,63,255,127,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,16,0,16,0,0,252,255,255,255,31,0,0,0,0,0,12,0,0,0,0,0,0,64,0,
-12,240,0,0,0,0,0,0,128,248,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,255,0,255,255,
-255,33,144,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,
-127,0,224,251,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,160,3,224,0,224,0,
-224,0,96,128,248,255,255,255,252,255,255,255,255,255,127,223,255,241,127,255,
-127,0,0,255,255,255,255,0,0,255,255,255,255,1,0,123,3,208,193,175,66,0,12,31,
-188,255,255,0,0,0,0,0,14,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,127,0,0,0,255,7,0,0,255,255,255,255,255,255,255,255,255,
-255,63,0,0,0,0,0,0,252,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,207,255,255,255,
-63,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,224,135,3,254,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,
-128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,127,255,255,255,255,0,
-0,0,0,0,0,255,255,255,251,255,255,255,255,255,255,255,255,255,255,15,0,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,63,0,0,0,255,15,30,255,255,255,1,252,193,224,0,0,0,0,
-0,0,0,0,0,0,0,30,1,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-255,255,0,0,0,0,255,255,255,255,15,0,0,0,255,255,255,127,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,
-255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,
-255,255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,0,0,0,
-0,0,0,192,0,224,0,0,0,0,0,0,0,0,0,0,0,128,15,112,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-255,0,255,255,127,0,3,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-64,0,0,0,0,15,255,3,0,0,0,0,0,0,240,0,0,0,0,0,0,0,0,0,16,192,0,0,255,255,3,23,
-0,0,0,0,0,248,0,0,0,0,8,128,0,0,0,0,0,0,0,0,0,0,8,0,255,63,0,192,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,240,0,0,128,3,0,0,0,0,0,0,0,128,2,0,0,192,0,0,67,0,0,0,0,0,
-0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,56,0,
-0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,2,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,252,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,48,255,255,255,3,255,255,255,255,255,255,247,
-255,127,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,254,255,0,252,1,0,0,248,1,0,
-0,248,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,127,127,0,48,135,255,255,255,255,255,
-143,255,0,0,0,0,0,0,224,255,255,127,255,15,1,0,0,0,0,0,255,255,255,255,255,63,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,
-15,0,0,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-128,255,0,0,128,255,0,0,0,0,128,255,0,0,0,0,0,0,0,0,0,248,0,0,192,143,0,0,0,
-128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,48,255,255,252,255,255,255,255,255,0,0,0,0,
-0,0,0,135,255,1,255,1,0,0,0,224,0,0,0,224,0,0,0,0,0,1,0,0,96,248,127,0,0,0,0,
-0,0,0,0,254,0,0,0,255,0,0,0,255,0,0,0,30,0,254,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,0,0,0,0,0,0,0,0,0,0,0,
-0,255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,224,127,0,0,0,192,255,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,192,63,252,255,63,0,0,128,3,0,0,0,0,0,0,254,3,32,0,0,0,0,0,0,0,
-0,0,0,0,0,24,0,15,0,0,0,0,0,56,0,0,0,0,0,0,0,0,0,225,63,0,232,254,255,31,0,0,
-0,0,0,0,0,96,63,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,
-24,0,32,0,0,192,31,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,68,
-248,0,104,0,0,0,0,0,0,0,0,0,0,0,0,76,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,128,255,255,255,0,0,0,0,0,0,0,0,0,0,0,0,128,14,0,0,0,255,
-31,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,8,0,252,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,7,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,24,128,255,0,0,0,0,0,
-0,0,0,0,0,223,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,62,0,0,252,255,31,3,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,52,0,0,0,0,0,0,0,0,0,128,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,128,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
-255,3,
-128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,192,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,63,0,0,0,0,0,0,0,255,255,48,0,0,248,
-3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,
-255,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,176,15,0,0,0,0,0,0,
-0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,63,
-0,255,255,255,255,127,254,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,255,1,0,0,255,255,255,255,255,255,255,255,
-63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,15,0,255,255,255,255,255,255,
-255,255,255,255,127,0,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,8,0,0,0,8,0,0,32,0,0,0,32,0,0,128,
-0,0,0,128,0,0,0,2,0,0,0,2,0,0,8,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,255,255,15,0,248,254,255,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,127,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,240,0,
-128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,255,127,0,0,0,0,0,0,0,
-0,0,0,0,0,0,112,7,0,192,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,254,255,255,255,255,255,255,255,31,0,0,0,0,0,0,0,0,0,254,255,
-255,255,255,255,255,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,255,255,255,255,255,
-15,255,255,255,255,255,255,255,255,255,255,255,255,15,0,255,127,254,255,254,
-255,254,255,255,255,63,0,255,31,255,255,255,255,0,0,0,252,0,0,0,28,0,0,0,252,
-255,255,255,31,0,0,0,0,0,0,192,255,255,255,7,0,255,255,255,255,255,15,255,1,3,
-0,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
-0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,255,63,0,255,31,255,7,255,255,255,255,255,255,255,255,
-255,255,255,255,255,255,15,0,255,255,255,255,255,255,255,255,255,255,255,1,
-255,15,0,0,255,15,255,255,255,255,255,255,255,0,255,3,255,255,255,255,255,0,
-255,255,255,63,0,0,0,0,0,0,0,0,0,0,255,239,255,255,255,255,255,255,255,255,
-255,255,255,255,123,252,255,255,255,255,231,199,255,255,255,231,255,255,255,
-255,255,255,255,255,255,255,255,255,255,255,255,255,15,0,255,63,15,7,7,0,63,0,
-0,0,0,0,0,0,0,0,0,0,0,0,
--
2.51.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47%
2026-01-23 13:20 [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47% Xan Phung
@ 2026-01-23 13:40 ` Xan Phung
2026-01-30 12:39 ` Xan Phung
2026-02-18 1:28 ` Xan Phung
2 siblings, 0 replies; 6+ messages in thread
From: Xan Phung @ 2026-01-23 13:40 UTC (permalink / raw)
To: musl
[-- Attachment #1.1: Type: text/plain, Size: 55102 bytes --]
The commit message in my V2 patch refers to a new gen_ctype tool.
This tool is attached. It is a direct replacement for gen_ctype in
musl-chartable-tools.
The new tool reads the same Unicode text data files as the old gen_ctype
tool.
On Sat, 24 Jan 2026 at 00:21, Xan Phung <xan.phung@gmail.com> wrote:
> Currently iswalpha and iswpunct have total text data size of over 8kb.
> A more efficient encoding has reduced the total size to 4.2kb, a 47%
> reduction.
>
> This encoding also features a speed improvement, with lookups for
> approx 220 of the 256 BMP code blocks done with just a single memory
> load.
>
> The remaining Unicode blocks are niche uses, and are accessed with
> three memory loads. Compared to the old musl encoding (which used a
> two level table) the topmost level remains much the same, providing
> offsets into code block units (256 codepoint granularity). The second
> level data uses fixed sizes, of one 32 bit word per codepage (where
> each 2 bit pair in word identifies a block of 16 codepoints as all 0,
> all 1, or mixed). The third level is a variable length series of
> extension bytes, indexed by the popcount of set high bits within the
> second level's 32 bit word. Popcount is used as not all 16 codepoint
> blocks need an extension byte. This popcount is calculated with
> nearly same latency as a 32 bit multiply (so it is comparable with
> the indexing speed of accessing a 2D array of non power of 2 size),
> but has the advantage of greater compactness for sparse data.
>
> Results have been tested against first 0x20000 codepoints, and match
> that returned by the pre-existing musl implementation.
>
> A new char table tool to replace the existing gen_ctype is provided
> separately. The new tool has similar command line use:
>
> gen_ctype [a|p] -- generates iswalpha & iswpunct table.h files
> gen_ctype [A|P] -- generates iswalpha & iswpunct dict.h files
>
> Signed-off-by: Xan Phung <xan.phung@gmail.com>
> ---
> src/ctype/alpha.h | 172 -----------------------------
> src/ctype/iswalpha.c | 62 +++++++++--
> src/ctype/iswalpha_dict.h | 15 +++
> src/ctype/iswalpha_table.h | 217 +++++++++++++++++++++++++++++++++++++
> src/ctype/iswpunct.c | 60 ++++++++--
> src/ctype/iswpunct_dict.h | 19 ++++
> src/ctype/iswpunct_table.h | 211 ++++++++++++++++++++++++++++++++++++
> src/ctype/punct.h | 141 ------------------------
> 8 files changed, 568 insertions(+), 329 deletions(-)
> delete mode 100644 src/ctype/alpha.h
> create mode 100644 src/ctype/iswalpha_dict.h
> create mode 100644 src/ctype/iswalpha_table.h
> create mode 100644 src/ctype/iswpunct_dict.h
> create mode 100644 src/ctype/iswpunct_table.h
> delete mode 100644 src/ctype/punct.h
>
> diff --git a/src/ctype/alpha.h b/src/ctype/alpha.h
> deleted file mode 100644
> index 4167f38..0000000
> --- a/src/ctype/alpha.h
> +++ /dev/null
> @@ -1,172 +0,0 @@
>
> -18,17,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,17,34,35,36,17,37,38,39,40,
>
> -41,42,43,44,17,45,46,47,16,16,48,16,16,16,16,16,16,16,49,50,51,16,52,53,16,16,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,54,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,55,17,17,17,17,56,17,57,58,59,60,61,62,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,63,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,64,65,17,66,67,
>
> -68,69,70,71,72,73,74,17,75,76,77,78,79,80,81,16,82,83,84,85,86,87,88,89,90,91,
>
> -92,93,16,94,95,96,16,17,17,17,97,98,99,16,16,16,16,16,16,16,16,16,16,17,17,17,
>
> -17,100,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,17,101,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,17,17,102,103,16,16,104,105,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,106,17,17,107,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,
>
> -108,109,16,16,16,16,16,16,16,16,16,110,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,111,112,113,114,16,16,16,16,16,16,16,16,115,116,
>
> -117,16,16,16,16,16,118,119,16,16,16,16,120,16,16,121,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,
>
> -16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,254,255,255,7,254,
>
> -255,255,7,0,0,0,0,0,4,32,4,255,255,127,255,255,255,127,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,195,255,3,0,31,80,0,0,0,0,0,0,0,0,0,0,32,0,0,0,0,0,223,188,64,215,255,255,
>
> -251,255,255,255,255,255,255,255,255,255,191,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,3,252,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,254,255,255,255,127,2,255,255,255,
>
> -255,255,1,0,0,0,0,255,191,182,0,255,255,255,135,7,0,0,0,255,7,255,255,255,255,
>
> -255,255,255,254,255,195,255,255,255,255,255,255,255,255,255,255,255,255,239,
> -31,254,225,255,
>
> -159,0,0,255,255,255,255,255,255,0,224,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,3,0,255,255,255,255,255,7,48,4,255,255,255,252,255,31,0,0,255,255,255,
>
> -1,255,7,0,0,0,0,0,0,255,255,223,63,0,0,240,255,248,3,255,255,255,255,255,255,
>
> -255,255,255,239,255,223,225,255,207,255,254,255,239,159,249,255,255,253,197,
>
> -227,159,89,128,176,207,255,3,16,238,135,249,255,255,253,109,195,135,25,2,94,
>
> -192,255,63,0,238,191,251,255,255,253,237,227,191,27,1,0,207,255,0,30,238,159,
>
> -249,255,255,253,237,227,159,25,192,176,207,255,2,0,236,199,61,214,24,199,255,
>
> -195,199,29,129,0,192,255,0,0,239,223,253,255,255,253,255,227,223,29,96,7,207,
>
> -255,0,0,239,223,253,255,255,253,239,227,223,29,96,64,207,255,6,0,239,223,253,
>
> -255,255,255,255,231,223,93,240,128,207,255,0,252,236,255,127,252,255,255,251,
>
> -47,127,128,95,255,192,255,12,0,254,255,255,255,255,127,255,7,63,32,255,3,0,0,
> -0,0,214,247,255,255,175,255,255,59,95,32,255,243,0,0,0,
>
> -0,1,0,0,0,255,3,0,0,255,254,255,255,255,31,254,255,3,255,255,254,255,255,255,
>
> -31,0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,249,255,3,255,255,255,255,255,
>
> -255,255,255,255,63,255,255,255,255,191,32,255,255,255,255,255,247,255,255,255,
>
> -255,255,255,255,255,255,61,127,61,255,255,255,255,255,61,255,255,255,255,61,
>
> -127,61,255,127,255,255,255,255,255,255,255,61,255,255,255,255,255,255,255,255,
>
> -7,0,0,0,0,255,255,0,0,255,255,255,255,255,255,255,255,255,255,63,63,254,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,159,255,255,254,255,255,7,255,255,255,255,255,255,255,255,
>
> -255,199,255,1,255,223,15,0,255,255,15,0,255,255,15,0,255,223,13,0,255,255,255,
>
> -255,255,255,207,255,255,1,128,16,255,3,0,0,0,0,255,3,255,255,255,255,255,255,
>
> -255,255,255,255,255,1,255,255,255,255,255,7,255,255,255,255,255,255,255,255,
> -63,
>
> -0,255,255,255,127,255,15,255,1,192,255,255,255,255,63,31,0,255,255,255,255,
>
> -255,15,255,255,255,3,255,3,0,0,0,0,255,255,255,15,255,255,255,255,255,255,255,
>
> -127,254,255,31,0,255,3,255,3,128,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
>
> -255,239,255,239,15,255,3,0,0,0,0,255,255,255,255,255,243,255,255,255,255,255,
>
> -255,191,255,3,0,255,255,255,255,255,255,127,0,255,227,255,255,255,255,255,63,
>
> -255,1,255,255,255,255,255,231,0,0,0,0,0,222,111,4,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,
>
> -128,255,31,0,255,255,63,63,255,255,255,255,63,63,255,170,255,255,255,63,255,
>
> -255,255,255,255,255,223,95,220,31,207,15,255,31,220,31,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,2,128,0,0,255,31,0,0,0,0,0,0,0,0,0,0,0,0,132,252,47,62,80,189,255,243,
> -224,67,0,0,255,255,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,192,255,255,255,255,255,255,3,0,
>
> -0,255,255,255,255,255,127,255,255,255,255,255,127,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,31,120,12,0,255,255,255,255,191,32,255,
>
> -255,255,255,255,255,255,128,0,0,255,255,127,0,127,127,127,127,127,127,127,127,
>
> -255,255,255,255,0,0,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,224,0,0,0,254,3,62,31,254,255,255,255,255,255,255,255,255,255,127,224,254,
>
> -255,255,255,255,255,255,255,255,255,255,247,224,255,255,255,255,255,254,255,
>
> -255,255,255,255,255,255,255,255,255,127,0,0,255,255,255,7,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,63,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,31,0,0,
>
> -0,0,0,0,0,0,255,255,255,255,255,63,255,31,255,255,255,15,0,0,255,255,255,255,
>
> -255,127,240,143,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,
>
> -0,128,255,252,255,255,255,255,255,255,255,255,255,255,255,255,249,255,255,255,
>
> -255,255,255,124,0,0,0,0,0,128,255,191,255,255,255,255,0,0,0,255,255,255,255,
>
> -255,255,15,0,255,255,255,255,255,255,255,255,47,0,255,3,0,0,252,232,255,255,
>
> -255,255,255,7,255,255,255,255,7,0,255,255,255,31,255,255,255,255,255,255,247,
>
> -255,0,128,255,3,255,255,255,127,255,255,255,255,255,255,127,0,255,63,255,3,
>
> -255,255,127,252,255,255,255,255,255,255,255,127,5,0,0,56,255,255,60,0,126,126,
>
> -126,0,127,127,255,255,255,255,255,247,255,0,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,7,255,3,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,15,0,255,255,127,248,255,255,255,255,
> -255,
>
> -15,255,255,255,255,255,255,255,255,255,255,255,255,255,63,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,3,0,0,0,0,127,0,248,224,255,253,127,95,219,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,3,0,0,0,248,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,63,0,0,255,255,255,255,255,255,255,255,
>
> -252,255,255,255,255,255,255,0,0,0,0,0,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,223,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,31,0,0,255,3,
>
> -254,255,255,7,254,255,255,7,192,255,255,255,255,255,255,255,255,255,255,127,
>
> -252,252,252,28,0,0,0,0,255,239,255,255,127,255,255,183,255,63,255,63,0,0,0,0,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,7,0,0,0,0,0,0,0,0,
>
> -255,255,255,255,255,255,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,255,255,255,31,255,255,255,255,255,255,1,0,0,0,0,
>
> -0,255,255,255,255,0,224,255,255,255,7,255,255,255,255,255,7,255,255,255,63,
>
> -255,255,255,255,15,255,62,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,63,255,3,255,255,255,255,15,255,255,255,
>
> -255,15,255,255,255,255,255,0,255,255,255,255,255,255,15,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,0,255,255,63,0,255,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,63,253,255,255,255,255,191,145,255,255,63,0,255,255,
>
> -127,0,255,255,255,127,0,0,0,0,0,0,0,0,255,255,55,0,255,255,63,0,255,255,255,3,
>
> -0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,192,0,0,0,0,0,0,0,0,111,240,239,
>
> -254,255,255,63,0,0,0,0,0,255,255,255,31,255,255,255,31,0,0,0,0,255,254,255,
>
> -255,31,0,0,0,255,255,255,255,255,255,63,0,255,255,63,0,255,255,7,0,255,255,3,
> -0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,1,0,0,0,0,0,0,255,255,255,255,255,255,7,
>
> -0,255,255,255,255,255,255,7,0,255,255,255,255,255,0,255,3,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,31,128,0,255,255,63,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,255,255,127,0,255,255,255,255,255,255,255,255,63,0,0,0,
>
> -192,255,0,0,252,255,255,255,255,255,255,1,0,0,255,255,255,1,255,3,255,255,255,
>
> -255,255,255,199,255,112,0,255,255,255,255,71,0,255,255,255,255,255,255,255,
>
> -255,30,0,255,23,0,0,0,0,255,255,251,255,255,255,159,64,0,0,0,0,0,0,0,0,127,
>
> -189,255,191,255,1,255,255,255,255,255,255,255,1,255,3,239,159,249,255,255,253,
>
> -237,227,159,25,129,224,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,
>
> -255,255,255,255,255,187,7,255,131,0,0,0,0,255,255,255,255,255,255,255,255,179,
> -0,255,3,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,63,127,0,0,0,63,0,0,
>
> -0,0,255,255,255,255,255,255,255,127,17,0,255,3,0,0,0,0,255,255,255,255,255,
>
> -255,63,1,255,3,0,0,0,0,0,0,255,255,255,231,255,7,255,3,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,3,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,255,252,255,255,255,255,255,252,26,0,0,0,255,255,255,255,255,255,231,
>
> -127,0,0,255,255,255,255,255,255,255,255,255,32,0,0,0,0,255,255,255,255,255,
>
> -255,255,1,255,253,255,255,255,255,127,127,1,0,255,3,0,0,252,255,255,255,252,
>
> -255,255,254,127,0,0,0,0,0,0,0,0,0,127,251,255,255,255,255,127,180,203,0,255,3,
> -191,253,255,255,255,127,123,1,255,3,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,127,0,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,3,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,255,255,127,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,255,255,255,255,1,255,255,255,127,255,3,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,63,0,0,255,255,255,255,255,255,0,0,15,0,255,3,248,255,255,224,255,255,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,255,255,255,255,135,255,255,255,255,255,255,255,128,
>
> -255,255,0,0,0,0,0,0,0,0,11,0,0,0,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,7,0,255,255,255,127,0,0,0,0,0,
>
> -0,7,0,240,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,15,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,7,255,31,255,1,255,67,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,255,255,255,255,255,223,255,255,255,255,255,255,255,
> -255,223,100,222,255,235,239,255,255,255,255,255,255,
>
> -255,191,231,223,223,255,255,255,123,95,252,253,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,63,255,255,255,
>
> -253,255,255,247,255,255,255,247,255,255,223,255,255,255,223,255,255,127,255,
>
> -255,255,127,255,255,255,253,255,255,255,253,255,255,247,207,255,255,255,255,
>
> -255,255,127,255,255,249,219,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,31,128,63,255,67,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
>
> -15,255,3,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,31,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,
> -143,8,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,239,255,255,255,150,254,247,10,132,234,150,170,150,247,247,94,255,251,255,
>
> -15,238,251,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,3,255,255,255,3,255,
> -255,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> diff --git a/src/ctype/iswalpha.c b/src/ctype/iswalpha.c
> index 1c5485d..d92c2e3 100644
> --- a/src/ctype/iswalpha.c
> +++ b/src/ctype/iswalpha.c
> @@ -1,16 +1,60 @@
> #include <wctype.h>
> +#include <endian.h>
>
> -static const unsigned char table[] = {
> -#include "alpha.h"
> +#define BYTE(x,y) ((x) >> (y^(__BYTE_ORDER == __BIG_ENDIAN ? 0:24)) & 255)
> +#define BYTES(x) BYTE(x,24),BYTE(x,16),BYTE(x,8),BYTE(x,0)
> +#define PAGE_SH 8
> +#define PAGE_MAX (1u << PAGE_SH)
> +#define PAGES (0x20000 / PAGE_MAX)
> +#define PAGEH (0x1000/8 + PAGES) /* Direct map 0x1000 codepts */
> +
> +const static union {
> + unsigned char b[PAGEH + 184*4];
> + unsigned int w[PAGEH/4 + 184];
> +} table = {{
> +#include "iswalpha_table.h"
> +}};
> +
> +const static unsigned short dict[87] = {
> +#include "iswalpha_dict.h"
> };
>
> -int iswalpha(wint_t wc)
> -{
> - if (wc<0x20000U)
> - return (table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1;
> - if (wc<0x2fffeU)
> - return 1;
> - return 0;
> +int iswalpha(wint_t wc) {
> + unsigned direct, page, bmap, shfr, lane, base, rev;
> + unsigned target, huff, type, popc, fast;
> + signed char ext;
> + if ((unsigned)wc >= 0x20000)
> + return (unsigned)wc < 0x2fffe;
> +
> + /* Direct path used in 220 of the 256 BMP code pages */
> + direct = wc < (PAGEH-PAGES)*8;
> + page = (wc >> PAGE_SH);
> + bmap = (wc >> 3) + PAGES;
> + base = table.b[direct ? bmap:page];
> + if (base <= direct*256 + 1)
> + return base >> (wc & -direct & 7) & 1;
> +
> + /* 2nd & 3rd level arrays: final level idx=popc^rev_direction */
> + target = wc & (PAGE_MAX-1);
> + shfr = (target & 15);
> + lane = (target >> 4);
> + huff = table.w[base += PAGEH/4 - 2]; /* base 0/1 reserved */
> + type = (huff >> (2 * lane)) & 3;
> + popc = (huff << (31 - 2 * lane));
> + popc = (popc & 0x11111111) + ((popc & 0x44444444) >> 2);
> + popc = (popc * 0x11111111) >> 28;
> + base+= (rev = -(page & 1)) + 1;
> + ext = (table.b + base*4)[(int)(popc^rev)];
> +
> + /* Dictionary lookup is only 1% of codepoints */
> + fast = (type != 2 | ext & 1);
> + if (fast) {
> + shfr = (shfr + 5) & -(type >= 2);
> + shfr = (shfr - (6 & -(type == 2)) + type);
> + return (ext << 8 | 0xfe) >> shfr & 1;
> + } else {
> + return dict[ext >> 1 & 0x7f] >> shfr & 1;
> + }
> }
>
> int __iswalpha_l(wint_t c, locale_t l)
> diff --git a/src/ctype/iswalpha_dict.h b/src/ctype/iswalpha_dict.h
> new file mode 100644
> index 0000000..ba3f1e0
> --- /dev/null
> +++ b/src/ctype/iswalpha_dict.h
> @@ -0,0 +1,15 @@
> +/* success: all odd pages matched to even < 0x1e2 */
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* wctype: dictionary 87 entries */
> +63871,8383,15743,32573,65341,65407,16191,40959,
> +4224,32767,128,4079,56832,1135,43775,24543,
> +8156,4047,32770,64644,15919,48464,17376,30751,
> +33023,32639,32768,224,1022,7998,57471,36848,
> +59644,64639,14336,32382,63615,57592,24447,64764,
> +7420,47103,57344,65295,64831,37311,61551,65263,
> +16543,48511,49151,40943,58349,6559,57473,1979,
> +33791,179,32575,16128,319,32743,64383,46207,
> +203,64959,379,34815,240,17407,56932,59327,
> +57311,31743,64607,65343,53239,2011,16256,2191,
> +65174,2807,60036,43670,63382,24311,64494,
> diff --git a/src/ctype/iswalpha_table.h b/src/ctype/iswalpha_table.h
> new file mode 100644
> index 0000000..3018209
> --- /dev/null
> +++ b/src/ctype/iswalpha_table.h
> @@ -0,0 +1,217 @@
> +/* success: all odd pages matched to even < 0x1e2 */
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* wctype: table 512 x 256 codepoints */
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 2, 1, 6, 5, 12, 1, 16, 11, 21, 30, 25, 40, 31, 43, 1, 36,
> + 37, 58, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 44, 15, 47, 0,
> + 51, 24, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 54, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 46,
> + 1, 1, 1, 1, 55, 1, 59, 94, 63, 20, 68, 50, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1,121, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 73,136, 1, 62, 76,100,
> + 79,103, 83, 67, 86, 75, 1,139, 90,142, 95, 72,101, 78, 0, 89,
> +104,108,109,146,113, 82,118,150,122,153,125, 0,128,117,133, 0,
> + 1, 1, 1,112,137,124, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 1, 1, 1, 1,140, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 1, 1,143, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,147,132, 0, 0,151,156,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1,127, 1, 1,154, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 1,159,157, 0, 0, 0, 0, 0, 0, 0, 0, 0,160, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,165,176,169,164, 0, 0, 0, 0, 0, 0, 0, 0,
> +173,168,177, 0, 0, 0, 0, 0,179, 85, 0, 0, 0, 0,181, 0,
> + 0,172, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0,254,255,255, 7,254,255,255, 7,
> + 0, 0, 0, 0, 0, 4, 32, 4,255,255,127,255,255,255,127,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,195,255, 3, 0, 31, 80, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0,223,188,
> + 64,215,255,255,251,255,255,255,255,255,255,255,255,255,191,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> + 3,252,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,254,255,255,255,127, 2,255,255,255,255,
> +255, 1, 0, 0, 0, 0,255,191,182, 0,255,255,255,135, 7, 0,
> + 0, 0,255, 7,255,255,255,255,255,255,255,254,255,195,255,255,
> +255,255,255,255,255,255,255,255,255,255,239, 31,254,225,255,159,
> + 0, 0,255,255,255,255,255,255, 0,224,255,255,255,255,255,255,
> +255,255,255,255,255,255, 3, 0,255,255,255,255,255, 7, 48, 4,
> +255,255,255,252,255, 31, 0, 0,255,255,255, 1,255, 7, 0, 0,
> + 0, 0, 0, 0,255,255,223, 63, 0, 0,240,255,248, 3,255,255,
> +255,255,255,255,255,255,255,239,255,223,225,255,207,255,254,255,
> +239,159,249,255,255,253,197,227,159, 89,128,176,207,255, 3, 16,
> +238,135,249,255,255,253,109,195,135, 25, 2, 94,192,255, 63, 0,
> +238,191,251,255,255,253,237,227,191, 27, 1, 0,207,255, 0, 30,
> +238,159,249,255,255,253,237,227,159, 25,192,176,207,255, 2, 0,
> +236,199, 61,214, 24,199,255,195,199, 29,129, 0,192,255, 0, 0,
> +239,223,253,255,255,253,255,227,223, 29, 96, 7,207,255, 0, 0,
> +239,223,253,255,255,253,239,227,223, 29, 96, 64,207,255, 6, 0,
> +239,223,253,255,255,255,255,231,223, 93,240,128,207,255, 0,252,
> +236,255,127,252,255,255,251, 47,127,128, 95,255,192,255, 12, 0,
> +254,255,255,255,255,127,255, 7, 63, 32,255, 3, 0, 0, 0, 0,
> +214,247,255,255,175,255,255, 59, 95, 32,255,243, 0, 0, 0, 0,
> + 1, 0, 0, 0,255, 3, 0, 0,255,254,255,255,255, 31,254,255,
> + 3,255,255,254,255,255,255, 31, 0, 0, 0, 0, 0, 0, 0, 0,
> +
> +/* wctype: pages U+01000,01300 [words -1-2] */
> +BYTES(0x96595695),0,7,127,2,239,12,15,8,BYTES(0x95510959),
> +
> +/* wctype: pages U+01200,01700 [words 3-8] */
> +BYTES(0x5a965a55),123,4,123,6,8,10,7,16,3,207,13,191,
> +15,15,15,191,BYTES(0x2ad5edde),
> +
> +/* wctype: pages U+01400,02d00 [words 9-12] */
> +BYTES(0x55555557),254,50,50,50,50,127,48,2,BYTES(0x5aad2565),
> +
> +/* wctype: pages U+01600,0a900 [words 13-17] */
> +BYTES(0xa55b6555),14,254,15,143,3,18,7,52,247,63,7,15,
> +BYTES(0x9ad59d65),
> +
> +/* wctype: pages U+01800,03100 [words 18-21] */
> +BYTES(0xd5659558),7,3,15,63,15,18,254,224,BYTES(0x409255d7),
> +
> +/* wctype: pages U+01a00,01900 [words 22-27] */
> +BYTES(0x002af959),31,18,254,31,7,7,20,7,7,31,31,127,
> +192,3,31,18,BYTES(0x0a65e7a9),
> +
> +/* wctype: pages U+01c00,01f00 [words 28-33] */
> +BYTES(0xa09696d5),127,199,127,3,207,24,26,32,63,34,32,30,
> +127,28,12,12,BYTES(0xaa959a59),
> +
> +/* wctype: pages U+02000,01b00 [words 34-37] */
> +BYTES(0x00088000),36,63,3,191,231,7,22,239,BYTES(0xf5650ad5),
> +
> +/* wctype: pages U+02400,01d00 [words 38-40] */
> +BYTES(0x25c00000),192,7,31,128,BYTES(0xf0555555),
> +
> +/* wctype: pages U+02c00,09f00 [words 41-43] */
> +BYTES(0xe5555965),18,18,46,12,BYTES(0x15555555),
> +
> +/* wctype: pages U+02e00,0ab00 [words 44-47] */
> +BYTES(0x00000020),52,7,15,1,239,50,126,70,BYTES(0xa555696e),
> +
> +/* wctype: pages U+03000,04d00 [words 48-51] */
> +BYTES(0x957957a2),54,56,58,254,60,254,239,63,BYTES(0x00d55555),
> +
> +/* wctype: pages U+0a400,02100 [words 52-55] */
> +BYTES(0x94025555),63,127,3,44,231,42,40,38,BYTES(0x000252aa),
> +
> +/* wctype: pages U+0a600,0fd00 [words 56-59] */
> +BYTES(0x1555a526),63,31,18,62,31,1,252,127,BYTES(0x825d5495),
> +
> +/* wctype: pages U+0a800,10300 [words 60-64] */
> +BYTES(0x8b55d527),191,1,15,47,7,64,62,86,127,15,15,84,
> +BYTES(0x0e599665),
> +
> +/* wctype: pages U+0aa00,10b00 [words 65-69] */
> +BYTES(0xdb959ad5),127,127,7,66,18,5,68,60,3,7,63,63,
> +BYTES(0x000dddd5),
> +
> +/* wctype: pages U+0fa00,10500 [words 70-72] */
> +BYTES(0x09556555),127,7,15,1,BYTES(0x00003565),
> +
> +/* wctype: pages U+0fe00,10d00 [words 73-75] */
> +BYTES(0x9555c000),223,63,7,1,BYTES(0x000000a5),
> +
> +/* wctype: pages U+10000,11500 [words 76-79] */
> +BYTES(0x95550aa6),223,10,82,127,127,15,118,116,BYTES(0x08950000),
> +
> +/* wctype: pages U+10200,1e900 [words 80-82] */
> +BYTES(0x0d590000),63,1,7,158,BYTES(0x00000a55),
> +
> +/* wctype: pages U+10400,10f00 [words 83-86] */
> +BYTES(0x99695555),127,7,86,31,127,63,20,63,BYTES(0xd0000369),
> +
> +/* wctype: pages U+10800,0a700 [words 87-91] */
> +BYTES(0xd009dd96),88,90,63,127,18,55,0,128,124,243,252,128,
> +BYTES(0xc356557c),
> +
> +/* wctype: pages U+10a00,0ff00 [words 92-97] */
> +BYTES(0x360990da),92,94,63,63,63,253,31,80,78,18,192,15,
> +254,15,254,7,BYTES(0x0a957bb8),
> +
> +/* wctype: pages U+10c00,10100 [words 98-100] */
> +BYTES(0xd5d50255),3,7,7,31,BYTES(0x0000d500),
> +
> +/* wctype: pages U+11000,11100 [words 101-105] */
> +BYTES(0xa4973355),63,192,252,3,3,7,0,47,30,71,112,199,
> +BYTES(0x0b55d7d5),
> +
> +/* wctype: pages U+11200,12300 [words 106-109] */
> +BYTES(0xa56a009d),251,96,98,100,3,3,7,7,BYTES(0x00095555),
> +
> +/* wctype: pages U+11400,11d00 [words 110-114] */
> +BYTES(0x0a550a55),110,112,114,7,7,132,18,130,7,128,126,124,
> +BYTES(0x002a6a96),
> +
> +/* wctype: pages U+11600,0d700 [words 115-118] */
> +BYTES(0x02950b95),18,17,7,120,7,31,72,15,BYTES(0x96755555),
> +
> +/* wctype: pages U+11800,12500 [words 119-121] */
> +BYTES(0xa5500095),3,7,52,15,BYTES(0x00000355),
> +
> +/* wctype: pages U+11a00,18700 [words 122-124] */
> +BYTES(0x95095495),122,65,3,1,BYTES(0x95555555),
> +
> +/* wctype: pages U+11c00,16b00 [words 125-129] */
> +BYTES(0x00edcb96),251,50,1,7,252,252,253,127,193,248,7,15,
> +BYTES(0x0001bb15),
> +
> +/* wctype: pages U+11e00,0fb00 [words 130-133] */
> +BYTES(0xd0000000),127,248,3,219,76,251,74,127,BYTES(0x5cd557ab),
> +
> +/* wctype: pages U+12400,10700 [words 134-136] */
> +BYTES(0x55552555),18,1,63,127,BYTES(0x00002dd5),
> +
> +/* wctype: pages U+13400,10900 [words 137-139] */
> +BYTES(0x00000025),18,129,7,63,BYTES(0x0095009d),
> +
> +/* wctype: pages U+14600,11300 [words 140-143] */
> +BYTES(0x00000355),127,15,108,106,104,251,249,102,BYTES(0x00003aae),
> +
> +/* wctype: pages U+16a00,11700 [words 144-147] */
> +BYTES(0x24002995),3,18,7,127,0,7,15,207,BYTES(0x000000a9),
> +
> +/* wctype: pages U+16e00,11900 [words 148-150] */
> +BYTES(0x00005500),0,26,249,249,BYTES(0x39600000),
> +
> +/* wctype: pages U+18a00,16f00 [words 151-153] */
> +BYTES(0xd5555555),7,11,48,134,BYTES(0x30065655),
> +
> +/* wctype: pages U+1b200,1b100 [words 154-156] */
> +BYTES(0x95555555),31,136,7,18,BYTES(0x55556c09),
> +
> +/* wctype: pages U+1bc00,1d700 [words 157-161] */
> +BYTES(0x000aa555),15,63,3,138,0,152,251,251,18,18,223,223,
> +BYTES(0x566666dd),
> +
> +/* wctype: pages U+1d400,1e100 [words 162-165] */
> +BYTES(0x57a95d55),223,191,140,215,239,138,156,63,BYTES(0x000002a5),
> +
> +/* wctype: pages U+1d600,1f100 [words 166-169] */
> +BYTES(0x9b655555),150,253,239,239,0,7,7,7,BYTES(0x00026640),
> +
> +/* wctype: pages U+1e000,1d500 [words 170-173] */
> +BYTES(0x0000002a),10,243,154,253,148,146,144,142,BYTES(0x55555e9a),
> +
> +/* wctype: pages U+1e200,1e800 [words 174-176] */
> +BYTES(0xa5000000),31,7,0,0,
> +/* wctype: pages U+1e800,1ee00 [words 176-178] */
> +BYTES(0x03555555),31,0,0,0,
> +/* wctype: pages U+1ee00,1ee00 [words 178-183] */
> +BYTES(0x00aaaaa7),239,160,162,164,166,168,170,247,31,172,31,0,
> +0,0,0,0,
> \ No newline at end of file
> diff --git a/src/ctype/iswpunct.c b/src/ctype/iswpunct.c
> index f0b9ea0..cff793b 100644
> --- a/src/ctype/iswpunct.c
> +++ b/src/ctype/iswpunct.c
> @@ -1,14 +1,60 @@
> #include <wctype.h>
> +#include <endian.h>
>
> -static const unsigned char table[] = {
> -#include "punct.h"
> +#define BYTE(x,y) ((x) >> (y^(__BYTE_ORDER == __BIG_ENDIAN ? 0:24)) & 255)
> +#define BYTES(x) BYTE(x,24),BYTE(x,16),BYTE(x,8),BYTE(x,0)
> +#define PAGE_SH 8
> +#define PAGE_MAX (1u << PAGE_SH)
> +#define PAGES (0x20000 / PAGE_MAX)
> +#define PAGEH (0x800/8 + PAGES) /* Direct map 0x800 codepts */
> +
> +const static union {
> + unsigned char b[PAGEH + 196*4];
> + unsigned int w[PAGEH/4 + 196];
> +} table = {{
> +#include "iswpunct_table.h"
> +}};
> +
> +const static unsigned short dict[74] = {
> +#include "iswpunct_dict.h"
> };
>
> -int iswpunct(wint_t wc)
> -{
> - if (wc<0x20000U)
> - return (table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1;
> - return 0;
> +int iswpunct(wint_t wc) {
> + unsigned direct, page, bmap, shfr, lane, base, rev;
> + unsigned target, huff, type, popc, fast;
> + signed char ext;
> + if ((unsigned)wc >= 0x20000)
> + return 0;
> +
> + /* Direct path used in 212 of the 256 BMP code pages */
> + direct = wc < (PAGEH-PAGES)*8;
> + page = (wc >> PAGE_SH);
> + bmap = (wc >> 3) + PAGES;
> + base = table.b[direct ? bmap:page];
> + if (base <= direct*256 + 1)
> + return base >> (wc & -direct & 7) & 1;
> +
> + /* 2nd & 3rd level arrays: final level idx=popc^rev_direction */
> + target = wc & (PAGE_MAX-1);
> + shfr = (target & 15);
> + lane = (target >> 4);
> + huff = table.w[base += PAGEH/4 - 2]; /* base 0/1 reserved */
> + type = (huff >> (2 * lane)) & 3;
> + popc = (huff << (31 - 2 * lane));
> + popc = (popc & 0x11111111) + ((popc & 0x44444444) >> 2);
> + popc = (popc * 0x11111111) >> 28;
> + base+= (rev = -(page & 1)) + 1;
> + ext = (table.b + base*4)[(int)(popc^rev)];
> +
> + /* Dictionary lookup is only 1% of codepoints */
> + fast = (type != 2 | (ext & 1) == 0);
> + if (fast) {
> + shfr = (shfr + 6) & -(type >= 2);
> + shfr = (shfr - (8 & -(type == 2)) + (type^1));
> + return (ext << 8 | 0x01) >> shfr & 1;
> + } else {
> + return dict[ext >> 1 & 0x7f] >> shfr & 1;
> + }
> }
>
> int __iswpunct_l(wint_t c, locale_t l)
> diff --git a/src/ctype/iswpunct_dict.h b/src/ctype/iswpunct_dict.h
> new file mode 100644
> index 0000000..41da56d
> --- /dev/null
> +++ b/src/ctype/iswpunct_dict.h
> @@ -0,0 +1,19 @@
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ec */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* success: all odd pages matched to even < 0x1f0 */
> +/* success: all odd pages matched to even < 0x1f2 */
> +/* success: all odd pages matched to even < 0x1f6 */
> +/* success: all odd pages matched to even < 0x1f8 */
> +/* success: all odd pages matched to even < 0x1fa */
> +/* wctype: dictionary 74 entries */
> +32767,19968,64519,28668,57347,253,2047,40960,
> +32512,1023,32768,57280,252,57343,1664,8191,
> +24576,12159,40928,16255,61452,63616,255,8703,
> +912,64480,64767,32753,891,49616,17071,48159,
> +65343,34784,65027,32769,64511,4095,65310,64513,
> +57537,28687,240,49168,5891,32776,16383,896,
> +3967,32639,36800,34560,511,63584,16320,8195,
> +16353,16224,8128,63556,26624,192,32792,128,
> +384,32771,45056,65151,1904,61439,64635,51175,
> +59391,1807,
> diff --git a/src/ctype/iswpunct_table.h b/src/ctype/iswpunct_table.h
> new file mode 100644
> index 0000000..8631fe2
> --- /dev/null
> +++ b/src/ctype/iswpunct_table.h
> @@ -0,0 +1,211 @@
> + 0, 0, 0, 0, 0, 0, 0, 0, 2, 25, 7, 48, 12, 6, 17, 11,
> + 21, 0, 0, 28, 26, 0, 29, 16, 32, 34, 35, 39, 40, 20, 0, 60,
> + 44, 56, 1, 1, 49, 1, 1, 1, 1, 1, 1, 64, 53,117, 57, 70,
> + 61,120, 65, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 68, 0, 71, 43, 75,124, 79, 78, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 83, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 67, 0, 82, 90,133,
> + 0, 86, 94, 89, 0,163, 0, 0, 98, 52,102,167,107, 0,111, 74,
> +114, 97,118, 93,121,128,125,137,129,170,134, 0,138,141,142,173,
> + 0, 0, 0, 0,145, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,148, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,151,106, 0, 0,154,176,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,157, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> +161,144,164,147, 0, 0,168,110, 1, 1,171, 0, 0, 0, 0, 0,
> + 0,150,174, 0, 0, 0, 0, 0,177,153, 0, 0,179,156,181, 0,
> +183,160,186, 1, 1, 1,189,113,191,101,194, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,254,255, 0,252, 1, 0, 0,248, 1, 0, 0,120,
> + 0, 0, 0, 0,255,251,223,251, 0, 0,128, 0, 0, 0,128, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 60, 0,252,255,224,175,255,255,
> +255,255,255,255,255,255,255,255,223,255,255,255,255,255, 32, 64,
> +176, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> +252, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,252, 0, 0, 0, 0,
> + 0,230,254,255,255,255, 0, 64, 73, 0, 0, 0, 0, 0, 24, 0,
> +255,255, 0,216, 0, 0, 0, 0, 0, 0, 0, 1, 0, 60, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16,224, 1, 30, 0, 96,
> +255,191, 0, 0, 0, 0, 0, 0,255, 7, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,248,207,227,
> +
> +/* wctype: pages U+00800,00d00 [words -1-3] */
> +BYTES(0x2c0008a8),6,64,1,3,8,5,16,8,19,17,15,48,
> +BYTES(0xc2008a80),
> +
> +/* wctype: pages U+00a00,00f00 [words 4-8] */
> +BYTES(0x8280c280),32,64,64,32,64,9,13,27,128,25,248,254,
> +BYTES(0x0a820067),
> +
> +/* wctype: pages U+00c00,01700 [words 9-13] */
> +BYTES(0x0283c203),16,64,128,16,32,64,19,35,252,48,112,16,
> +BYTES(0x8ac000cc),
> +
> +/* wctype: pages U+00e00,01d00 [words 14-17] */
> +BYTES(0x02800aa0),21,21,23,24,8,62,51,127,BYTES(0xb5000000),
> +
> +/* wctype: pages U+01000,00900 [words 18-22] */
> +BYTES(0x80080280),29,248,128,16,7,64,32,1,48,30,64,32,
> +BYTES(0x8280fe80),
> +
> +/* wctype: pages U+01400,01300 [words 23-25] */
> +BYTES(0x00000003),1,19,31,192,BYTES(0x00089800),
> +
> +/* wctype: pages U+01600,04d00 [words 26-28] */
> +BYTES(0x200b2000),33,1,48,112,BYTES(0x55000000),
> +
> +/* wctype: pages U+01800,01900 [words 29-31] */
> +BYTES(0x00000002),1,136,49,28,BYTES(0x58000380),
> +
> +/* wctype: pages U+01a00,01b00 [words 32-36] */
> +BYTES(0x00a0b008),128,1,37,39,1,41,64,24,31,248,16,16,
> +BYTES(0xb0209bc0),
> +
> +/* wctype: pages U+01c00,0a700 [words 37-40] */
> +BYTES(0xa6008080),43,128,45,47,49,12,3,127,BYTES(0x0002003d),
> +
> +/* wctype: pages U+02000,00b00 [words 41-45] */
> +BYTES(0xd452b966),43,53,1,223,55,1,1,13,64,11,64,32,
> +BYTES(0x82008280),
> +
> +/* wctype: pages U+02400,10900 [words 46-49] */
> +BYTES(0x60d55235),127,13,63,248,252,96,21,101,BYTES(0x5d800088),
> +
> +/* wctype: pages U+02c00,02100 [words 50-53] */
> +BYTES(0xa0000000),67,69,28,63,24,61,59,57,BYTES(0x555606aa),
> +
> +/* wctype: pages U+02e00,01f00 [words 54-57] */
> +BYTES(0xd5590165),1,73,15,33,192,192,9,15,BYTES(0xaa800000),
> +
> +/* wctype: pages U+03000,02b00 [words 58-61] */
> +BYTES(0x803800a6),77,79,81,60,1,16,65,207,BYTES(0x5559d555),
> +
> +/* wctype: pages U+03200,0fb00 [words 62-64] */
> +BYTES(0x55555559),1,3,252,4,BYTES(0x03c00020),
> +
> +/* wctype: pages U+0a400,02f00 [words 65-67] */
> +BYTES(0x83540000),127,128,75,63,BYTES(0x8d555555),
> +
> +/* wctype: pages U+0a600,10f00 [words 68-71] */
> +BYTES(0x8000a002),192,21,83,45,19,192,127,192,BYTES(0x00000b38),
> +
> +/* wctype: pages U+0a800,0ab00 [words 72-75] */
> +BYTES(0x920080a3),64,30,19,85,87,89,112,16,BYTES(0x20000800),
> +
> +/* wctype: pages U+0aa00,0fd00 [words 76-79] */
> +BYTES(0xcb808800),224,95,21,2,128,67,96,128,BYTES(0x80000080),
> +
> +/* wctype: pages U+0e000,10100 [words 80-83] */
> +BYTES(0x00000003),1,93,1,75,1,224,143,135,BYTES(0x943ac0d7),
> +
> +/* wctype: pages U+0f800,10300 [words 84-86] */
> +BYTES(0x80000000),21,1,21,15,BYTES(0x0c080030),
> +
> +/* wctype: pages U+0fe00,11300 [words 87-90] */
> +BYTES(0x80002d59),19,247,97,21,31,117,64,48,BYTES(0x0000e280),
> +
> +/* wctype: pages U+10200,11100 [words 91-94] */
> +BYTES(0x90000000),75,31,254,208,113,56,15,24,BYTES(0xfa00c3c0),
> +
> +/* wctype: pages U+10800,1f900 [words 95-98] */
> +BYTES(0x8030cc00),128,128,128,240,145,143,141,139,BYTES(0x56659556),
> +
> +/* wctype: pages U+10a00,16b00 [words 99-103] */
> +BYTES(0xe2088a80),103,105,105,192,192,2,107,127,0,3,240,48,
> +BYTES(0x00003b40),
> +
> +/* wctype: pages U+10c00,1d700 [words 104-107] */
> +BYTES(0x80000000),248,8,4,4,21,21,32,32,BYTES(0x032222cc),
> +
> +/* wctype: pages U+10e00,1f700 [words 108-110] */
> +BYTES(0x00009000),1,75,105,15,BYTES(0x2955d555),
> +
> +/* wctype: pages U+11000,02d00 [words 111-114] */
> +BYTES(0x0283be00),109,252,63,21,3,252,111,71,BYTES(0x00008000),
> +
> +/* wctype: pages U+11200,03100 [words 115-117] */
> +BYTES(0x20200080),115,4,12,15,BYTES(0x35040000),
> +
> +/* wctype: pages U+11400,0a900 [words 118-121] */
> +BYTES(0x03000a00),119,121,76,128,93,8,91,240,BYTES(0x0ac00820),
> +
> +/* wctype: pages U+11600,11500 [words 122-125] */
> +BYTES(0x00802380),21,14,31,123,0,0,45,21,BYTES(0x09800000),
> +
> +/* wctype: pages U+11800,0ff00 [words 126-130] */
> +BYTES(0xe0000080),28,248,7,96,99,63,240,1,240,1,248,254,
> +BYTES(0xa0003bbb),
> +
> +/* wctype: pages U+11a00,11700 [words 131-134] */
> +BYTES(0x00380280),125,45,190,7,0,0,248,16,BYTES(0x000000a0),
> +
> +/* wctype: pages U+11c00,11d00 [words 135-138] */
> +BYTES(0x0000eb80),21,62,248,31,3,0,127,52,BYTES(0x00080300),
> +
> +/* wctype: pages U+11e00,1d100 [words 139-141] */
> +BYTES(0x80000000),129,0,105,135,BYTES(0x25555565),
> +
> +/* wctype: pages U+12400,1d300 [words 142-144] */
> +BYTES(0x0000c000),31,0,105,127,BYTES(0x00009d55),
> +
> +/* wctype: pages U+13400,1e100 [words 145-147] */
> +BYTES(0x00000080),105,0,21,127,BYTES(0x000002c0),
> +
> +/* wctype: pages U+16a00,1e900 [words 148-150] */
> +BYTES(0xc0002000),128,63,128,137,BYTES(0x00000a00),
> +
> +/* wctype: pages U+16e00,1ed00 [words 151-153] */
> +BYTES(0x00090000),13,0,93,254,BYTES(0x00000097),
> +
> +/* wctype: pages U+1bc00,1f100 [words 154-157] */
> +BYTES(0x00380000),133,15,192,31,248,56,248,31,BYTES(0x70262216),
> +
> +/* wctype: pages U+1d000,10500 [words 158-160] */
> +BYTES(0xd5555555),63,0,0,21,BYTES(0x00002000),
> +
> +/* wctype: pages U+1d200,10b00 [words 161-164] */
> +BYTES(0xd0000355),63,15,0,252,60,254,254,252,BYTES(0x00288880),
> +
> +/* wctype: pages U+1d600,11900 [words 165-167] */
> +BYTES(0x8b000000),2,16,16,5,BYTES(0x30000000),
> +
> +/* wctype: pages U+1da00,11f00 [words 168-170] */
> +BYTES(0x003a5555),75,240,254,131,BYTES(0x95000000),
> +
> +/* wctype: pages U+1e200,16f00 [words 171-173] */
> +BYTES(0xa0000000),224,21,0,4,BYTES(0x30000000),
> +
> +/* wctype: pages U+1e800,1ec00 [words 174-176] */
> +BYTES(0x0f000000),128,127,0,0,
> +/* wctype: pages U+1ec00,1ee00 [words 176-178] */
> +BYTES(0x00d5c000),254,31,0,0,
> +/* wctype: pages U+1ee00,1f000 [words 178-180] */
> +BYTES(0xc0000000),3,0,0,0,
> +/* wctype: pages U+1f000,1f200 [words 180-183] */
> +BYTES(0xdfed5565),75,15,1,254,254,254,63,0,
> +/* wctype: pages U+1f200,1f600 [words 183-186] */
> +BYTES(0x00003e97),7,75,105,3,63,0,0,0,
> +/* wctype: pages U+1f600,1f800 [words 186-188] */
> +BYTES(0xad555555),63,31,13,0,
> +/* wctype: pages U+1f800,1fa00 [words 188-191] */
> +BYTES(0x00265a56),75,45,19,45,93,0,0,0,
> +/* wctype: pages U+1fa00,1fa00 [words 191-195] */
> +BYTES(0x000fad55),15,93,147,7,63,0,0,0,0,0,0,0,
> diff --git a/src/ctype/punct.h b/src/ctype/punct.h
> deleted file mode 100644
> index 6792947..0000000
> --- a/src/ctype/punct.h
> +++ /dev/null
> @@ -1,141 +0,0 @@
>
> -18,16,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,16,16,34,35,16,36,37,38,39,
>
> -40,41,42,43,16,44,45,46,17,17,47,17,17,17,17,17,17,48,49,50,51,52,53,54,55,17,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,56,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,57,16,58,59,60,61,62,63,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,64,16,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,65,16,16,66,16,67,68,
>
> -69,16,70,71,72,16,73,16,16,74,75,76,77,78,16,79,80,81,82,83,84,85,86,87,88,89,
>
> -90,91,16,92,93,94,95,16,16,16,16,96,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,97,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,98,99,16,16,100,101,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,102,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,103,104,105,106,16,16,107,108,17,17,109,16,16,16,16,16,16,110,111,16,
>
> -16,16,16,16,112,113,16,16,114,115,116,16,117,118,119,17,17,17,120,121,122,123,
> -124,16,16,16,16,
>
> -16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,254,255,0,252,1,0,0,248,1,
>
> -0,0,120,0,0,0,0,255,251,223,251,0,0,128,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,60,0,252,255,224,175,255,255,255,255,255,255,255,255,
>
> -255,255,223,255,255,255,255,255,32,64,176,0,0,0,0,0,0,0,0,0,0,0,0,0,64,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,252,0,0,0,0,0,230,254,255,255,255,0,64,73,0,0,0,0,0,24,0,255,255,0,216,
> -0,0,0,0,0,0,0,1,0,60,0,0,0,0,0,0,0,0,0,0,0,0,16,224,1,30,0,
>
> -96,255,191,0,0,0,0,0,0,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,248,207,
>
> -227,0,0,0,3,0,32,255,127,0,0,0,78,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,0,7,252,0,0,0,
>
> -0,0,0,0,0,0,16,0,32,30,0,48,0,1,0,0,0,0,0,0,0,0,16,0,32,0,0,0,0,252,111,0,0,0,
>
> -0,0,0,0,16,0,32,0,0,0,0,64,0,0,0,0,0,0,0,0,16,0,32,0,0,0,0,3,224,0,0,0,0,0,0,
>
> -0,16,0,32,0,0,0,0,253,0,0,0,0,0,0,0,0,0,0,32,0,0,0,0,255,7,16,0,0,0,0,0,0,0,0,
>
> -32,0,0,0,0,128,255,16,0,0,0,0,0,0,16,0,32,0,0,0,0,0,0,0,0,0,0,0,0,0,24,0,160,
>
> -0,127,0,0,255,3,0,0,0,0,0,0,0,0,0,4,0,0,0,0,16,0,0,0,0,0,0,128,0,128,192,223,
> -0,12,0,0,0,0,0,0,0,0,0,0,0,4,0,31,0,0,0,0,0,
>
> -0,254,255,255,255,0,252,255,255,0,0,0,0,0,0,0,0,252,0,0,0,0,0,0,192,255,223,
>
> -255,7,0,0,0,0,0,0,0,0,0,0,128,6,0,252,0,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,0,
>
> -0,0,8,0,0,0,0,0,0,0,0,0,0,0,224,255,255,255,31,0,0,255,3,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,96,0,0,1,0,0,24,0,0,0,0,0,0,0,0,0,56,0,0,0,0,16,0,0,0,112,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,48,0,0,254,127,47,0,0,255,3,255,127,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,14,49,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,0,0,0,196,255,255,255,
>
> -255,0,0,0,192,0,0,0,0,0,0,0,0,1,0,224,159,0,0,0,0,127,63,255,127,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,16,0,16,0,0,252,255,255,255,31,0,0,0,0,0,12,0,0,0,0,0,0,64,0,
>
> -12,240,0,0,0,0,0,0,128,248,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,255,0,255,255,
>
> -255,33,144,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,
>
> -127,0,224,251,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,160,3,224,0,224,0,
>
> -224,0,96,128,248,255,255,255,252,255,255,255,255,255,127,223,255,241,127,255,
>
> -127,0,0,255,255,255,255,0,0,255,255,255,255,1,0,123,3,208,193,175,66,0,12,31,
>
> -188,255,255,0,0,0,0,0,14,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,127,0,0,0,255,7,0,0,255,255,255,255,255,255,255,255,255,
> -255,63,0,0,0,0,0,0,252,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,207,255,255,255,
>
> -63,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,224,135,3,254,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,127,255,255,255,255,0,
>
> -0,0,0,0,0,255,255,255,251,255,255,255,255,255,255,255,255,255,255,15,0,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,63,0,0,0,255,15,30,255,255,255,1,252,193,224,0,0,0,0,
>
> -0,0,0,0,0,0,0,30,1,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -255,255,0,0,0,0,255,255,255,255,15,0,0,0,255,255,255,127,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> -255,255,255,
>
> -255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,
>
> -255,255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,0,0,0,
>
> -0,0,0,192,0,224,0,0,0,0,0,0,0,0,0,0,0,128,15,112,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -255,0,255,255,127,0,3,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -64,0,0,0,0,15,255,3,0,0,0,0,0,0,240,0,0,0,0,0,0,0,0,0,16,192,0,0,255,255,3,23,
>
> -0,0,0,0,0,248,0,0,0,0,8,128,0,0,0,0,0,0,0,0,0,0,8,0,255,63,0,192,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,240,0,0,128,3,0,0,0,0,0,0,0,128,2,0,0,192,0,0,67,0,0,0,0,0,
> -0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,56,0,
>
> -0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,2,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,252,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,48,255,255,255,3,255,255,255,255,255,255,247,
>
> -255,127,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,254,255,0,252,1,0,0,248,1,0,
>
> -0,248,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,127,127,0,48,135,255,255,255,255,255,
>
> -143,255,0,0,0,0,0,0,224,255,255,127,255,15,1,0,0,0,0,0,255,255,255,255,255,63,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,
>
> -15,0,0,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -128,255,0,0,128,255,0,0,0,0,128,255,0,0,0,0,0,0,0,0,0,248,0,0,192,143,0,0,0,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,48,255,255,252,255,255,255,255,255,0,0,0,0,
>
> -0,0,0,135,255,1,255,1,0,0,0,224,0,0,0,224,0,0,0,0,0,1,0,0,96,248,127,0,0,0,0,
>
> -0,0,0,0,254,0,0,0,255,0,0,0,255,0,0,0,30,0,254,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,0,0,0,0,0,0,0,0,0,0,0,
> -0,255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,224,127,0,0,0,192,255,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,192,63,252,255,63,0,0,128,3,0,0,0,0,0,0,254,3,32,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,24,0,15,0,0,0,0,0,56,0,0,0,0,0,0,0,0,0,225,63,0,232,254,255,31,0,0,
>
> -0,0,0,0,0,96,63,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,
>
> -24,0,32,0,0,192,31,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,68,
>
> -248,0,104,0,0,0,0,0,0,0,0,0,0,0,0,76,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,128,255,255,255,0,0,0,0,0,0,0,0,0,0,0,0,128,14,0,0,0,255,
> -31,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,8,0,252,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,7,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,24,128,255,0,0,0,0,0,
>
> -0,0,0,0,0,223,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,62,0,0,252,255,31,3,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,52,0,0,0,0,0,0,0,0,0,128,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,128,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
> -255,3,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,192,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,63,0,0,0,0,0,0,0,255,255,48,0,0,248,
>
> -3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,176,15,0,0,0,0,0,0,
>
> -0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> -255,255,255,255,255,255,255,255,255,255,255,255,255,63,
>
> -0,255,255,255,255,127,254,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,1,0,0,255,255,255,255,255,255,255,255,
>
> -63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,15,0,255,255,255,255,255,255,
>
> -255,255,255,255,127,0,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,8,0,0,0,8,0,0,32,0,0,0,32,0,0,128,
>
> -0,0,0,128,0,0,0,2,0,0,0,2,0,0,8,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,15,0,248,254,255,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,127,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,240,0,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,255,127,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,112,7,0,192,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,254,255,255,255,255,255,255,255,31,0,0,0,0,0,0,0,0,0,254,255,
>
> -255,255,255,255,255,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,255,255,255,255,255,
>
> -15,255,255,255,255,255,255,255,255,255,255,255,255,15,0,255,127,254,255,254,
>
> -255,254,255,255,255,63,0,255,31,255,255,255,255,0,0,0,252,0,0,0,28,0,0,0,252,
>
> -255,255,255,31,0,0,0,0,0,0,192,255,255,255,7,0,255,255,255,255,255,15,255,1,3,
> -0,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,63,0,255,31,255,7,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,15,0,255,255,255,255,255,255,255,255,255,255,255,1,
>
> -255,15,0,0,255,15,255,255,255,255,255,255,255,0,255,3,255,255,255,255,255,0,
>
> -255,255,255,63,0,0,0,0,0,0,0,0,0,0,255,239,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,123,252,255,255,255,255,231,199,255,255,255,231,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,15,0,255,63,15,7,7,0,63,0,
> -0,0,0,0,0,0,0,0,0,0,0,0,
> --
> 2.51.2
>
>
[-- Attachment #1.2: Type: text/html, Size: 60363 bytes --]
[-- Attachment #2: gen_ctype.c --]
[-- Type: text/x-csrc, Size: 9734 bytes --]
#include <wctype.h>
#include <endian.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#define PAGE_SH 8
#define PAGE_MAX (1u << PAGE_SH)
#define PAGES (0x20000 / PAGE_MAX)
#define PAGEH (0x1000/8 + PAGES) /* Direct map 0x1000 codepts (or less) */
static int tabnr, dictnr, matchnr, directwc;
static int vflag;
/* Table max PAGEH+254 words and dictionary max 128, but both may be smaller */
static union {
unsigned char b[PAGEH + 254*4];
unsigned w[PAGEH/4 + 254];
} table;
static unsigned short dict[128];
/* Only include headers to test against old style data, otherwise zero init */
unsigned char src_table[8192] = {
#if WCTYPE_USE_OLD == 1
#include "alpha.h"
#elif WCTYPE_USE_OLD == 2
#include "punct.h"
#else
0
#endif
};
static int getwcprop(wint_t wc, const unsigned char src[]) {
if (wc<0x20000) {
return (src[src[wc>>8]*32 + ((wc&255)>>3)]>>(wc&7)) & 1;
}
return 1;
}
static void read_src(unsigned wanted_set, unsigned char src[8192]) {
char *set = calloc(0x110000,1);
char buf[128], dummy;
unsigned a, b;
FILE *f1, *f2;
f1 = fopen("data/UnicodeData.txt", "rb");
f2 = fopen("data/DerivedCoreProperties.txt", "rb");
if (!f1 || !f2 || wanted_set<1 || wanted_set>2) {
printf("error: no Unicode data files or unknown set %d\n", wanted_set);
return;
}
while (fgets(buf, sizeof buf, f1)) {
if (sscanf(buf, "%x;%*[^;];Nd%c", &a, &dummy)==2)
set[a] = 1;
else if (sscanf(buf, "%x;%*[^;];%c", &a, &dummy)==2)
set[a] = 2;
}
fclose(f1);
while (fgets(buf, sizeof buf, f2)) {
if (sscanf(buf, "%x..%x ; Alphabetic%c", &a, &b, &dummy)==3)
for (; a<=b; a++) set[a]=1;
else if (sscanf(buf, "%x ; Alphabetic%c", &a, &dummy)==2)
set[a] = 1;
}
fclose(f2);
/* Fix misclassified Thai characters */
set[0xe2f] = set[0xe46] = 2;
/* Clear digits */
for (a=0x30; a<=0x39; a++) set[a]=0;
/* Clear spaces */
set[0x0020] = 0;
for (a=0x2000; a<=0x2006; a++) set[a]=0;
for (a=0x2008; a<=0x200a; a++) set[a]=0;
set[0x205f] = 0;
set[0x3000] = 0;
/* Clear controls */
for (a=0x00; a<=0x1f; a++) set[a]=0;
for (a=0x7f; a<=0x9f; a++) set[a]=0;
for (a=0x2028; a<=0x2029; a++) set[a]=0;
for (a=0xfff9; a<=0xfffb; a++) set[a]=0;
for (a=0xd800; a<=0xdfff; a++) set[a]=0;
/* Fill in elided CJK ranges */
for (a=0x3400; a<=0x4db5; a++) set[a]=1;
for (a=0x4e00; a<=0x9fcc; a++) set[a]=1;
for (a=0xac00; a<=0xd7a3; a++) set[a]=1;
for (a=0x20000; a<=0x2a6d6; a++) set[a]=1;
for (a=0x2a700; a<=0x2b734; a++) set[a]=1;
for (a=0x2b740; a<=0x2b81d; a++) set[a]=1;
for (a=0; a<=0x10ffff; a++)
if (set[a]!=wanted_set) set[a]=0;
int i=0, p=0, step=256;
for (a=0; a<0x20000; a+=step, i++) {
for (b=0; b<step; b++)
if (set[a+b]!=set[a]) break;
if (b!=step)
src[i]=18+p, p++;
else
src[i]=16+!!set[a];
}
for (a=0; a<32; a++) src[i++] = 0;
for (a=0; a<32; a++) src[i++] = 255;
for (a=0; a<0x20000; a+=step) {
unsigned x=0;
for (b=0; b<step; b++)
if (set[a+b]!=set[a]) break;
if (b!=step) for (b=0; b<step; b++) {
x=x/2+128*!!set[a+b];
if (!(b+1&7))
src[i]=x&255, x=0, i++;
}
}
if (vflag)
printf("read Unicode data into src[%d]\n", i);
free(set);
}
static int encode_table(unsigned pat_low, const unsigned char src[]) {
typedef unsigned huffword;
huffword words [PAGES][PAGE_MAX/16 + 1] = {0};
huffword wordnr[PAGES] = {0};
unsigned align=sizeof(huffword);
unsigned start=directwc/PAGE_MAX;
unsigned h=PAGEH/align-2;
int p, q, i, j, d, e, err;
memset(table.b, 0, sizeof table.b);
for (p=0, err=0; p < PAGES; p++) {
unsigned wc = p * PAGE_MAX;
char *exts = (char *)words[p] + align;
/* Some code pages where direct bitmap encoding is better */
if (wc<directwc) {
for (i=0; i<PAGE_MAX; i++) {
unsigned a = getwcprop(wc+i, src);
table.b[PAGES + (wc+i)/8] |= a << ((wc+i) & 7);
}
continue;
}
/* Skip all-one or all-zero pages */
for (i=0; i<PAGE_MAX; i++)
if (getwcprop(wc+i, src)!=getwcprop(wc, src)) break;
table.b[p] = i==PAGE_MAX ? getwcprop(wc, src):255;
if (i==PAGE_MAX) continue;
/* Mixed block: process PAGE_MAX bits as words of 16b at a time */
if (vflag)
printf("page U+%05x: ", wc);
for (i=0, e=0, d=dictnr; i<PAGE_MAX/16; i++) {
unsigned word=0, quad=(~pat_low & 1), ext=0;
for (j=0; j<16; j++)
word |= getwcprop(wc + i*16 + j, src) << j;
if (word==0x0000 || word==0xffff) {
words[p][i*2 / (align*8)] |= (word&1) << (i*2 % (align*8));
continue;
}
if (dictnr>=128)
err=1, printf("error: data doesn't fit constraints\n");
if (word<=0x3fff || word>=0xc000) {
unsigned upper=word & 0xff80;
if (upper==0 || upper==0xff80)
quad=2, ext=word;
else if ((word&0xff)==pat_low)
quad=(pat_low&1), ext=(word>>7) & 0xfe;
}
if (quad==(~pat_low&1)) {
for (j=0; j<dictnr;j++)
if (dict[j] == word) break;
if (j==dictnr)
dict[dictnr++] = word;
ext=j<<1;
}
words[p][i*2/(align*8)] |= (quad>=2 ? 3:2) << (i*2 % (align*8));
exts[e++]=(ext&0xff) | (quad&1);
if (vflag)
printf("0x%04x [%02x] ", word, exts[e-1]);
}
wordnr[p]=e;
if (vflag)
printf("\ngenerated U+%05x: extnr %2d dictnr %3d total %d\n",
wc, e, dictnr, 4+e+(dictnr-d)*2);
}
if (!err) for (p=start, tabnr=2*align; p<PAGES; p+=2) {
int threshold=0, gap=0;
if (table.b[p]!=255)
continue;
/* Match odd pages with even pages so that padding gaps are minimised */
for (q=start+1; q<PAGES; q+=2) {
gap=(align-(wordnr[p]+wordnr[q]) % align) % align;
if (table.b[q]==255 && gap<=threshold)
break;
if (q+2>PAGES)
threshold++, q=start-1;
if (threshold>align) {
printf("/* success: all odd pages matched to even < 0x%03x */\n", p);
gap=(align-wordnr[p] % align) % align;
matchnr = matchnr ? matchnr:p;
q=-1;
break;
}
}
/* Write data into table */
if (q>=0 && (wordnr[p]+wordnr[q]+gap) % align || tabnr % align) {
printf("error: data alignment corrupted at %d:%d %d:%d %d [%d]\n",
p, q, wordnr[p], q>=0 ? wordnr[q]:0, gap, tabnr);
return 2;
}
table.b[p]=tabnr/align;
table.w[h+tabnr/align]=words[p][0];
tabnr += 4;
memcpy(&table.w[h + tabnr/align], words[p]+1, wordnr[p]);
tabnr += wordnr[p]+gap;
if (q>=0) {
for (j=0; j<wordnr[q]; j++)
(table.b + h*align)[tabnr++]=((char *)words[q])[align-1+wordnr[q]-j];
table.w[h+tabnr/align]=words[q][0];
table.b[q]=tabnr/align;
tabnr += 4;
}
if (tabnr>=254*align || h*align+tabnr>=sizeof table.b) {
printf("error: data too big or resize table to fit data\n");
return 3;
}
if (vflag)
printf("inserted U+%05x: tabnr %d wordnr %d:%d, %d:%d gap %d\n",
p * PAGE_MAX, tabnr, p, q, wordnr[p], q>=0 ? wordnr[q]:0, gap);
}
return err;
}
void verify_table(int wanted_set, const unsigned char src[]) {
unsigned wc;
unsigned log=0, fails=0;
for (wc=0x0; wc<0x20000; wc++) {
int exp=getwcprop(wc, src);
int got=wanted_set<=1 ? iswalpha(wc):iswpunct(wc);
if (exp!=!!got)
log+=!log, fails++;
if (log && log++<=20) {
unsigned direct, page, bmap, shfr, lane;
unsigned target;
char *msg = (exp == got) ? "PASS":"FAIL";
direct = wc < directwc;
page = (wc >> PAGE_SH);
bmap = (wc >> 3) + PAGES & -direct;
target = wc & (PAGE_MAX-1);
shfr = (target & 15);
lane = (target >> 4);
printf("/* %s U+%05X: exp %d got %d page %03x bmap %03x "
"target 0x%03x shfr %2d lane %2d */\n",
msg, wc, exp, got, page, bmap, target, shfr, lane);
}
}
printf("/* wctype: %u codepoints verified, %u errors */\n", wc, fails);
}
void export_table() {
unsigned align=4;
unsigned start=directwc/PAGE_MAX;
unsigned h=PAGEH/align-2, b=h*align;
int p, q, j, beg, end;
printf("/* wctype: table %d x %d codepoints */\n", PAGES, PAGE_MAX);
for (p=0; p<PAGES+directwc/8; p++)
printf("%3d%s", table.b[p], p%16==15 ? ",\n":",");
for (p=start; p<PAGES; p+=2) {
unsigned char *s;
if ((beg = table.b[p]) <= 1)
continue;
for (j=p+2; j<PAGES && (end=table.b[j])<2; j+=2);
if (j>=PAGES)
end=tabnr/align + 1; /* +1 is for guard word at end */
end -= (p<matchnr);
s=memchr(table.b+start, end, PAGES-start);
q=s ? s-table.b:p;
printf("\n/* wctype: pages U+%05x,%05x [words %d-%d] */\n",
p*PAGE_MAX, q*PAGE_MAX, beg-2, end-2);
printf("BYTES(0x%08x),", table.w[h+beg]);
for (j=beg+1; j<end; j++) {
printf("%d,%d,%d,%d,%s", table.b[b+j*align], table.b[b+j*align+1],
table.b[b+j*align+2], table.b[b+j*align+3], (j-beg)%4==3 ? "\n":"");
}
if (p < matchnr)
printf("BYTES(0x%08x),\n", table.w[h+j++]);
}
}
void export_dict() {
unsigned i;
printf("/* wctype: dictionary %d entries */\n", dictnr);
for (i=0; i<dictnr; i++)
printf("%d%s", dict[i], i%8==7 ? ",\n":",");
printf("\n");
}
int main(int argc, char *argv[]) {
char *arg0=*argv++;
char *cmd;
if (*argv && strcmp(*argv, "-v")==0) {
/* Verbose output */
++argv;
vflag=1;
}
cmd=argv[0];
if (cmd && (*cmd | 32)=='t') {
/* Test iswalpha (ta) or ispunct (tp) */
char opt = argv[0][1];
directwc=(opt=='p') ? 0x800:0x1000;
if (!opt || opt>='a' && *cmd=='t')
read_src(1 + (opt=='p'), src_table);
verify_table(1 + (opt=='p'||opt=='P'), src_table);
} else if (cmd && (*cmd | 32)=='a') {
/* Export table (a) or dictionary (A) for iswalpha */
directwc=0x1000;
read_src(1, src_table);
encode_table(0xff, src_table);
*cmd == 'a' ? export_table():export_dict();
} else if (cmd && (*cmd | 32)=='p') {
/* Export table (p) or dictionary (P) for iswpunct */
directwc=0x800;
read_src(2, src_table);
encode_table(0x00, src_table);
*cmd == 'p' ? export_table():export_dict();
} else {
printf("usage: %s [-v] a|p|A|P|t|ta|tp|Ta|Tp\n", arg0);
}
return 0;
}
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47%
2026-01-23 13:20 [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47% Xan Phung
2026-01-23 13:40 ` Xan Phung
@ 2026-01-30 12:39 ` Xan Phung
2026-02-18 1:28 ` Xan Phung
2 siblings, 0 replies; 6+ messages in thread
From: Xan Phung @ 2026-01-30 12:39 UTC (permalink / raw)
To: Openwall musl
[-- Attachment #1: Type: text/plain, Size: 55674 bytes --]
Hi, I hope everyone is having a good week.
V2 of my patch has addressed the issues that were identified in V1,
including:
- need for data generation tool (done, direct replacement for gen_ctype
provided)
- aliasing violation fixed (using union type)
- decimal literals instead of hex literals for more compact source code
- endianness portability: used a BYTES macro to encode correct endianness
for the 32b words contained in the literal byte array data.
Let me know if there are other issues to be considered?
The benefit of this patch is a 4.2kb reduction in binary footprint (code +
all data), whilst having higher performance (only 1 memory access needed
for 85% of code pages, including the high usage ones, while the remaining
15% 'niche' code pages use 3 memory accesses).
Best regards
Xan Phung
On Sat, 24 Jan 2026 at 00:21, Xan Phung <xan.phung@gmail.com> wrote:
> Currently iswalpha and iswpunct have total text data size of over 8kb.
> A more efficient encoding has reduced the total size to 4.2kb, a 47%
> reduction.
>
> This encoding also features a speed improvement, with lookups for
> approx 220 of the 256 BMP code blocks done with just a single memory
> load.
>
> The remaining Unicode blocks are niche uses, and are accessed with
> three memory loads. Compared to the old musl encoding (which used a
> two level table) the topmost level remains much the same, providing
> offsets into code block units (256 codepoint granularity). The second
> level data uses fixed sizes, of one 32 bit word per codepage (where
> each 2 bit pair in word identifies a block of 16 codepoints as all 0,
> all 1, or mixed). The third level is a variable length series of
> extension bytes, indexed by the popcount of set high bits within the
> second level's 32 bit word. Popcount is used as not all 16 codepoint
> blocks need an extension byte. This popcount is calculated with
> nearly same latency as a 32 bit multiply (so it is comparable with
> the indexing speed of accessing a 2D array of non power of 2 size),
> but has the advantage of greater compactness for sparse data.
>
> Results have been tested against first 0x20000 codepoints, and match
> that returned by the pre-existing musl implementation.
>
> A new char table tool to replace the existing gen_ctype is provided
> separately. The new tool has similar command line use:
>
> gen_ctype [a|p] -- generates iswalpha & iswpunct table.h files
> gen_ctype [A|P] -- generates iswalpha & iswpunct dict.h files
>
> Signed-off-by: Xan Phung <xan.phung@gmail.com>
> ---
> src/ctype/alpha.h | 172 -----------------------------
> src/ctype/iswalpha.c | 62 +++++++++--
> src/ctype/iswalpha_dict.h | 15 +++
> src/ctype/iswalpha_table.h | 217 +++++++++++++++++++++++++++++++++++++
> src/ctype/iswpunct.c | 60 ++++++++--
> src/ctype/iswpunct_dict.h | 19 ++++
> src/ctype/iswpunct_table.h | 211 ++++++++++++++++++++++++++++++++++++
> src/ctype/punct.h | 141 ------------------------
> 8 files changed, 568 insertions(+), 329 deletions(-)
> delete mode 100644 src/ctype/alpha.h
> create mode 100644 src/ctype/iswalpha_dict.h
> create mode 100644 src/ctype/iswalpha_table.h
> create mode 100644 src/ctype/iswpunct_dict.h
> create mode 100644 src/ctype/iswpunct_table.h
> delete mode 100644 src/ctype/punct.h
>
> diff --git a/src/ctype/alpha.h b/src/ctype/alpha.h
> deleted file mode 100644
> index 4167f38..0000000
> --- a/src/ctype/alpha.h
> +++ /dev/null
> @@ -1,172 +0,0 @@
>
> -18,17,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,17,34,35,36,17,37,38,39,40,
>
> -41,42,43,44,17,45,46,47,16,16,48,16,16,16,16,16,16,16,49,50,51,16,52,53,16,16,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,54,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,55,17,17,17,17,56,17,57,58,59,60,61,62,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,63,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,64,65,17,66,67,
>
> -68,69,70,71,72,73,74,17,75,76,77,78,79,80,81,16,82,83,84,85,86,87,88,89,90,91,
>
> -92,93,16,94,95,96,16,17,17,17,97,98,99,16,16,16,16,16,16,16,16,16,16,17,17,17,
>
> -17,100,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,17,101,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,17,17,102,103,16,16,104,105,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,106,17,17,107,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,
>
> -108,109,16,16,16,16,16,16,16,16,16,110,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,111,112,113,114,16,16,16,16,16,16,16,16,115,116,
>
> -117,16,16,16,16,16,118,119,16,16,16,16,120,16,16,121,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,
>
> -16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,254,255,255,7,254,
>
> -255,255,7,0,0,0,0,0,4,32,4,255,255,127,255,255,255,127,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,195,255,3,0,31,80,0,0,0,0,0,0,0,0,0,0,32,0,0,0,0,0,223,188,64,215,255,255,
>
> -251,255,255,255,255,255,255,255,255,255,191,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,3,252,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,254,255,255,255,127,2,255,255,255,
>
> -255,255,1,0,0,0,0,255,191,182,0,255,255,255,135,7,0,0,0,255,7,255,255,255,255,
>
> -255,255,255,254,255,195,255,255,255,255,255,255,255,255,255,255,255,255,239,
> -31,254,225,255,
>
> -159,0,0,255,255,255,255,255,255,0,224,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,3,0,255,255,255,255,255,7,48,4,255,255,255,252,255,31,0,0,255,255,255,
>
> -1,255,7,0,0,0,0,0,0,255,255,223,63,0,0,240,255,248,3,255,255,255,255,255,255,
>
> -255,255,255,239,255,223,225,255,207,255,254,255,239,159,249,255,255,253,197,
>
> -227,159,89,128,176,207,255,3,16,238,135,249,255,255,253,109,195,135,25,2,94,
>
> -192,255,63,0,238,191,251,255,255,253,237,227,191,27,1,0,207,255,0,30,238,159,
>
> -249,255,255,253,237,227,159,25,192,176,207,255,2,0,236,199,61,214,24,199,255,
>
> -195,199,29,129,0,192,255,0,0,239,223,253,255,255,253,255,227,223,29,96,7,207,
>
> -255,0,0,239,223,253,255,255,253,239,227,223,29,96,64,207,255,6,0,239,223,253,
>
> -255,255,255,255,231,223,93,240,128,207,255,0,252,236,255,127,252,255,255,251,
>
> -47,127,128,95,255,192,255,12,0,254,255,255,255,255,127,255,7,63,32,255,3,0,0,
> -0,0,214,247,255,255,175,255,255,59,95,32,255,243,0,0,0,
>
> -0,1,0,0,0,255,3,0,0,255,254,255,255,255,31,254,255,3,255,255,254,255,255,255,
>
> -31,0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,249,255,3,255,255,255,255,255,
>
> -255,255,255,255,63,255,255,255,255,191,32,255,255,255,255,255,247,255,255,255,
>
> -255,255,255,255,255,255,61,127,61,255,255,255,255,255,61,255,255,255,255,61,
>
> -127,61,255,127,255,255,255,255,255,255,255,61,255,255,255,255,255,255,255,255,
>
> -7,0,0,0,0,255,255,0,0,255,255,255,255,255,255,255,255,255,255,63,63,254,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,159,255,255,254,255,255,7,255,255,255,255,255,255,255,255,
>
> -255,199,255,1,255,223,15,0,255,255,15,0,255,255,15,0,255,223,13,0,255,255,255,
>
> -255,255,255,207,255,255,1,128,16,255,3,0,0,0,0,255,3,255,255,255,255,255,255,
>
> -255,255,255,255,255,1,255,255,255,255,255,7,255,255,255,255,255,255,255,255,
> -63,
>
> -0,255,255,255,127,255,15,255,1,192,255,255,255,255,63,31,0,255,255,255,255,
>
> -255,15,255,255,255,3,255,3,0,0,0,0,255,255,255,15,255,255,255,255,255,255,255,
>
> -127,254,255,31,0,255,3,255,3,128,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
>
> -255,239,255,239,15,255,3,0,0,0,0,255,255,255,255,255,243,255,255,255,255,255,
>
> -255,191,255,3,0,255,255,255,255,255,255,127,0,255,227,255,255,255,255,255,63,
>
> -255,1,255,255,255,255,255,231,0,0,0,0,0,222,111,4,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,
>
> -128,255,31,0,255,255,63,63,255,255,255,255,63,63,255,170,255,255,255,63,255,
>
> -255,255,255,255,255,223,95,220,31,207,15,255,31,220,31,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,2,128,0,0,255,31,0,0,0,0,0,0,0,0,0,0,0,0,132,252,47,62,80,189,255,243,
> -224,67,0,0,255,255,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,192,255,255,255,255,255,255,3,0,
>
> -0,255,255,255,255,255,127,255,255,255,255,255,127,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,31,120,12,0,255,255,255,255,191,32,255,
>
> -255,255,255,255,255,255,128,0,0,255,255,127,0,127,127,127,127,127,127,127,127,
>
> -255,255,255,255,0,0,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,224,0,0,0,254,3,62,31,254,255,255,255,255,255,255,255,255,255,127,224,254,
>
> -255,255,255,255,255,255,255,255,255,255,247,224,255,255,255,255,255,254,255,
>
> -255,255,255,255,255,255,255,255,255,127,0,0,255,255,255,7,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,63,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,31,0,0,
>
> -0,0,0,0,0,0,255,255,255,255,255,63,255,31,255,255,255,15,0,0,255,255,255,255,
>
> -255,127,240,143,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,
>
> -0,128,255,252,255,255,255,255,255,255,255,255,255,255,255,255,249,255,255,255,
>
> -255,255,255,124,0,0,0,0,0,128,255,191,255,255,255,255,0,0,0,255,255,255,255,
>
> -255,255,15,0,255,255,255,255,255,255,255,255,47,0,255,3,0,0,252,232,255,255,
>
> -255,255,255,7,255,255,255,255,7,0,255,255,255,31,255,255,255,255,255,255,247,
>
> -255,0,128,255,3,255,255,255,127,255,255,255,255,255,255,127,0,255,63,255,3,
>
> -255,255,127,252,255,255,255,255,255,255,255,127,5,0,0,56,255,255,60,0,126,126,
>
> -126,0,127,127,255,255,255,255,255,247,255,0,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,7,255,3,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,15,0,255,255,127,248,255,255,255,255,
> -255,
>
> -15,255,255,255,255,255,255,255,255,255,255,255,255,255,63,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,3,0,0,0,0,127,0,248,224,255,253,127,95,219,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,3,0,0,0,248,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,63,0,0,255,255,255,255,255,255,255,255,
>
> -252,255,255,255,255,255,255,0,0,0,0,0,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,223,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,31,0,0,255,3,
>
> -254,255,255,7,254,255,255,7,192,255,255,255,255,255,255,255,255,255,255,127,
>
> -252,252,252,28,0,0,0,0,255,239,255,255,127,255,255,183,255,63,255,63,0,0,0,0,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,7,0,0,0,0,0,0,0,0,
>
> -255,255,255,255,255,255,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,255,255,255,31,255,255,255,255,255,255,1,0,0,0,0,
>
> -0,255,255,255,255,0,224,255,255,255,7,255,255,255,255,255,7,255,255,255,63,
>
> -255,255,255,255,15,255,62,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,63,255,3,255,255,255,255,15,255,255,255,
>
> -255,15,255,255,255,255,255,0,255,255,255,255,255,255,15,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,0,255,255,63,0,255,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,63,253,255,255,255,255,191,145,255,255,63,0,255,255,
>
> -127,0,255,255,255,127,0,0,0,0,0,0,0,0,255,255,55,0,255,255,63,0,255,255,255,3,
>
> -0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,192,0,0,0,0,0,0,0,0,111,240,239,
>
> -254,255,255,63,0,0,0,0,0,255,255,255,31,255,255,255,31,0,0,0,0,255,254,255,
>
> -255,31,0,0,0,255,255,255,255,255,255,63,0,255,255,63,0,255,255,7,0,255,255,3,
> -0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,1,0,0,0,0,0,0,255,255,255,255,255,255,7,
>
> -0,255,255,255,255,255,255,7,0,255,255,255,255,255,0,255,3,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,31,128,0,255,255,63,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,255,255,127,0,255,255,255,255,255,255,255,255,63,0,0,0,
>
> -192,255,0,0,252,255,255,255,255,255,255,1,0,0,255,255,255,1,255,3,255,255,255,
>
> -255,255,255,199,255,112,0,255,255,255,255,71,0,255,255,255,255,255,255,255,
>
> -255,30,0,255,23,0,0,0,0,255,255,251,255,255,255,159,64,0,0,0,0,0,0,0,0,127,
>
> -189,255,191,255,1,255,255,255,255,255,255,255,1,255,3,239,159,249,255,255,253,
>
> -237,227,159,25,129,224,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,
>
> -255,255,255,255,255,187,7,255,131,0,0,0,0,255,255,255,255,255,255,255,255,179,
> -0,255,3,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,63,127,0,0,0,63,0,0,
>
> -0,0,255,255,255,255,255,255,255,127,17,0,255,3,0,0,0,0,255,255,255,255,255,
>
> -255,63,1,255,3,0,0,0,0,0,0,255,255,255,231,255,7,255,3,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,3,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,255,252,255,255,255,255,255,252,26,0,0,0,255,255,255,255,255,255,231,
>
> -127,0,0,255,255,255,255,255,255,255,255,255,32,0,0,0,0,255,255,255,255,255,
>
> -255,255,1,255,253,255,255,255,255,127,127,1,0,255,3,0,0,252,255,255,255,252,
>
> -255,255,254,127,0,0,0,0,0,0,0,0,0,127,251,255,255,255,255,127,180,203,0,255,3,
> -191,253,255,255,255,127,123,1,255,3,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,127,0,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,3,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,255,255,127,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,255,255,255,255,1,255,255,255,127,255,3,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,63,0,0,255,255,255,255,255,255,0,0,15,0,255,3,248,255,255,224,255,255,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,255,255,255,255,135,255,255,255,255,255,255,255,128,
>
> -255,255,0,0,0,0,0,0,0,0,11,0,0,0,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,7,0,255,255,255,127,0,0,0,0,0,
>
> -0,7,0,240,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,15,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,7,255,31,255,1,255,67,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,255,255,255,255,255,223,255,255,255,255,255,255,255,
> -255,223,100,222,255,235,239,255,255,255,255,255,255,
>
> -255,191,231,223,223,255,255,255,123,95,252,253,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,63,255,255,255,
>
> -253,255,255,247,255,255,255,247,255,255,223,255,255,255,223,255,255,127,255,
>
> -255,255,127,255,255,255,253,255,255,255,253,255,255,247,207,255,255,255,255,
>
> -255,255,127,255,255,249,219,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,31,128,63,255,67,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
>
> -15,255,3,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,31,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,
> -143,8,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,239,255,255,255,150,254,247,10,132,234,150,170,150,247,247,94,255,251,255,
>
> -15,238,251,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,3,255,255,255,3,255,
> -255,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> diff --git a/src/ctype/iswalpha.c b/src/ctype/iswalpha.c
> index 1c5485d..d92c2e3 100644
> --- a/src/ctype/iswalpha.c
> +++ b/src/ctype/iswalpha.c
> @@ -1,16 +1,60 @@
> #include <wctype.h>
> +#include <endian.h>
>
> -static const unsigned char table[] = {
> -#include "alpha.h"
> +#define BYTE(x,y) ((x) >> (y^(__BYTE_ORDER == __BIG_ENDIAN ? 0:24)) & 255)
> +#define BYTES(x) BYTE(x,24),BYTE(x,16),BYTE(x,8),BYTE(x,0)
> +#define PAGE_SH 8
> +#define PAGE_MAX (1u << PAGE_SH)
> +#define PAGES (0x20000 / PAGE_MAX)
> +#define PAGEH (0x1000/8 + PAGES) /* Direct map 0x1000 codepts */
> +
> +const static union {
> + unsigned char b[PAGEH + 184*4];
> + unsigned int w[PAGEH/4 + 184];
> +} table = {{
> +#include "iswalpha_table.h"
> +}};
> +
> +const static unsigned short dict[87] = {
> +#include "iswalpha_dict.h"
> };
>
> -int iswalpha(wint_t wc)
> -{
> - if (wc<0x20000U)
> - return (table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1;
> - if (wc<0x2fffeU)
> - return 1;
> - return 0;
> +int iswalpha(wint_t wc) {
> + unsigned direct, page, bmap, shfr, lane, base, rev;
> + unsigned target, huff, type, popc, fast;
> + signed char ext;
> + if ((unsigned)wc >= 0x20000)
> + return (unsigned)wc < 0x2fffe;
> +
> + /* Direct path used in 220 of the 256 BMP code pages */
> + direct = wc < (PAGEH-PAGES)*8;
> + page = (wc >> PAGE_SH);
> + bmap = (wc >> 3) + PAGES;
> + base = table.b[direct ? bmap:page];
> + if (base <= direct*256 + 1)
> + return base >> (wc & -direct & 7) & 1;
> +
> + /* 2nd & 3rd level arrays: final level idx=popc^rev_direction */
> + target = wc & (PAGE_MAX-1);
> + shfr = (target & 15);
> + lane = (target >> 4);
> + huff = table.w[base += PAGEH/4 - 2]; /* base 0/1 reserved */
> + type = (huff >> (2 * lane)) & 3;
> + popc = (huff << (31 - 2 * lane));
> + popc = (popc & 0x11111111) + ((popc & 0x44444444) >> 2);
> + popc = (popc * 0x11111111) >> 28;
> + base+= (rev = -(page & 1)) + 1;
> + ext = (table.b + base*4)[(int)(popc^rev)];
> +
> + /* Dictionary lookup is only 1% of codepoints */
> + fast = (type != 2 | ext & 1);
> + if (fast) {
> + shfr = (shfr + 5) & -(type >= 2);
> + shfr = (shfr - (6 & -(type == 2)) + type);
> + return (ext << 8 | 0xfe) >> shfr & 1;
> + } else {
> + return dict[ext >> 1 & 0x7f] >> shfr & 1;
> + }
> }
>
> int __iswalpha_l(wint_t c, locale_t l)
> diff --git a/src/ctype/iswalpha_dict.h b/src/ctype/iswalpha_dict.h
> new file mode 100644
> index 0000000..ba3f1e0
> --- /dev/null
> +++ b/src/ctype/iswalpha_dict.h
> @@ -0,0 +1,15 @@
> +/* success: all odd pages matched to even < 0x1e2 */
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* wctype: dictionary 87 entries */
> +63871,8383,15743,32573,65341,65407,16191,40959,
> +4224,32767,128,4079,56832,1135,43775,24543,
> +8156,4047,32770,64644,15919,48464,17376,30751,
> +33023,32639,32768,224,1022,7998,57471,36848,
> +59644,64639,14336,32382,63615,57592,24447,64764,
> +7420,47103,57344,65295,64831,37311,61551,65263,
> +16543,48511,49151,40943,58349,6559,57473,1979,
> +33791,179,32575,16128,319,32743,64383,46207,
> +203,64959,379,34815,240,17407,56932,59327,
> +57311,31743,64607,65343,53239,2011,16256,2191,
> +65174,2807,60036,43670,63382,24311,64494,
> diff --git a/src/ctype/iswalpha_table.h b/src/ctype/iswalpha_table.h
> new file mode 100644
> index 0000000..3018209
> --- /dev/null
> +++ b/src/ctype/iswalpha_table.h
> @@ -0,0 +1,217 @@
> +/* success: all odd pages matched to even < 0x1e2 */
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* wctype: table 512 x 256 codepoints */
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 2, 1, 6, 5, 12, 1, 16, 11, 21, 30, 25, 40, 31, 43, 1, 36,
> + 37, 58, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 44, 15, 47, 0,
> + 51, 24, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 54, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 46,
> + 1, 1, 1, 1, 55, 1, 59, 94, 63, 20, 68, 50, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1,121, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 73,136, 1, 62, 76,100,
> + 79,103, 83, 67, 86, 75, 1,139, 90,142, 95, 72,101, 78, 0, 89,
> +104,108,109,146,113, 82,118,150,122,153,125, 0,128,117,133, 0,
> + 1, 1, 1,112,137,124, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 1, 1, 1, 1,140, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 1, 1,143, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,147,132, 0, 0,151,156,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1,127, 1, 1,154, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 1,159,157, 0, 0, 0, 0, 0, 0, 0, 0, 0,160, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,165,176,169,164, 0, 0, 0, 0, 0, 0, 0, 0,
> +173,168,177, 0, 0, 0, 0, 0,179, 85, 0, 0, 0, 0,181, 0,
> + 0,172, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0,254,255,255, 7,254,255,255, 7,
> + 0, 0, 0, 0, 0, 4, 32, 4,255,255,127,255,255,255,127,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,195,255, 3, 0, 31, 80, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0,223,188,
> + 64,215,255,255,251,255,255,255,255,255,255,255,255,255,191,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> + 3,252,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,254,255,255,255,127, 2,255,255,255,255,
> +255, 1, 0, 0, 0, 0,255,191,182, 0,255,255,255,135, 7, 0,
> + 0, 0,255, 7,255,255,255,255,255,255,255,254,255,195,255,255,
> +255,255,255,255,255,255,255,255,255,255,239, 31,254,225,255,159,
> + 0, 0,255,255,255,255,255,255, 0,224,255,255,255,255,255,255,
> +255,255,255,255,255,255, 3, 0,255,255,255,255,255, 7, 48, 4,
> +255,255,255,252,255, 31, 0, 0,255,255,255, 1,255, 7, 0, 0,
> + 0, 0, 0, 0,255,255,223, 63, 0, 0,240,255,248, 3,255,255,
> +255,255,255,255,255,255,255,239,255,223,225,255,207,255,254,255,
> +239,159,249,255,255,253,197,227,159, 89,128,176,207,255, 3, 16,
> +238,135,249,255,255,253,109,195,135, 25, 2, 94,192,255, 63, 0,
> +238,191,251,255,255,253,237,227,191, 27, 1, 0,207,255, 0, 30,
> +238,159,249,255,255,253,237,227,159, 25,192,176,207,255, 2, 0,
> +236,199, 61,214, 24,199,255,195,199, 29,129, 0,192,255, 0, 0,
> +239,223,253,255,255,253,255,227,223, 29, 96, 7,207,255, 0, 0,
> +239,223,253,255,255,253,239,227,223, 29, 96, 64,207,255, 6, 0,
> +239,223,253,255,255,255,255,231,223, 93,240,128,207,255, 0,252,
> +236,255,127,252,255,255,251, 47,127,128, 95,255,192,255, 12, 0,
> +254,255,255,255,255,127,255, 7, 63, 32,255, 3, 0, 0, 0, 0,
> +214,247,255,255,175,255,255, 59, 95, 32,255,243, 0, 0, 0, 0,
> + 1, 0, 0, 0,255, 3, 0, 0,255,254,255,255,255, 31,254,255,
> + 3,255,255,254,255,255,255, 31, 0, 0, 0, 0, 0, 0, 0, 0,
> +
> +/* wctype: pages U+01000,01300 [words -1-2] */
> +BYTES(0x96595695),0,7,127,2,239,12,15,8,BYTES(0x95510959),
> +
> +/* wctype: pages U+01200,01700 [words 3-8] */
> +BYTES(0x5a965a55),123,4,123,6,8,10,7,16,3,207,13,191,
> +15,15,15,191,BYTES(0x2ad5edde),
> +
> +/* wctype: pages U+01400,02d00 [words 9-12] */
> +BYTES(0x55555557),254,50,50,50,50,127,48,2,BYTES(0x5aad2565),
> +
> +/* wctype: pages U+01600,0a900 [words 13-17] */
> +BYTES(0xa55b6555),14,254,15,143,3,18,7,52,247,63,7,15,
> +BYTES(0x9ad59d65),
> +
> +/* wctype: pages U+01800,03100 [words 18-21] */
> +BYTES(0xd5659558),7,3,15,63,15,18,254,224,BYTES(0x409255d7),
> +
> +/* wctype: pages U+01a00,01900 [words 22-27] */
> +BYTES(0x002af959),31,18,254,31,7,7,20,7,7,31,31,127,
> +192,3,31,18,BYTES(0x0a65e7a9),
> +
> +/* wctype: pages U+01c00,01f00 [words 28-33] */
> +BYTES(0xa09696d5),127,199,127,3,207,24,26,32,63,34,32,30,
> +127,28,12,12,BYTES(0xaa959a59),
> +
> +/* wctype: pages U+02000,01b00 [words 34-37] */
> +BYTES(0x00088000),36,63,3,191,231,7,22,239,BYTES(0xf5650ad5),
> +
> +/* wctype: pages U+02400,01d00 [words 38-40] */
> +BYTES(0x25c00000),192,7,31,128,BYTES(0xf0555555),
> +
> +/* wctype: pages U+02c00,09f00 [words 41-43] */
> +BYTES(0xe5555965),18,18,46,12,BYTES(0x15555555),
> +
> +/* wctype: pages U+02e00,0ab00 [words 44-47] */
> +BYTES(0x00000020),52,7,15,1,239,50,126,70,BYTES(0xa555696e),
> +
> +/* wctype: pages U+03000,04d00 [words 48-51] */
> +BYTES(0x957957a2),54,56,58,254,60,254,239,63,BYTES(0x00d55555),
> +
> +/* wctype: pages U+0a400,02100 [words 52-55] */
> +BYTES(0x94025555),63,127,3,44,231,42,40,38,BYTES(0x000252aa),
> +
> +/* wctype: pages U+0a600,0fd00 [words 56-59] */
> +BYTES(0x1555a526),63,31,18,62,31,1,252,127,BYTES(0x825d5495),
> +
> +/* wctype: pages U+0a800,10300 [words 60-64] */
> +BYTES(0x8b55d527),191,1,15,47,7,64,62,86,127,15,15,84,
> +BYTES(0x0e599665),
> +
> +/* wctype: pages U+0aa00,10b00 [words 65-69] */
> +BYTES(0xdb959ad5),127,127,7,66,18,5,68,60,3,7,63,63,
> +BYTES(0x000dddd5),
> +
> +/* wctype: pages U+0fa00,10500 [words 70-72] */
> +BYTES(0x09556555),127,7,15,1,BYTES(0x00003565),
> +
> +/* wctype: pages U+0fe00,10d00 [words 73-75] */
> +BYTES(0x9555c000),223,63,7,1,BYTES(0x000000a5),
> +
> +/* wctype: pages U+10000,11500 [words 76-79] */
> +BYTES(0x95550aa6),223,10,82,127,127,15,118,116,BYTES(0x08950000),
> +
> +/* wctype: pages U+10200,1e900 [words 80-82] */
> +BYTES(0x0d590000),63,1,7,158,BYTES(0x00000a55),
> +
> +/* wctype: pages U+10400,10f00 [words 83-86] */
> +BYTES(0x99695555),127,7,86,31,127,63,20,63,BYTES(0xd0000369),
> +
> +/* wctype: pages U+10800,0a700 [words 87-91] */
> +BYTES(0xd009dd96),88,90,63,127,18,55,0,128,124,243,252,128,
> +BYTES(0xc356557c),
> +
> +/* wctype: pages U+10a00,0ff00 [words 92-97] */
> +BYTES(0x360990da),92,94,63,63,63,253,31,80,78,18,192,15,
> +254,15,254,7,BYTES(0x0a957bb8),
> +
> +/* wctype: pages U+10c00,10100 [words 98-100] */
> +BYTES(0xd5d50255),3,7,7,31,BYTES(0x0000d500),
> +
> +/* wctype: pages U+11000,11100 [words 101-105] */
> +BYTES(0xa4973355),63,192,252,3,3,7,0,47,30,71,112,199,
> +BYTES(0x0b55d7d5),
> +
> +/* wctype: pages U+11200,12300 [words 106-109] */
> +BYTES(0xa56a009d),251,96,98,100,3,3,7,7,BYTES(0x00095555),
> +
> +/* wctype: pages U+11400,11d00 [words 110-114] */
> +BYTES(0x0a550a55),110,112,114,7,7,132,18,130,7,128,126,124,
> +BYTES(0x002a6a96),
> +
> +/* wctype: pages U+11600,0d700 [words 115-118] */
> +BYTES(0x02950b95),18,17,7,120,7,31,72,15,BYTES(0x96755555),
> +
> +/* wctype: pages U+11800,12500 [words 119-121] */
> +BYTES(0xa5500095),3,7,52,15,BYTES(0x00000355),
> +
> +/* wctype: pages U+11a00,18700 [words 122-124] */
> +BYTES(0x95095495),122,65,3,1,BYTES(0x95555555),
> +
> +/* wctype: pages U+11c00,16b00 [words 125-129] */
> +BYTES(0x00edcb96),251,50,1,7,252,252,253,127,193,248,7,15,
> +BYTES(0x0001bb15),
> +
> +/* wctype: pages U+11e00,0fb00 [words 130-133] */
> +BYTES(0xd0000000),127,248,3,219,76,251,74,127,BYTES(0x5cd557ab),
> +
> +/* wctype: pages U+12400,10700 [words 134-136] */
> +BYTES(0x55552555),18,1,63,127,BYTES(0x00002dd5),
> +
> +/* wctype: pages U+13400,10900 [words 137-139] */
> +BYTES(0x00000025),18,129,7,63,BYTES(0x0095009d),
> +
> +/* wctype: pages U+14600,11300 [words 140-143] */
> +BYTES(0x00000355),127,15,108,106,104,251,249,102,BYTES(0x00003aae),
> +
> +/* wctype: pages U+16a00,11700 [words 144-147] */
> +BYTES(0x24002995),3,18,7,127,0,7,15,207,BYTES(0x000000a9),
> +
> +/* wctype: pages U+16e00,11900 [words 148-150] */
> +BYTES(0x00005500),0,26,249,249,BYTES(0x39600000),
> +
> +/* wctype: pages U+18a00,16f00 [words 151-153] */
> +BYTES(0xd5555555),7,11,48,134,BYTES(0x30065655),
> +
> +/* wctype: pages U+1b200,1b100 [words 154-156] */
> +BYTES(0x95555555),31,136,7,18,BYTES(0x55556c09),
> +
> +/* wctype: pages U+1bc00,1d700 [words 157-161] */
> +BYTES(0x000aa555),15,63,3,138,0,152,251,251,18,18,223,223,
> +BYTES(0x566666dd),
> +
> +/* wctype: pages U+1d400,1e100 [words 162-165] */
> +BYTES(0x57a95d55),223,191,140,215,239,138,156,63,BYTES(0x000002a5),
> +
> +/* wctype: pages U+1d600,1f100 [words 166-169] */
> +BYTES(0x9b655555),150,253,239,239,0,7,7,7,BYTES(0x00026640),
> +
> +/* wctype: pages U+1e000,1d500 [words 170-173] */
> +BYTES(0x0000002a),10,243,154,253,148,146,144,142,BYTES(0x55555e9a),
> +
> +/* wctype: pages U+1e200,1e800 [words 174-176] */
> +BYTES(0xa5000000),31,7,0,0,
> +/* wctype: pages U+1e800,1ee00 [words 176-178] */
> +BYTES(0x03555555),31,0,0,0,
> +/* wctype: pages U+1ee00,1ee00 [words 178-183] */
> +BYTES(0x00aaaaa7),239,160,162,164,166,168,170,247,31,172,31,0,
> +0,0,0,0,
> \ No newline at end of file
> diff --git a/src/ctype/iswpunct.c b/src/ctype/iswpunct.c
> index f0b9ea0..cff793b 100644
> --- a/src/ctype/iswpunct.c
> +++ b/src/ctype/iswpunct.c
> @@ -1,14 +1,60 @@
> #include <wctype.h>
> +#include <endian.h>
>
> -static const unsigned char table[] = {
> -#include "punct.h"
> +#define BYTE(x,y) ((x) >> (y^(__BYTE_ORDER == __BIG_ENDIAN ? 0:24)) & 255)
> +#define BYTES(x) BYTE(x,24),BYTE(x,16),BYTE(x,8),BYTE(x,0)
> +#define PAGE_SH 8
> +#define PAGE_MAX (1u << PAGE_SH)
> +#define PAGES (0x20000 / PAGE_MAX)
> +#define PAGEH (0x800/8 + PAGES) /* Direct map 0x800 codepts */
> +
> +const static union {
> + unsigned char b[PAGEH + 196*4];
> + unsigned int w[PAGEH/4 + 196];
> +} table = {{
> +#include "iswpunct_table.h"
> +}};
> +
> +const static unsigned short dict[74] = {
> +#include "iswpunct_dict.h"
> };
>
> -int iswpunct(wint_t wc)
> -{
> - if (wc<0x20000U)
> - return (table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1;
> - return 0;
> +int iswpunct(wint_t wc) {
> + unsigned direct, page, bmap, shfr, lane, base, rev;
> + unsigned target, huff, type, popc, fast;
> + signed char ext;
> + if ((unsigned)wc >= 0x20000)
> + return 0;
> +
> + /* Direct path used in 212 of the 256 BMP code pages */
> + direct = wc < (PAGEH-PAGES)*8;
> + page = (wc >> PAGE_SH);
> + bmap = (wc >> 3) + PAGES;
> + base = table.b[direct ? bmap:page];
> + if (base <= direct*256 + 1)
> + return base >> (wc & -direct & 7) & 1;
> +
> + /* 2nd & 3rd level arrays: final level idx=popc^rev_direction */
> + target = wc & (PAGE_MAX-1);
> + shfr = (target & 15);
> + lane = (target >> 4);
> + huff = table.w[base += PAGEH/4 - 2]; /* base 0/1 reserved */
> + type = (huff >> (2 * lane)) & 3;
> + popc = (huff << (31 - 2 * lane));
> + popc = (popc & 0x11111111) + ((popc & 0x44444444) >> 2);
> + popc = (popc * 0x11111111) >> 28;
> + base+= (rev = -(page & 1)) + 1;
> + ext = (table.b + base*4)[(int)(popc^rev)];
> +
> + /* Dictionary lookup is only 1% of codepoints */
> + fast = (type != 2 | (ext & 1) == 0);
> + if (fast) {
> + shfr = (shfr + 6) & -(type >= 2);
> + shfr = (shfr - (8 & -(type == 2)) + (type^1));
> + return (ext << 8 | 0x01) >> shfr & 1;
> + } else {
> + return dict[ext >> 1 & 0x7f] >> shfr & 1;
> + }
> }
>
> int __iswpunct_l(wint_t c, locale_t l)
> diff --git a/src/ctype/iswpunct_dict.h b/src/ctype/iswpunct_dict.h
> new file mode 100644
> index 0000000..41da56d
> --- /dev/null
> +++ b/src/ctype/iswpunct_dict.h
> @@ -0,0 +1,19 @@
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ec */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* success: all odd pages matched to even < 0x1f0 */
> +/* success: all odd pages matched to even < 0x1f2 */
> +/* success: all odd pages matched to even < 0x1f6 */
> +/* success: all odd pages matched to even < 0x1f8 */
> +/* success: all odd pages matched to even < 0x1fa */
> +/* wctype: dictionary 74 entries */
> +32767,19968,64519,28668,57347,253,2047,40960,
> +32512,1023,32768,57280,252,57343,1664,8191,
> +24576,12159,40928,16255,61452,63616,255,8703,
> +912,64480,64767,32753,891,49616,17071,48159,
> +65343,34784,65027,32769,64511,4095,65310,64513,
> +57537,28687,240,49168,5891,32776,16383,896,
> +3967,32639,36800,34560,511,63584,16320,8195,
> +16353,16224,8128,63556,26624,192,32792,128,
> +384,32771,45056,65151,1904,61439,64635,51175,
> +59391,1807,
> diff --git a/src/ctype/iswpunct_table.h b/src/ctype/iswpunct_table.h
> new file mode 100644
> index 0000000..8631fe2
> --- /dev/null
> +++ b/src/ctype/iswpunct_table.h
> @@ -0,0 +1,211 @@
> + 0, 0, 0, 0, 0, 0, 0, 0, 2, 25, 7, 48, 12, 6, 17, 11,
> + 21, 0, 0, 28, 26, 0, 29, 16, 32, 34, 35, 39, 40, 20, 0, 60,
> + 44, 56, 1, 1, 49, 1, 1, 1, 1, 1, 1, 64, 53,117, 57, 70,
> + 61,120, 65, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 68, 0, 71, 43, 75,124, 79, 78, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 83, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 67, 0, 82, 90,133,
> + 0, 86, 94, 89, 0,163, 0, 0, 98, 52,102,167,107, 0,111, 74,
> +114, 97,118, 93,121,128,125,137,129,170,134, 0,138,141,142,173,
> + 0, 0, 0, 0,145, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,148, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,151,106, 0, 0,154,176,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,157, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> +161,144,164,147, 0, 0,168,110, 1, 1,171, 0, 0, 0, 0, 0,
> + 0,150,174, 0, 0, 0, 0, 0,177,153, 0, 0,179,156,181, 0,
> +183,160,186, 1, 1, 1,189,113,191,101,194, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,254,255, 0,252, 1, 0, 0,248, 1, 0, 0,120,
> + 0, 0, 0, 0,255,251,223,251, 0, 0,128, 0, 0, 0,128, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 60, 0,252,255,224,175,255,255,
> +255,255,255,255,255,255,255,255,223,255,255,255,255,255, 32, 64,
> +176, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> +252, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,252, 0, 0, 0, 0,
> + 0,230,254,255,255,255, 0, 64, 73, 0, 0, 0, 0, 0, 24, 0,
> +255,255, 0,216, 0, 0, 0, 0, 0, 0, 0, 1, 0, 60, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16,224, 1, 30, 0, 96,
> +255,191, 0, 0, 0, 0, 0, 0,255, 7, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,248,207,227,
> +
> +/* wctype: pages U+00800,00d00 [words -1-3] */
> +BYTES(0x2c0008a8),6,64,1,3,8,5,16,8,19,17,15,48,
> +BYTES(0xc2008a80),
> +
> +/* wctype: pages U+00a00,00f00 [words 4-8] */
> +BYTES(0x8280c280),32,64,64,32,64,9,13,27,128,25,248,254,
> +BYTES(0x0a820067),
> +
> +/* wctype: pages U+00c00,01700 [words 9-13] */
> +BYTES(0x0283c203),16,64,128,16,32,64,19,35,252,48,112,16,
> +BYTES(0x8ac000cc),
> +
> +/* wctype: pages U+00e00,01d00 [words 14-17] */
> +BYTES(0x02800aa0),21,21,23,24,8,62,51,127,BYTES(0xb5000000),
> +
> +/* wctype: pages U+01000,00900 [words 18-22] */
> +BYTES(0x80080280),29,248,128,16,7,64,32,1,48,30,64,32,
> +BYTES(0x8280fe80),
> +
> +/* wctype: pages U+01400,01300 [words 23-25] */
> +BYTES(0x00000003),1,19,31,192,BYTES(0x00089800),
> +
> +/* wctype: pages U+01600,04d00 [words 26-28] */
> +BYTES(0x200b2000),33,1,48,112,BYTES(0x55000000),
> +
> +/* wctype: pages U+01800,01900 [words 29-31] */
> +BYTES(0x00000002),1,136,49,28,BYTES(0x58000380),
> +
> +/* wctype: pages U+01a00,01b00 [words 32-36] */
> +BYTES(0x00a0b008),128,1,37,39,1,41,64,24,31,248,16,16,
> +BYTES(0xb0209bc0),
> +
> +/* wctype: pages U+01c00,0a700 [words 37-40] */
> +BYTES(0xa6008080),43,128,45,47,49,12,3,127,BYTES(0x0002003d),
> +
> +/* wctype: pages U+02000,00b00 [words 41-45] */
> +BYTES(0xd452b966),43,53,1,223,55,1,1,13,64,11,64,32,
> +BYTES(0x82008280),
> +
> +/* wctype: pages U+02400,10900 [words 46-49] */
> +BYTES(0x60d55235),127,13,63,248,252,96,21,101,BYTES(0x5d800088),
> +
> +/* wctype: pages U+02c00,02100 [words 50-53] */
> +BYTES(0xa0000000),67,69,28,63,24,61,59,57,BYTES(0x555606aa),
> +
> +/* wctype: pages U+02e00,01f00 [words 54-57] */
> +BYTES(0xd5590165),1,73,15,33,192,192,9,15,BYTES(0xaa800000),
> +
> +/* wctype: pages U+03000,02b00 [words 58-61] */
> +BYTES(0x803800a6),77,79,81,60,1,16,65,207,BYTES(0x5559d555),
> +
> +/* wctype: pages U+03200,0fb00 [words 62-64] */
> +BYTES(0x55555559),1,3,252,4,BYTES(0x03c00020),
> +
> +/* wctype: pages U+0a400,02f00 [words 65-67] */
> +BYTES(0x83540000),127,128,75,63,BYTES(0x8d555555),
> +
> +/* wctype: pages U+0a600,10f00 [words 68-71] */
> +BYTES(0x8000a002),192,21,83,45,19,192,127,192,BYTES(0x00000b38),
> +
> +/* wctype: pages U+0a800,0ab00 [words 72-75] */
> +BYTES(0x920080a3),64,30,19,85,87,89,112,16,BYTES(0x20000800),
> +
> +/* wctype: pages U+0aa00,0fd00 [words 76-79] */
> +BYTES(0xcb808800),224,95,21,2,128,67,96,128,BYTES(0x80000080),
> +
> +/* wctype: pages U+0e000,10100 [words 80-83] */
> +BYTES(0x00000003),1,93,1,75,1,224,143,135,BYTES(0x943ac0d7),
> +
> +/* wctype: pages U+0f800,10300 [words 84-86] */
> +BYTES(0x80000000),21,1,21,15,BYTES(0x0c080030),
> +
> +/* wctype: pages U+0fe00,11300 [words 87-90] */
> +BYTES(0x80002d59),19,247,97,21,31,117,64,48,BYTES(0x0000e280),
> +
> +/* wctype: pages U+10200,11100 [words 91-94] */
> +BYTES(0x90000000),75,31,254,208,113,56,15,24,BYTES(0xfa00c3c0),
> +
> +/* wctype: pages U+10800,1f900 [words 95-98] */
> +BYTES(0x8030cc00),128,128,128,240,145,143,141,139,BYTES(0x56659556),
> +
> +/* wctype: pages U+10a00,16b00 [words 99-103] */
> +BYTES(0xe2088a80),103,105,105,192,192,2,107,127,0,3,240,48,
> +BYTES(0x00003b40),
> +
> +/* wctype: pages U+10c00,1d700 [words 104-107] */
> +BYTES(0x80000000),248,8,4,4,21,21,32,32,BYTES(0x032222cc),
> +
> +/* wctype: pages U+10e00,1f700 [words 108-110] */
> +BYTES(0x00009000),1,75,105,15,BYTES(0x2955d555),
> +
> +/* wctype: pages U+11000,02d00 [words 111-114] */
> +BYTES(0x0283be00),109,252,63,21,3,252,111,71,BYTES(0x00008000),
> +
> +/* wctype: pages U+11200,03100 [words 115-117] */
> +BYTES(0x20200080),115,4,12,15,BYTES(0x35040000),
> +
> +/* wctype: pages U+11400,0a900 [words 118-121] */
> +BYTES(0x03000a00),119,121,76,128,93,8,91,240,BYTES(0x0ac00820),
> +
> +/* wctype: pages U+11600,11500 [words 122-125] */
> +BYTES(0x00802380),21,14,31,123,0,0,45,21,BYTES(0x09800000),
> +
> +/* wctype: pages U+11800,0ff00 [words 126-130] */
> +BYTES(0xe0000080),28,248,7,96,99,63,240,1,240,1,248,254,
> +BYTES(0xa0003bbb),
> +
> +/* wctype: pages U+11a00,11700 [words 131-134] */
> +BYTES(0x00380280),125,45,190,7,0,0,248,16,BYTES(0x000000a0),
> +
> +/* wctype: pages U+11c00,11d00 [words 135-138] */
> +BYTES(0x0000eb80),21,62,248,31,3,0,127,52,BYTES(0x00080300),
> +
> +/* wctype: pages U+11e00,1d100 [words 139-141] */
> +BYTES(0x80000000),129,0,105,135,BYTES(0x25555565),
> +
> +/* wctype: pages U+12400,1d300 [words 142-144] */
> +BYTES(0x0000c000),31,0,105,127,BYTES(0x00009d55),
> +
> +/* wctype: pages U+13400,1e100 [words 145-147] */
> +BYTES(0x00000080),105,0,21,127,BYTES(0x000002c0),
> +
> +/* wctype: pages U+16a00,1e900 [words 148-150] */
> +BYTES(0xc0002000),128,63,128,137,BYTES(0x00000a00),
> +
> +/* wctype: pages U+16e00,1ed00 [words 151-153] */
> +BYTES(0x00090000),13,0,93,254,BYTES(0x00000097),
> +
> +/* wctype: pages U+1bc00,1f100 [words 154-157] */
> +BYTES(0x00380000),133,15,192,31,248,56,248,31,BYTES(0x70262216),
> +
> +/* wctype: pages U+1d000,10500 [words 158-160] */
> +BYTES(0xd5555555),63,0,0,21,BYTES(0x00002000),
> +
> +/* wctype: pages U+1d200,10b00 [words 161-164] */
> +BYTES(0xd0000355),63,15,0,252,60,254,254,252,BYTES(0x00288880),
> +
> +/* wctype: pages U+1d600,11900 [words 165-167] */
> +BYTES(0x8b000000),2,16,16,5,BYTES(0x30000000),
> +
> +/* wctype: pages U+1da00,11f00 [words 168-170] */
> +BYTES(0x003a5555),75,240,254,131,BYTES(0x95000000),
> +
> +/* wctype: pages U+1e200,16f00 [words 171-173] */
> +BYTES(0xa0000000),224,21,0,4,BYTES(0x30000000),
> +
> +/* wctype: pages U+1e800,1ec00 [words 174-176] */
> +BYTES(0x0f000000),128,127,0,0,
> +/* wctype: pages U+1ec00,1ee00 [words 176-178] */
> +BYTES(0x00d5c000),254,31,0,0,
> +/* wctype: pages U+1ee00,1f000 [words 178-180] */
> +BYTES(0xc0000000),3,0,0,0,
> +/* wctype: pages U+1f000,1f200 [words 180-183] */
> +BYTES(0xdfed5565),75,15,1,254,254,254,63,0,
> +/* wctype: pages U+1f200,1f600 [words 183-186] */
> +BYTES(0x00003e97),7,75,105,3,63,0,0,0,
> +/* wctype: pages U+1f600,1f800 [words 186-188] */
> +BYTES(0xad555555),63,31,13,0,
> +/* wctype: pages U+1f800,1fa00 [words 188-191] */
> +BYTES(0x00265a56),75,45,19,45,93,0,0,0,
> +/* wctype: pages U+1fa00,1fa00 [words 191-195] */
> +BYTES(0x000fad55),15,93,147,7,63,0,0,0,0,0,0,0,
> diff --git a/src/ctype/punct.h b/src/ctype/punct.h
> deleted file mode 100644
> index 6792947..0000000
> --- a/src/ctype/punct.h
> +++ /dev/null
> @@ -1,141 +0,0 @@
>
> -18,16,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,16,16,34,35,16,36,37,38,39,
>
> -40,41,42,43,16,44,45,46,17,17,47,17,17,17,17,17,17,48,49,50,51,52,53,54,55,17,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,56,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,57,16,58,59,60,61,62,63,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,64,16,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,65,16,16,66,16,67,68,
>
> -69,16,70,71,72,16,73,16,16,74,75,76,77,78,16,79,80,81,82,83,84,85,86,87,88,89,
>
> -90,91,16,92,93,94,95,16,16,16,16,96,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,97,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,98,99,16,16,100,101,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,102,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,103,104,105,106,16,16,107,108,17,17,109,16,16,16,16,16,16,110,111,16,
>
> -16,16,16,16,112,113,16,16,114,115,116,16,117,118,119,17,17,17,120,121,122,123,
> -124,16,16,16,16,
>
> -16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,254,255,0,252,1,0,0,248,1,
>
> -0,0,120,0,0,0,0,255,251,223,251,0,0,128,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,60,0,252,255,224,175,255,255,255,255,255,255,255,255,
>
> -255,255,223,255,255,255,255,255,32,64,176,0,0,0,0,0,0,0,0,0,0,0,0,0,64,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,252,0,0,0,0,0,230,254,255,255,255,0,64,73,0,0,0,0,0,24,0,255,255,0,216,
> -0,0,0,0,0,0,0,1,0,60,0,0,0,0,0,0,0,0,0,0,0,0,16,224,1,30,0,
>
> -96,255,191,0,0,0,0,0,0,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,248,207,
>
> -227,0,0,0,3,0,32,255,127,0,0,0,78,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,0,7,252,0,0,0,
>
> -0,0,0,0,0,0,16,0,32,30,0,48,0,1,0,0,0,0,0,0,0,0,16,0,32,0,0,0,0,252,111,0,0,0,
>
> -0,0,0,0,16,0,32,0,0,0,0,64,0,0,0,0,0,0,0,0,16,0,32,0,0,0,0,3,224,0,0,0,0,0,0,
>
> -0,16,0,32,0,0,0,0,253,0,0,0,0,0,0,0,0,0,0,32,0,0,0,0,255,7,16,0,0,0,0,0,0,0,0,
>
> -32,0,0,0,0,128,255,16,0,0,0,0,0,0,16,0,32,0,0,0,0,0,0,0,0,0,0,0,0,0,24,0,160,
>
> -0,127,0,0,255,3,0,0,0,0,0,0,0,0,0,4,0,0,0,0,16,0,0,0,0,0,0,128,0,128,192,223,
> -0,12,0,0,0,0,0,0,0,0,0,0,0,4,0,31,0,0,0,0,0,
>
> -0,254,255,255,255,0,252,255,255,0,0,0,0,0,0,0,0,252,0,0,0,0,0,0,192,255,223,
>
> -255,7,0,0,0,0,0,0,0,0,0,0,128,6,0,252,0,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,0,
>
> -0,0,8,0,0,0,0,0,0,0,0,0,0,0,224,255,255,255,31,0,0,255,3,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,96,0,0,1,0,0,24,0,0,0,0,0,0,0,0,0,56,0,0,0,0,16,0,0,0,112,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,48,0,0,254,127,47,0,0,255,3,255,127,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,14,49,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,0,0,0,196,255,255,255,
>
> -255,0,0,0,192,0,0,0,0,0,0,0,0,1,0,224,159,0,0,0,0,127,63,255,127,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,16,0,16,0,0,252,255,255,255,31,0,0,0,0,0,12,0,0,0,0,0,0,64,0,
>
> -12,240,0,0,0,0,0,0,128,248,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,255,0,255,255,
>
> -255,33,144,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,
>
> -127,0,224,251,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,160,3,224,0,224,0,
>
> -224,0,96,128,248,255,255,255,252,255,255,255,255,255,127,223,255,241,127,255,
>
> -127,0,0,255,255,255,255,0,0,255,255,255,255,1,0,123,3,208,193,175,66,0,12,31,
>
> -188,255,255,0,0,0,0,0,14,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,127,0,0,0,255,7,0,0,255,255,255,255,255,255,255,255,255,
> -255,63,0,0,0,0,0,0,252,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,207,255,255,255,
>
> -63,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,224,135,3,254,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,127,255,255,255,255,0,
>
> -0,0,0,0,0,255,255,255,251,255,255,255,255,255,255,255,255,255,255,15,0,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,63,0,0,0,255,15,30,255,255,255,1,252,193,224,0,0,0,0,
>
> -0,0,0,0,0,0,0,30,1,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -255,255,0,0,0,0,255,255,255,255,15,0,0,0,255,255,255,127,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> -255,255,255,
>
> -255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,
>
> -255,255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,0,0,0,
>
> -0,0,0,192,0,224,0,0,0,0,0,0,0,0,0,0,0,128,15,112,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -255,0,255,255,127,0,3,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -64,0,0,0,0,15,255,3,0,0,0,0,0,0,240,0,0,0,0,0,0,0,0,0,16,192,0,0,255,255,3,23,
>
> -0,0,0,0,0,248,0,0,0,0,8,128,0,0,0,0,0,0,0,0,0,0,8,0,255,63,0,192,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,240,0,0,128,3,0,0,0,0,0,0,0,128,2,0,0,192,0,0,67,0,0,0,0,0,
> -0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,56,0,
>
> -0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,2,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,252,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,48,255,255,255,3,255,255,255,255,255,255,247,
>
> -255,127,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,254,255,0,252,1,0,0,248,1,0,
>
> -0,248,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,127,127,0,48,135,255,255,255,255,255,
>
> -143,255,0,0,0,0,0,0,224,255,255,127,255,15,1,0,0,0,0,0,255,255,255,255,255,63,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,
>
> -15,0,0,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -128,255,0,0,128,255,0,0,0,0,128,255,0,0,0,0,0,0,0,0,0,248,0,0,192,143,0,0,0,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,48,255,255,252,255,255,255,255,255,0,0,0,0,
>
> -0,0,0,135,255,1,255,1,0,0,0,224,0,0,0,224,0,0,0,0,0,1,0,0,96,248,127,0,0,0,0,
>
> -0,0,0,0,254,0,0,0,255,0,0,0,255,0,0,0,30,0,254,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,0,0,0,0,0,0,0,0,0,0,0,
> -0,255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,224,127,0,0,0,192,255,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,192,63,252,255,63,0,0,128,3,0,0,0,0,0,0,254,3,32,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,24,0,15,0,0,0,0,0,56,0,0,0,0,0,0,0,0,0,225,63,0,232,254,255,31,0,0,
>
> -0,0,0,0,0,96,63,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,
>
> -24,0,32,0,0,192,31,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,68,
>
> -248,0,104,0,0,0,0,0,0,0,0,0,0,0,0,76,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,128,255,255,255,0,0,0,0,0,0,0,0,0,0,0,0,128,14,0,0,0,255,
> -31,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,8,0,252,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,7,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,24,128,255,0,0,0,0,0,
>
> -0,0,0,0,0,223,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,62,0,0,252,255,31,3,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,52,0,0,0,0,0,0,0,0,0,128,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,128,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
> -255,3,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,192,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,63,0,0,0,0,0,0,0,255,255,48,0,0,248,
>
> -3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,176,15,0,0,0,0,0,0,
>
> -0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> -255,255,255,255,255,255,255,255,255,255,255,255,255,63,
>
> -0,255,255,255,255,127,254,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,1,0,0,255,255,255,255,255,255,255,255,
>
> -63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,15,0,255,255,255,255,255,255,
>
> -255,255,255,255,127,0,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,8,0,0,0,8,0,0,32,0,0,0,32,0,0,128,
>
> -0,0,0,128,0,0,0,2,0,0,0,2,0,0,8,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,15,0,248,254,255,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,127,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,240,0,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,255,127,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,112,7,0,192,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,254,255,255,255,255,255,255,255,31,0,0,0,0,0,0,0,0,0,254,255,
>
> -255,255,255,255,255,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,255,255,255,255,255,
>
> -15,255,255,255,255,255,255,255,255,255,255,255,255,15,0,255,127,254,255,254,
>
> -255,254,255,255,255,63,0,255,31,255,255,255,255,0,0,0,252,0,0,0,28,0,0,0,252,
>
> -255,255,255,31,0,0,0,0,0,0,192,255,255,255,7,0,255,255,255,255,255,15,255,1,3,
> -0,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,63,0,255,31,255,7,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,15,0,255,255,255,255,255,255,255,255,255,255,255,1,
>
> -255,15,0,0,255,15,255,255,255,255,255,255,255,0,255,3,255,255,255,255,255,0,
>
> -255,255,255,63,0,0,0,0,0,0,0,0,0,0,255,239,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,123,252,255,255,255,255,231,199,255,255,255,231,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,15,0,255,63,15,7,7,0,63,0,
> -0,0,0,0,0,0,0,0,0,0,0,0,
> --
> 2.51.2
>
>
[-- Attachment #2: Type: text/html, Size: 61678 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47%
2026-01-23 13:20 [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47% Xan Phung
2026-01-23 13:40 ` Xan Phung
2026-01-30 12:39 ` Xan Phung
@ 2026-02-18 1:28 ` Xan Phung
2026-02-25 15:06 ` Rich Felker
2 siblings, 1 reply; 6+ messages in thread
From: Xan Phung @ 2026-02-18 1:28 UTC (permalink / raw)
To: Openwall musl
[-- Attachment #1: Type: text/plain, Size: 56560 bytes --]
Hi,
I haven't heard back on where V2 of this patch stands - has it been
rejected or is it still under evaluation?
Whilst waiting for a response, I have done the following benchmarks/speed
tests - which Szabolcs has previously expressed an interest in seeing.
My primary motivation for the patch was so that Unicode text processing
would only have a 4kb (not 8kb) overhead in static linking of iswalpha &
iswpunct.
But it looks like there is also a very nice performance benefit as well.
In my benchmarks, my code/data is faster than current musl code, because it
only needs a single memory access for 80-90% of code pages vs two memory
accesses needed for current musl iswalpha code, and also likely because it
has fewer cache misses (total data size is halved).
For the cycle times below, "iswalpha_base" is current musl baseline,
"iswalpha_small" is my Patch V2 code:
iswalpha_base (cp sample range 0-0xFFFF): chksum = -13446
22.80 cycles per lookup
iswalpha_small (cp sample range 0-0xFFFF): chksum = -13446
17.73 cycles per lookup
iswalpha_base (cp sample range 0-0xFFF): chksum = -13452
15.20 cycles per lookup
iswalpha_small (cp sample range 0-0xFFF): chksum = -13452
11.40 cycles per lookup
Benchmark notes:
- single threaded, compiled with gcc -Os (gcc v15.2.0)
- executed on Ryzen 7 8845HS CPU running on Linux porteux 6.16.7-porteux
- iswalpha lookup cycle time includes functional call overhead, and
averaged over 30 iterations
- inner loop code was the following:
t0 = timer_start();
for (int i = 0; i < ITERATIONS; i++) {
test_cp = (test_cp + (result & 3)) & 0x1FF;
result += iswalpha_small(((test_cp << (i&0x7)) + test_cp) & 0xFFFF);
}
t1 = timer_end();
On Sat, 24 Jan 2026 at 00:21, Xan Phung <xan.phung@gmail.com> wrote:
> Currently iswalpha and iswpunct have total text data size of over 8kb.
> A more efficient encoding has reduced the total size to 4.2kb, a 47%
> reduction.
>
> This encoding also features a speed improvement, with lookups for
> approx 220 of the 256 BMP code blocks done with just a single memory
> load.
>
> The remaining Unicode blocks are niche uses, and are accessed with
> three memory loads. Compared to the old musl encoding (which used a
> two level table) the topmost level remains much the same, providing
> offsets into code block units (256 codepoint granularity). The second
> level data uses fixed sizes, of one 32 bit word per codepage (where
> each 2 bit pair in word identifies a block of 16 codepoints as all 0,
> all 1, or mixed). The third level is a variable length series of
> extension bytes, indexed by the popcount of set high bits within the
> second level's 32 bit word. Popcount is used as not all 16 codepoint
> blocks need an extension byte. This popcount is calculated with
> nearly same latency as a 32 bit multiply (so it is comparable with
> the indexing speed of accessing a 2D array of non power of 2 size),
> but has the advantage of greater compactness for sparse data.
>
> Results have been tested against first 0x20000 codepoints, and match
> that returned by the pre-existing musl implementation.
>
> A new char table tool to replace the existing gen_ctype is provided
> separately. The new tool has similar command line use:
>
> gen_ctype [a|p] -- generates iswalpha & iswpunct table.h files
> gen_ctype [A|P] -- generates iswalpha & iswpunct dict.h files
>
> Signed-off-by: Xan Phung <xan.phung@gmail.com>
> ---
> src/ctype/alpha.h | 172 -----------------------------
> src/ctype/iswalpha.c | 62 +++++++++--
> src/ctype/iswalpha_dict.h | 15 +++
> src/ctype/iswalpha_table.h | 217 +++++++++++++++++++++++++++++++++++++
> src/ctype/iswpunct.c | 60 ++++++++--
> src/ctype/iswpunct_dict.h | 19 ++++
> src/ctype/iswpunct_table.h | 211 ++++++++++++++++++++++++++++++++++++
> src/ctype/punct.h | 141 ------------------------
> 8 files changed, 568 insertions(+), 329 deletions(-)
> delete mode 100644 src/ctype/alpha.h
> create mode 100644 src/ctype/iswalpha_dict.h
> create mode 100644 src/ctype/iswalpha_table.h
> create mode 100644 src/ctype/iswpunct_dict.h
> create mode 100644 src/ctype/iswpunct_table.h
> delete mode 100644 src/ctype/punct.h
>
> diff --git a/src/ctype/alpha.h b/src/ctype/alpha.h
> deleted file mode 100644
> index 4167f38..0000000
> --- a/src/ctype/alpha.h
> +++ /dev/null
> @@ -1,172 +0,0 @@
>
> -18,17,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,17,34,35,36,17,37,38,39,40,
>
> -41,42,43,44,17,45,46,47,16,16,48,16,16,16,16,16,16,16,49,50,51,16,52,53,16,16,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,54,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,55,17,17,17,17,56,17,57,58,59,60,61,62,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,63,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,64,65,17,66,67,
>
> -68,69,70,71,72,73,74,17,75,76,77,78,79,80,81,16,82,83,84,85,86,87,88,89,90,91,
>
> -92,93,16,94,95,96,16,17,17,17,97,98,99,16,16,16,16,16,16,16,16,16,16,17,17,17,
>
> -17,100,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,17,101,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,17,17,102,103,16,16,104,105,17,17,17,17,17,17,17,17,17,17,17,17,17,17,
>
> -17,17,17,17,17,17,17,17,17,106,17,17,107,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,
>
> -108,109,16,16,16,16,16,16,16,16,16,110,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,111,112,113,114,16,16,16,16,16,16,16,16,115,116,
>
> -117,16,16,16,16,16,118,119,16,16,16,16,120,16,16,121,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,
>
> -16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,254,255,255,7,254,
>
> -255,255,7,0,0,0,0,0,4,32,4,255,255,127,255,255,255,127,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,195,255,3,0,31,80,0,0,0,0,0,0,0,0,0,0,32,0,0,0,0,0,223,188,64,215,255,255,
>
> -251,255,255,255,255,255,255,255,255,255,191,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,3,252,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,254,255,255,255,127,2,255,255,255,
>
> -255,255,1,0,0,0,0,255,191,182,0,255,255,255,135,7,0,0,0,255,7,255,255,255,255,
>
> -255,255,255,254,255,195,255,255,255,255,255,255,255,255,255,255,255,255,239,
> -31,254,225,255,
>
> -159,0,0,255,255,255,255,255,255,0,224,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,3,0,255,255,255,255,255,7,48,4,255,255,255,252,255,31,0,0,255,255,255,
>
> -1,255,7,0,0,0,0,0,0,255,255,223,63,0,0,240,255,248,3,255,255,255,255,255,255,
>
> -255,255,255,239,255,223,225,255,207,255,254,255,239,159,249,255,255,253,197,
>
> -227,159,89,128,176,207,255,3,16,238,135,249,255,255,253,109,195,135,25,2,94,
>
> -192,255,63,0,238,191,251,255,255,253,237,227,191,27,1,0,207,255,0,30,238,159,
>
> -249,255,255,253,237,227,159,25,192,176,207,255,2,0,236,199,61,214,24,199,255,
>
> -195,199,29,129,0,192,255,0,0,239,223,253,255,255,253,255,227,223,29,96,7,207,
>
> -255,0,0,239,223,253,255,255,253,239,227,223,29,96,64,207,255,6,0,239,223,253,
>
> -255,255,255,255,231,223,93,240,128,207,255,0,252,236,255,127,252,255,255,251,
>
> -47,127,128,95,255,192,255,12,0,254,255,255,255,255,127,255,7,63,32,255,3,0,0,
> -0,0,214,247,255,255,175,255,255,59,95,32,255,243,0,0,0,
>
> -0,1,0,0,0,255,3,0,0,255,254,255,255,255,31,254,255,3,255,255,254,255,255,255,
>
> -31,0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,249,255,3,255,255,255,255,255,
>
> -255,255,255,255,63,255,255,255,255,191,32,255,255,255,255,255,247,255,255,255,
>
> -255,255,255,255,255,255,61,127,61,255,255,255,255,255,61,255,255,255,255,61,
>
> -127,61,255,127,255,255,255,255,255,255,255,61,255,255,255,255,255,255,255,255,
>
> -7,0,0,0,0,255,255,0,0,255,255,255,255,255,255,255,255,255,255,63,63,254,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,159,255,255,254,255,255,7,255,255,255,255,255,255,255,255,
>
> -255,199,255,1,255,223,15,0,255,255,15,0,255,255,15,0,255,223,13,0,255,255,255,
>
> -255,255,255,207,255,255,1,128,16,255,3,0,0,0,0,255,3,255,255,255,255,255,255,
>
> -255,255,255,255,255,1,255,255,255,255,255,7,255,255,255,255,255,255,255,255,
> -63,
>
> -0,255,255,255,127,255,15,255,1,192,255,255,255,255,63,31,0,255,255,255,255,
>
> -255,15,255,255,255,3,255,3,0,0,0,0,255,255,255,15,255,255,255,255,255,255,255,
>
> -127,254,255,31,0,255,3,255,3,128,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
>
> -255,239,255,239,15,255,3,0,0,0,0,255,255,255,255,255,243,255,255,255,255,255,
>
> -255,191,255,3,0,255,255,255,255,255,255,127,0,255,227,255,255,255,255,255,63,
>
> -255,1,255,255,255,255,255,231,0,0,0,0,0,222,111,4,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,
>
> -128,255,31,0,255,255,63,63,255,255,255,255,63,63,255,170,255,255,255,63,255,
>
> -255,255,255,255,255,223,95,220,31,207,15,255,31,220,31,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,2,128,0,0,255,31,0,0,0,0,0,0,0,0,0,0,0,0,132,252,47,62,80,189,255,243,
> -224,67,0,0,255,255,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,192,255,255,255,255,255,255,3,0,
>
> -0,255,255,255,255,255,127,255,255,255,255,255,127,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,31,120,12,0,255,255,255,255,191,32,255,
>
> -255,255,255,255,255,255,128,0,0,255,255,127,0,127,127,127,127,127,127,127,127,
>
> -255,255,255,255,0,0,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,224,0,0,0,254,3,62,31,254,255,255,255,255,255,255,255,255,255,127,224,254,
>
> -255,255,255,255,255,255,255,255,255,255,247,224,255,255,255,255,255,254,255,
>
> -255,255,255,255,255,255,255,255,255,127,0,0,255,255,255,7,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,63,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,31,0,0,
>
> -0,0,0,0,0,0,255,255,255,255,255,63,255,31,255,255,255,15,0,0,255,255,255,255,
>
> -255,127,240,143,255,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,
>
> -0,128,255,252,255,255,255,255,255,255,255,255,255,255,255,255,249,255,255,255,
>
> -255,255,255,124,0,0,0,0,0,128,255,191,255,255,255,255,0,0,0,255,255,255,255,
>
> -255,255,15,0,255,255,255,255,255,255,255,255,47,0,255,3,0,0,252,232,255,255,
>
> -255,255,255,7,255,255,255,255,7,0,255,255,255,31,255,255,255,255,255,255,247,
>
> -255,0,128,255,3,255,255,255,127,255,255,255,255,255,255,127,0,255,63,255,3,
>
> -255,255,127,252,255,255,255,255,255,255,255,127,5,0,0,56,255,255,60,0,126,126,
>
> -126,0,127,127,255,255,255,255,255,247,255,0,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,7,255,3,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,15,0,255,255,127,248,255,255,255,255,
> -255,
>
> -15,255,255,255,255,255,255,255,255,255,255,255,255,255,63,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,3,0,0,0,0,127,0,248,224,255,253,127,95,219,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,3,0,0,0,248,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,63,0,0,255,255,255,255,255,255,255,255,
>
> -252,255,255,255,255,255,255,0,0,0,0,0,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,223,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,31,0,0,255,3,
>
> -254,255,255,7,254,255,255,7,192,255,255,255,255,255,255,255,255,255,255,127,
>
> -252,252,252,28,0,0,0,0,255,239,255,255,127,255,255,183,255,63,255,63,0,0,0,0,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,7,0,0,0,0,0,0,0,0,
>
> -255,255,255,255,255,255,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,255,255,255,31,255,255,255,255,255,255,1,0,0,0,0,
>
> -0,255,255,255,255,0,224,255,255,255,7,255,255,255,255,255,7,255,255,255,63,
>
> -255,255,255,255,15,255,62,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,63,255,3,255,255,255,255,15,255,255,255,
>
> -255,15,255,255,255,255,255,0,255,255,255,255,255,255,15,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,0,255,255,63,0,255,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,63,253,255,255,255,255,191,145,255,255,63,0,255,255,
>
> -127,0,255,255,255,127,0,0,0,0,0,0,0,0,255,255,55,0,255,255,63,0,255,255,255,3,
>
> -0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,192,0,0,0,0,0,0,0,0,111,240,239,
>
> -254,255,255,63,0,0,0,0,0,255,255,255,31,255,255,255,31,0,0,0,0,255,254,255,
>
> -255,31,0,0,0,255,255,255,255,255,255,63,0,255,255,63,0,255,255,7,0,255,255,3,
> -0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,1,0,0,0,0,0,0,255,255,255,255,255,255,7,
>
> -0,255,255,255,255,255,255,7,0,255,255,255,255,255,0,255,3,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,31,128,0,255,255,63,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,255,255,127,0,255,255,255,255,255,255,255,255,63,0,0,0,
>
> -192,255,0,0,252,255,255,255,255,255,255,1,0,0,255,255,255,1,255,3,255,255,255,
>
> -255,255,255,199,255,112,0,255,255,255,255,71,0,255,255,255,255,255,255,255,
>
> -255,30,0,255,23,0,0,0,0,255,255,251,255,255,255,159,64,0,0,0,0,0,0,0,0,127,
>
> -189,255,191,255,1,255,255,255,255,255,255,255,1,255,3,239,159,249,255,255,253,
>
> -237,227,159,25,129,224,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,
>
> -255,255,255,255,255,187,7,255,131,0,0,0,0,255,255,255,255,255,255,255,255,179,
> -0,255,3,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,63,127,0,0,0,63,0,0,
>
> -0,0,255,255,255,255,255,255,255,127,17,0,255,3,0,0,0,0,255,255,255,255,255,
>
> -255,63,1,255,3,0,0,0,0,0,0,255,255,255,231,255,7,255,3,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,3,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,255,252,255,255,255,255,255,252,26,0,0,0,255,255,255,255,255,255,231,
>
> -127,0,0,255,255,255,255,255,255,255,255,255,32,0,0,0,0,255,255,255,255,255,
>
> -255,255,1,255,253,255,255,255,255,127,127,1,0,255,3,0,0,252,255,255,255,252,
>
> -255,255,254,127,0,0,0,0,0,0,0,0,0,127,251,255,255,255,255,127,180,203,0,255,3,
> -191,253,255,255,255,127,123,1,255,3,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,127,0,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,3,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,255,255,127,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,255,255,255,255,1,255,255,255,127,255,3,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,63,0,0,255,255,255,255,255,255,0,0,15,0,255,3,248,255,255,224,255,255,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,255,255,255,255,135,255,255,255,255,255,255,255,128,
>
> -255,255,0,0,0,0,0,0,0,0,11,0,0,0,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,7,0,255,255,255,127,0,0,0,0,0,
>
> -0,7,0,240,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,15,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,7,255,31,255,1,255,67,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,255,255,255,255,255,223,255,255,255,255,255,255,255,
> -255,223,100,222,255,235,239,255,255,255,255,255,255,
>
> -255,191,231,223,223,255,255,255,123,95,252,253,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,63,255,255,255,
>
> -253,255,255,247,255,255,255,247,255,255,223,255,255,255,223,255,255,127,255,
>
> -255,255,127,255,255,255,253,255,255,255,253,255,255,247,207,255,255,255,255,
>
> -255,255,127,255,255,249,219,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,255,255,255,255,31,128,63,255,67,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
>
> -15,255,3,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,31,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,
> -143,8,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,239,255,255,255,150,254,247,10,132,234,150,170,150,247,247,94,255,251,255,
>
> -15,238,251,255,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,3,255,255,255,3,255,
> -255,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> diff --git a/src/ctype/iswalpha.c b/src/ctype/iswalpha.c
> index 1c5485d..d92c2e3 100644
> --- a/src/ctype/iswalpha.c
> +++ b/src/ctype/iswalpha.c
> @@ -1,16 +1,60 @@
> #include <wctype.h>
> +#include <endian.h>
>
> -static const unsigned char table[] = {
> -#include "alpha.h"
> +#define BYTE(x,y) ((x) >> (y^(__BYTE_ORDER == __BIG_ENDIAN ? 0:24)) & 255)
> +#define BYTES(x) BYTE(x,24),BYTE(x,16),BYTE(x,8),BYTE(x,0)
> +#define PAGE_SH 8
> +#define PAGE_MAX (1u << PAGE_SH)
> +#define PAGES (0x20000 / PAGE_MAX)
> +#define PAGEH (0x1000/8 + PAGES) /* Direct map 0x1000 codepts */
> +
> +const static union {
> + unsigned char b[PAGEH + 184*4];
> + unsigned int w[PAGEH/4 + 184];
> +} table = {{
> +#include "iswalpha_table.h"
> +}};
> +
> +const static unsigned short dict[87] = {
> +#include "iswalpha_dict.h"
> };
>
> -int iswalpha(wint_t wc)
> -{
> - if (wc<0x20000U)
> - return (table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1;
> - if (wc<0x2fffeU)
> - return 1;
> - return 0;
> +int iswalpha(wint_t wc) {
> + unsigned direct, page, bmap, shfr, lane, base, rev;
> + unsigned target, huff, type, popc, fast;
> + signed char ext;
> + if ((unsigned)wc >= 0x20000)
> + return (unsigned)wc < 0x2fffe;
> +
> + /* Direct path used in 220 of the 256 BMP code pages */
> + direct = wc < (PAGEH-PAGES)*8;
> + page = (wc >> PAGE_SH);
> + bmap = (wc >> 3) + PAGES;
> + base = table.b[direct ? bmap:page];
> + if (base <= direct*256 + 1)
> + return base >> (wc & -direct & 7) & 1;
> +
> + /* 2nd & 3rd level arrays: final level idx=popc^rev_direction */
> + target = wc & (PAGE_MAX-1);
> + shfr = (target & 15);
> + lane = (target >> 4);
> + huff = table.w[base += PAGEH/4 - 2]; /* base 0/1 reserved */
> + type = (huff >> (2 * lane)) & 3;
> + popc = (huff << (31 - 2 * lane));
> + popc = (popc & 0x11111111) + ((popc & 0x44444444) >> 2);
> + popc = (popc * 0x11111111) >> 28;
> + base+= (rev = -(page & 1)) + 1;
> + ext = (table.b + base*4)[(int)(popc^rev)];
> +
> + /* Dictionary lookup is only 1% of codepoints */
> + fast = (type != 2 | ext & 1);
> + if (fast) {
> + shfr = (shfr + 5) & -(type >= 2);
> + shfr = (shfr - (6 & -(type == 2)) + type);
> + return (ext << 8 | 0xfe) >> shfr & 1;
> + } else {
> + return dict[ext >> 1 & 0x7f] >> shfr & 1;
> + }
> }
>
> int __iswalpha_l(wint_t c, locale_t l)
> diff --git a/src/ctype/iswalpha_dict.h b/src/ctype/iswalpha_dict.h
> new file mode 100644
> index 0000000..ba3f1e0
> --- /dev/null
> +++ b/src/ctype/iswalpha_dict.h
> @@ -0,0 +1,15 @@
> +/* success: all odd pages matched to even < 0x1e2 */
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* wctype: dictionary 87 entries */
> +63871,8383,15743,32573,65341,65407,16191,40959,
> +4224,32767,128,4079,56832,1135,43775,24543,
> +8156,4047,32770,64644,15919,48464,17376,30751,
> +33023,32639,32768,224,1022,7998,57471,36848,
> +59644,64639,14336,32382,63615,57592,24447,64764,
> +7420,47103,57344,65295,64831,37311,61551,65263,
> +16543,48511,49151,40943,58349,6559,57473,1979,
> +33791,179,32575,16128,319,32743,64383,46207,
> +203,64959,379,34815,240,17407,56932,59327,
> +57311,31743,64607,65343,53239,2011,16256,2191,
> +65174,2807,60036,43670,63382,24311,64494,
> diff --git a/src/ctype/iswalpha_table.h b/src/ctype/iswalpha_table.h
> new file mode 100644
> index 0000000..3018209
> --- /dev/null
> +++ b/src/ctype/iswalpha_table.h
> @@ -0,0 +1,217 @@
> +/* success: all odd pages matched to even < 0x1e2 */
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* wctype: table 512 x 256 codepoints */
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 2, 1, 6, 5, 12, 1, 16, 11, 21, 30, 25, 40, 31, 43, 1, 36,
> + 37, 58, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 44, 15, 47, 0,
> + 51, 24, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 54, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 46,
> + 1, 1, 1, 1, 55, 1, 59, 94, 63, 20, 68, 50, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1,121, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 73,136, 1, 62, 76,100,
> + 79,103, 83, 67, 86, 75, 1,139, 90,142, 95, 72,101, 78, 0, 89,
> +104,108,109,146,113, 82,118,150,122,153,125, 0,128,117,133, 0,
> + 1, 1, 1,112,137,124, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 1, 1, 1, 1,140, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 1, 1,143, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,147,132, 0, 0,151,156,
> + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> + 1, 1, 1, 1, 1, 1, 1,127, 1, 1,154, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 1,159,157, 0, 0, 0, 0, 0, 0, 0, 0, 0,160, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,165,176,169,164, 0, 0, 0, 0, 0, 0, 0, 0,
> +173,168,177, 0, 0, 0, 0, 0,179, 85, 0, 0, 0, 0,181, 0,
> + 0,172, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0,254,255,255, 7,254,255,255, 7,
> + 0, 0, 0, 0, 0, 4, 32, 4,255,255,127,255,255,255,127,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,255,255,195,255, 3, 0, 31, 80, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0,223,188,
> + 64,215,255,255,251,255,255,255,255,255,255,255,255,255,191,255,
> +255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> + 3,252,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> +255,255,255,255,255,255,254,255,255,255,127, 2,255,255,255,255,
> +255, 1, 0, 0, 0, 0,255,191,182, 0,255,255,255,135, 7, 0,
> + 0, 0,255, 7,255,255,255,255,255,255,255,254,255,195,255,255,
> +255,255,255,255,255,255,255,255,255,255,239, 31,254,225,255,159,
> + 0, 0,255,255,255,255,255,255, 0,224,255,255,255,255,255,255,
> +255,255,255,255,255,255, 3, 0,255,255,255,255,255, 7, 48, 4,
> +255,255,255,252,255, 31, 0, 0,255,255,255, 1,255, 7, 0, 0,
> + 0, 0, 0, 0,255,255,223, 63, 0, 0,240,255,248, 3,255,255,
> +255,255,255,255,255,255,255,239,255,223,225,255,207,255,254,255,
> +239,159,249,255,255,253,197,227,159, 89,128,176,207,255, 3, 16,
> +238,135,249,255,255,253,109,195,135, 25, 2, 94,192,255, 63, 0,
> +238,191,251,255,255,253,237,227,191, 27, 1, 0,207,255, 0, 30,
> +238,159,249,255,255,253,237,227,159, 25,192,176,207,255, 2, 0,
> +236,199, 61,214, 24,199,255,195,199, 29,129, 0,192,255, 0, 0,
> +239,223,253,255,255,253,255,227,223, 29, 96, 7,207,255, 0, 0,
> +239,223,253,255,255,253,239,227,223, 29, 96, 64,207,255, 6, 0,
> +239,223,253,255,255,255,255,231,223, 93,240,128,207,255, 0,252,
> +236,255,127,252,255,255,251, 47,127,128, 95,255,192,255, 12, 0,
> +254,255,255,255,255,127,255, 7, 63, 32,255, 3, 0, 0, 0, 0,
> +214,247,255,255,175,255,255, 59, 95, 32,255,243, 0, 0, 0, 0,
> + 1, 0, 0, 0,255, 3, 0, 0,255,254,255,255,255, 31,254,255,
> + 3,255,255,254,255,255,255, 31, 0, 0, 0, 0, 0, 0, 0, 0,
> +
> +/* wctype: pages U+01000,01300 [words -1-2] */
> +BYTES(0x96595695),0,7,127,2,239,12,15,8,BYTES(0x95510959),
> +
> +/* wctype: pages U+01200,01700 [words 3-8] */
> +BYTES(0x5a965a55),123,4,123,6,8,10,7,16,3,207,13,191,
> +15,15,15,191,BYTES(0x2ad5edde),
> +
> +/* wctype: pages U+01400,02d00 [words 9-12] */
> +BYTES(0x55555557),254,50,50,50,50,127,48,2,BYTES(0x5aad2565),
> +
> +/* wctype: pages U+01600,0a900 [words 13-17] */
> +BYTES(0xa55b6555),14,254,15,143,3,18,7,52,247,63,7,15,
> +BYTES(0x9ad59d65),
> +
> +/* wctype: pages U+01800,03100 [words 18-21] */
> +BYTES(0xd5659558),7,3,15,63,15,18,254,224,BYTES(0x409255d7),
> +
> +/* wctype: pages U+01a00,01900 [words 22-27] */
> +BYTES(0x002af959),31,18,254,31,7,7,20,7,7,31,31,127,
> +192,3,31,18,BYTES(0x0a65e7a9),
> +
> +/* wctype: pages U+01c00,01f00 [words 28-33] */
> +BYTES(0xa09696d5),127,199,127,3,207,24,26,32,63,34,32,30,
> +127,28,12,12,BYTES(0xaa959a59),
> +
> +/* wctype: pages U+02000,01b00 [words 34-37] */
> +BYTES(0x00088000),36,63,3,191,231,7,22,239,BYTES(0xf5650ad5),
> +
> +/* wctype: pages U+02400,01d00 [words 38-40] */
> +BYTES(0x25c00000),192,7,31,128,BYTES(0xf0555555),
> +
> +/* wctype: pages U+02c00,09f00 [words 41-43] */
> +BYTES(0xe5555965),18,18,46,12,BYTES(0x15555555),
> +
> +/* wctype: pages U+02e00,0ab00 [words 44-47] */
> +BYTES(0x00000020),52,7,15,1,239,50,126,70,BYTES(0xa555696e),
> +
> +/* wctype: pages U+03000,04d00 [words 48-51] */
> +BYTES(0x957957a2),54,56,58,254,60,254,239,63,BYTES(0x00d55555),
> +
> +/* wctype: pages U+0a400,02100 [words 52-55] */
> +BYTES(0x94025555),63,127,3,44,231,42,40,38,BYTES(0x000252aa),
> +
> +/* wctype: pages U+0a600,0fd00 [words 56-59] */
> +BYTES(0x1555a526),63,31,18,62,31,1,252,127,BYTES(0x825d5495),
> +
> +/* wctype: pages U+0a800,10300 [words 60-64] */
> +BYTES(0x8b55d527),191,1,15,47,7,64,62,86,127,15,15,84,
> +BYTES(0x0e599665),
> +
> +/* wctype: pages U+0aa00,10b00 [words 65-69] */
> +BYTES(0xdb959ad5),127,127,7,66,18,5,68,60,3,7,63,63,
> +BYTES(0x000dddd5),
> +
> +/* wctype: pages U+0fa00,10500 [words 70-72] */
> +BYTES(0x09556555),127,7,15,1,BYTES(0x00003565),
> +
> +/* wctype: pages U+0fe00,10d00 [words 73-75] */
> +BYTES(0x9555c000),223,63,7,1,BYTES(0x000000a5),
> +
> +/* wctype: pages U+10000,11500 [words 76-79] */
> +BYTES(0x95550aa6),223,10,82,127,127,15,118,116,BYTES(0x08950000),
> +
> +/* wctype: pages U+10200,1e900 [words 80-82] */
> +BYTES(0x0d590000),63,1,7,158,BYTES(0x00000a55),
> +
> +/* wctype: pages U+10400,10f00 [words 83-86] */
> +BYTES(0x99695555),127,7,86,31,127,63,20,63,BYTES(0xd0000369),
> +
> +/* wctype: pages U+10800,0a700 [words 87-91] */
> +BYTES(0xd009dd96),88,90,63,127,18,55,0,128,124,243,252,128,
> +BYTES(0xc356557c),
> +
> +/* wctype: pages U+10a00,0ff00 [words 92-97] */
> +BYTES(0x360990da),92,94,63,63,63,253,31,80,78,18,192,15,
> +254,15,254,7,BYTES(0x0a957bb8),
> +
> +/* wctype: pages U+10c00,10100 [words 98-100] */
> +BYTES(0xd5d50255),3,7,7,31,BYTES(0x0000d500),
> +
> +/* wctype: pages U+11000,11100 [words 101-105] */
> +BYTES(0xa4973355),63,192,252,3,3,7,0,47,30,71,112,199,
> +BYTES(0x0b55d7d5),
> +
> +/* wctype: pages U+11200,12300 [words 106-109] */
> +BYTES(0xa56a009d),251,96,98,100,3,3,7,7,BYTES(0x00095555),
> +
> +/* wctype: pages U+11400,11d00 [words 110-114] */
> +BYTES(0x0a550a55),110,112,114,7,7,132,18,130,7,128,126,124,
> +BYTES(0x002a6a96),
> +
> +/* wctype: pages U+11600,0d700 [words 115-118] */
> +BYTES(0x02950b95),18,17,7,120,7,31,72,15,BYTES(0x96755555),
> +
> +/* wctype: pages U+11800,12500 [words 119-121] */
> +BYTES(0xa5500095),3,7,52,15,BYTES(0x00000355),
> +
> +/* wctype: pages U+11a00,18700 [words 122-124] */
> +BYTES(0x95095495),122,65,3,1,BYTES(0x95555555),
> +
> +/* wctype: pages U+11c00,16b00 [words 125-129] */
> +BYTES(0x00edcb96),251,50,1,7,252,252,253,127,193,248,7,15,
> +BYTES(0x0001bb15),
> +
> +/* wctype: pages U+11e00,0fb00 [words 130-133] */
> +BYTES(0xd0000000),127,248,3,219,76,251,74,127,BYTES(0x5cd557ab),
> +
> +/* wctype: pages U+12400,10700 [words 134-136] */
> +BYTES(0x55552555),18,1,63,127,BYTES(0x00002dd5),
> +
> +/* wctype: pages U+13400,10900 [words 137-139] */
> +BYTES(0x00000025),18,129,7,63,BYTES(0x0095009d),
> +
> +/* wctype: pages U+14600,11300 [words 140-143] */
> +BYTES(0x00000355),127,15,108,106,104,251,249,102,BYTES(0x00003aae),
> +
> +/* wctype: pages U+16a00,11700 [words 144-147] */
> +BYTES(0x24002995),3,18,7,127,0,7,15,207,BYTES(0x000000a9),
> +
> +/* wctype: pages U+16e00,11900 [words 148-150] */
> +BYTES(0x00005500),0,26,249,249,BYTES(0x39600000),
> +
> +/* wctype: pages U+18a00,16f00 [words 151-153] */
> +BYTES(0xd5555555),7,11,48,134,BYTES(0x30065655),
> +
> +/* wctype: pages U+1b200,1b100 [words 154-156] */
> +BYTES(0x95555555),31,136,7,18,BYTES(0x55556c09),
> +
> +/* wctype: pages U+1bc00,1d700 [words 157-161] */
> +BYTES(0x000aa555),15,63,3,138,0,152,251,251,18,18,223,223,
> +BYTES(0x566666dd),
> +
> +/* wctype: pages U+1d400,1e100 [words 162-165] */
> +BYTES(0x57a95d55),223,191,140,215,239,138,156,63,BYTES(0x000002a5),
> +
> +/* wctype: pages U+1d600,1f100 [words 166-169] */
> +BYTES(0x9b655555),150,253,239,239,0,7,7,7,BYTES(0x00026640),
> +
> +/* wctype: pages U+1e000,1d500 [words 170-173] */
> +BYTES(0x0000002a),10,243,154,253,148,146,144,142,BYTES(0x55555e9a),
> +
> +/* wctype: pages U+1e200,1e800 [words 174-176] */
> +BYTES(0xa5000000),31,7,0,0,
> +/* wctype: pages U+1e800,1ee00 [words 176-178] */
> +BYTES(0x03555555),31,0,0,0,
> +/* wctype: pages U+1ee00,1ee00 [words 178-183] */
> +BYTES(0x00aaaaa7),239,160,162,164,166,168,170,247,31,172,31,0,
> +0,0,0,0,
> \ No newline at end of file
> diff --git a/src/ctype/iswpunct.c b/src/ctype/iswpunct.c
> index f0b9ea0..cff793b 100644
> --- a/src/ctype/iswpunct.c
> +++ b/src/ctype/iswpunct.c
> @@ -1,14 +1,60 @@
> #include <wctype.h>
> +#include <endian.h>
>
> -static const unsigned char table[] = {
> -#include "punct.h"
> +#define BYTE(x,y) ((x) >> (y^(__BYTE_ORDER == __BIG_ENDIAN ? 0:24)) & 255)
> +#define BYTES(x) BYTE(x,24),BYTE(x,16),BYTE(x,8),BYTE(x,0)
> +#define PAGE_SH 8
> +#define PAGE_MAX (1u << PAGE_SH)
> +#define PAGES (0x20000 / PAGE_MAX)
> +#define PAGEH (0x800/8 + PAGES) /* Direct map 0x800 codepts */
> +
> +const static union {
> + unsigned char b[PAGEH + 196*4];
> + unsigned int w[PAGEH/4 + 196];
> +} table = {{
> +#include "iswpunct_table.h"
> +}};
> +
> +const static unsigned short dict[74] = {
> +#include "iswpunct_dict.h"
> };
>
> -int iswpunct(wint_t wc)
> -{
> - if (wc<0x20000U)
> - return (table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1;
> - return 0;
> +int iswpunct(wint_t wc) {
> + unsigned direct, page, bmap, shfr, lane, base, rev;
> + unsigned target, huff, type, popc, fast;
> + signed char ext;
> + if ((unsigned)wc >= 0x20000)
> + return 0;
> +
> + /* Direct path used in 212 of the 256 BMP code pages */
> + direct = wc < (PAGEH-PAGES)*8;
> + page = (wc >> PAGE_SH);
> + bmap = (wc >> 3) + PAGES;
> + base = table.b[direct ? bmap:page];
> + if (base <= direct*256 + 1)
> + return base >> (wc & -direct & 7) & 1;
> +
> + /* 2nd & 3rd level arrays: final level idx=popc^rev_direction */
> + target = wc & (PAGE_MAX-1);
> + shfr = (target & 15);
> + lane = (target >> 4);
> + huff = table.w[base += PAGEH/4 - 2]; /* base 0/1 reserved */
> + type = (huff >> (2 * lane)) & 3;
> + popc = (huff << (31 - 2 * lane));
> + popc = (popc & 0x11111111) + ((popc & 0x44444444) >> 2);
> + popc = (popc * 0x11111111) >> 28;
> + base+= (rev = -(page & 1)) + 1;
> + ext = (table.b + base*4)[(int)(popc^rev)];
> +
> + /* Dictionary lookup is only 1% of codepoints */
> + fast = (type != 2 | (ext & 1) == 0);
> + if (fast) {
> + shfr = (shfr + 6) & -(type >= 2);
> + shfr = (shfr - (8 & -(type == 2)) + (type^1));
> + return (ext << 8 | 0x01) >> shfr & 1;
> + } else {
> + return dict[ext >> 1 & 0x7f] >> shfr & 1;
> + }
> }
>
> int __iswpunct_l(wint_t c, locale_t l)
> diff --git a/src/ctype/iswpunct_dict.h b/src/ctype/iswpunct_dict.h
> new file mode 100644
> index 0000000..41da56d
> --- /dev/null
> +++ b/src/ctype/iswpunct_dict.h
> @@ -0,0 +1,19 @@
> +/* success: all odd pages matched to even < 0x1e8 */
> +/* success: all odd pages matched to even < 0x1ec */
> +/* success: all odd pages matched to even < 0x1ee */
> +/* success: all odd pages matched to even < 0x1f0 */
> +/* success: all odd pages matched to even < 0x1f2 */
> +/* success: all odd pages matched to even < 0x1f6 */
> +/* success: all odd pages matched to even < 0x1f8 */
> +/* success: all odd pages matched to even < 0x1fa */
> +/* wctype: dictionary 74 entries */
> +32767,19968,64519,28668,57347,253,2047,40960,
> +32512,1023,32768,57280,252,57343,1664,8191,
> +24576,12159,40928,16255,61452,63616,255,8703,
> +912,64480,64767,32753,891,49616,17071,48159,
> +65343,34784,65027,32769,64511,4095,65310,64513,
> +57537,28687,240,49168,5891,32776,16383,896,
> +3967,32639,36800,34560,511,63584,16320,8195,
> +16353,16224,8128,63556,26624,192,32792,128,
> +384,32771,45056,65151,1904,61439,64635,51175,
> +59391,1807,
> diff --git a/src/ctype/iswpunct_table.h b/src/ctype/iswpunct_table.h
> new file mode 100644
> index 0000000..8631fe2
> --- /dev/null
> +++ b/src/ctype/iswpunct_table.h
> @@ -0,0 +1,211 @@
> + 0, 0, 0, 0, 0, 0, 0, 0, 2, 25, 7, 48, 12, 6, 17, 11,
> + 21, 0, 0, 28, 26, 0, 29, 16, 32, 34, 35, 39, 40, 20, 0, 60,
> + 44, 56, 1, 1, 49, 1, 1, 1, 1, 1, 1, 64, 53,117, 57, 70,
> + 61,120, 65, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 68, 0, 71, 43, 75,124, 79, 78, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 83, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 67, 0, 82, 90,133,
> + 0, 86, 94, 89, 0,163, 0, 0, 98, 52,102,167,107, 0,111, 74,
> +114, 97,118, 93,121,128,125,137,129,170,134, 0,138,141,142,173,
> + 0, 0, 0, 0,145, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,148, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,151,106, 0, 0,154,176,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,157, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> +161,144,164,147, 0, 0,168,110, 1, 1,171, 0, 0, 0, 0, 0,
> + 0,150,174, 0, 0, 0, 0, 0,177,153, 0, 0,179,156,181, 0,
> +183,160,186, 1, 1, 1,189,113,191,101,194, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0,254,255, 0,252, 1, 0, 0,248, 1, 0, 0,120,
> + 0, 0, 0, 0,255,251,223,251, 0, 0,128, 0, 0, 0,128, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 60, 0,252,255,224,175,255,255,
> +255,255,255,255,255,255,255,255,223,255,255,255,255,255, 32, 64,
> +176, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> +252, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,252, 0, 0, 0, 0,
> + 0,230,254,255,255,255, 0, 64, 73, 0, 0, 0, 0, 0, 24, 0,
> +255,255, 0,216, 0, 0, 0, 0, 0, 0, 0, 1, 0, 60, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16,224, 1, 30, 0, 96,
> +255,191, 0, 0, 0, 0, 0, 0,255, 7, 0, 0, 0, 0, 0, 0,
> + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,248,207,227,
> +
> +/* wctype: pages U+00800,00d00 [words -1-3] */
> +BYTES(0x2c0008a8),6,64,1,3,8,5,16,8,19,17,15,48,
> +BYTES(0xc2008a80),
> +
> +/* wctype: pages U+00a00,00f00 [words 4-8] */
> +BYTES(0x8280c280),32,64,64,32,64,9,13,27,128,25,248,254,
> +BYTES(0x0a820067),
> +
> +/* wctype: pages U+00c00,01700 [words 9-13] */
> +BYTES(0x0283c203),16,64,128,16,32,64,19,35,252,48,112,16,
> +BYTES(0x8ac000cc),
> +
> +/* wctype: pages U+00e00,01d00 [words 14-17] */
> +BYTES(0x02800aa0),21,21,23,24,8,62,51,127,BYTES(0xb5000000),
> +
> +/* wctype: pages U+01000,00900 [words 18-22] */
> +BYTES(0x80080280),29,248,128,16,7,64,32,1,48,30,64,32,
> +BYTES(0x8280fe80),
> +
> +/* wctype: pages U+01400,01300 [words 23-25] */
> +BYTES(0x00000003),1,19,31,192,BYTES(0x00089800),
> +
> +/* wctype: pages U+01600,04d00 [words 26-28] */
> +BYTES(0x200b2000),33,1,48,112,BYTES(0x55000000),
> +
> +/* wctype: pages U+01800,01900 [words 29-31] */
> +BYTES(0x00000002),1,136,49,28,BYTES(0x58000380),
> +
> +/* wctype: pages U+01a00,01b00 [words 32-36] */
> +BYTES(0x00a0b008),128,1,37,39,1,41,64,24,31,248,16,16,
> +BYTES(0xb0209bc0),
> +
> +/* wctype: pages U+01c00,0a700 [words 37-40] */
> +BYTES(0xa6008080),43,128,45,47,49,12,3,127,BYTES(0x0002003d),
> +
> +/* wctype: pages U+02000,00b00 [words 41-45] */
> +BYTES(0xd452b966),43,53,1,223,55,1,1,13,64,11,64,32,
> +BYTES(0x82008280),
> +
> +/* wctype: pages U+02400,10900 [words 46-49] */
> +BYTES(0x60d55235),127,13,63,248,252,96,21,101,BYTES(0x5d800088),
> +
> +/* wctype: pages U+02c00,02100 [words 50-53] */
> +BYTES(0xa0000000),67,69,28,63,24,61,59,57,BYTES(0x555606aa),
> +
> +/* wctype: pages U+02e00,01f00 [words 54-57] */
> +BYTES(0xd5590165),1,73,15,33,192,192,9,15,BYTES(0xaa800000),
> +
> +/* wctype: pages U+03000,02b00 [words 58-61] */
> +BYTES(0x803800a6),77,79,81,60,1,16,65,207,BYTES(0x5559d555),
> +
> +/* wctype: pages U+03200,0fb00 [words 62-64] */
> +BYTES(0x55555559),1,3,252,4,BYTES(0x03c00020),
> +
> +/* wctype: pages U+0a400,02f00 [words 65-67] */
> +BYTES(0x83540000),127,128,75,63,BYTES(0x8d555555),
> +
> +/* wctype: pages U+0a600,10f00 [words 68-71] */
> +BYTES(0x8000a002),192,21,83,45,19,192,127,192,BYTES(0x00000b38),
> +
> +/* wctype: pages U+0a800,0ab00 [words 72-75] */
> +BYTES(0x920080a3),64,30,19,85,87,89,112,16,BYTES(0x20000800),
> +
> +/* wctype: pages U+0aa00,0fd00 [words 76-79] */
> +BYTES(0xcb808800),224,95,21,2,128,67,96,128,BYTES(0x80000080),
> +
> +/* wctype: pages U+0e000,10100 [words 80-83] */
> +BYTES(0x00000003),1,93,1,75,1,224,143,135,BYTES(0x943ac0d7),
> +
> +/* wctype: pages U+0f800,10300 [words 84-86] */
> +BYTES(0x80000000),21,1,21,15,BYTES(0x0c080030),
> +
> +/* wctype: pages U+0fe00,11300 [words 87-90] */
> +BYTES(0x80002d59),19,247,97,21,31,117,64,48,BYTES(0x0000e280),
> +
> +/* wctype: pages U+10200,11100 [words 91-94] */
> +BYTES(0x90000000),75,31,254,208,113,56,15,24,BYTES(0xfa00c3c0),
> +
> +/* wctype: pages U+10800,1f900 [words 95-98] */
> +BYTES(0x8030cc00),128,128,128,240,145,143,141,139,BYTES(0x56659556),
> +
> +/* wctype: pages U+10a00,16b00 [words 99-103] */
> +BYTES(0xe2088a80),103,105,105,192,192,2,107,127,0,3,240,48,
> +BYTES(0x00003b40),
> +
> +/* wctype: pages U+10c00,1d700 [words 104-107] */
> +BYTES(0x80000000),248,8,4,4,21,21,32,32,BYTES(0x032222cc),
> +
> +/* wctype: pages U+10e00,1f700 [words 108-110] */
> +BYTES(0x00009000),1,75,105,15,BYTES(0x2955d555),
> +
> +/* wctype: pages U+11000,02d00 [words 111-114] */
> +BYTES(0x0283be00),109,252,63,21,3,252,111,71,BYTES(0x00008000),
> +
> +/* wctype: pages U+11200,03100 [words 115-117] */
> +BYTES(0x20200080),115,4,12,15,BYTES(0x35040000),
> +
> +/* wctype: pages U+11400,0a900 [words 118-121] */
> +BYTES(0x03000a00),119,121,76,128,93,8,91,240,BYTES(0x0ac00820),
> +
> +/* wctype: pages U+11600,11500 [words 122-125] */
> +BYTES(0x00802380),21,14,31,123,0,0,45,21,BYTES(0x09800000),
> +
> +/* wctype: pages U+11800,0ff00 [words 126-130] */
> +BYTES(0xe0000080),28,248,7,96,99,63,240,1,240,1,248,254,
> +BYTES(0xa0003bbb),
> +
> +/* wctype: pages U+11a00,11700 [words 131-134] */
> +BYTES(0x00380280),125,45,190,7,0,0,248,16,BYTES(0x000000a0),
> +
> +/* wctype: pages U+11c00,11d00 [words 135-138] */
> +BYTES(0x0000eb80),21,62,248,31,3,0,127,52,BYTES(0x00080300),
> +
> +/* wctype: pages U+11e00,1d100 [words 139-141] */
> +BYTES(0x80000000),129,0,105,135,BYTES(0x25555565),
> +
> +/* wctype: pages U+12400,1d300 [words 142-144] */
> +BYTES(0x0000c000),31,0,105,127,BYTES(0x00009d55),
> +
> +/* wctype: pages U+13400,1e100 [words 145-147] */
> +BYTES(0x00000080),105,0,21,127,BYTES(0x000002c0),
> +
> +/* wctype: pages U+16a00,1e900 [words 148-150] */
> +BYTES(0xc0002000),128,63,128,137,BYTES(0x00000a00),
> +
> +/* wctype: pages U+16e00,1ed00 [words 151-153] */
> +BYTES(0x00090000),13,0,93,254,BYTES(0x00000097),
> +
> +/* wctype: pages U+1bc00,1f100 [words 154-157] */
> +BYTES(0x00380000),133,15,192,31,248,56,248,31,BYTES(0x70262216),
> +
> +/* wctype: pages U+1d000,10500 [words 158-160] */
> +BYTES(0xd5555555),63,0,0,21,BYTES(0x00002000),
> +
> +/* wctype: pages U+1d200,10b00 [words 161-164] */
> +BYTES(0xd0000355),63,15,0,252,60,254,254,252,BYTES(0x00288880),
> +
> +/* wctype: pages U+1d600,11900 [words 165-167] */
> +BYTES(0x8b000000),2,16,16,5,BYTES(0x30000000),
> +
> +/* wctype: pages U+1da00,11f00 [words 168-170] */
> +BYTES(0x003a5555),75,240,254,131,BYTES(0x95000000),
> +
> +/* wctype: pages U+1e200,16f00 [words 171-173] */
> +BYTES(0xa0000000),224,21,0,4,BYTES(0x30000000),
> +
> +/* wctype: pages U+1e800,1ec00 [words 174-176] */
> +BYTES(0x0f000000),128,127,0,0,
> +/* wctype: pages U+1ec00,1ee00 [words 176-178] */
> +BYTES(0x00d5c000),254,31,0,0,
> +/* wctype: pages U+1ee00,1f000 [words 178-180] */
> +BYTES(0xc0000000),3,0,0,0,
> +/* wctype: pages U+1f000,1f200 [words 180-183] */
> +BYTES(0xdfed5565),75,15,1,254,254,254,63,0,
> +/* wctype: pages U+1f200,1f600 [words 183-186] */
> +BYTES(0x00003e97),7,75,105,3,63,0,0,0,
> +/* wctype: pages U+1f600,1f800 [words 186-188] */
> +BYTES(0xad555555),63,31,13,0,
> +/* wctype: pages U+1f800,1fa00 [words 188-191] */
> +BYTES(0x00265a56),75,45,19,45,93,0,0,0,
> +/* wctype: pages U+1fa00,1fa00 [words 191-195] */
> +BYTES(0x000fad55),15,93,147,7,63,0,0,0,0,0,0,0,
> diff --git a/src/ctype/punct.h b/src/ctype/punct.h
> deleted file mode 100644
> index 6792947..0000000
> --- a/src/ctype/punct.h
> +++ /dev/null
> @@ -1,141 +0,0 @@
>
> -18,16,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,16,16,34,35,16,36,37,38,39,
>
> -40,41,42,43,16,44,45,46,17,17,47,17,17,17,17,17,17,48,49,50,51,52,53,54,55,17,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,56,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,57,16,58,59,60,61,62,63,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,64,16,16,16,16,16,16,16,16,16,
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,65,16,16,66,16,67,68,
>
> -69,16,70,71,72,16,73,16,16,74,75,76,77,78,16,79,80,81,82,83,84,85,86,87,88,89,
>
> -90,91,16,92,93,94,95,16,16,16,16,96,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,97,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,98,99,16,16,100,101,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,16,16,16,16,16,102,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,
>
> -16,16,16,103,104,105,106,16,16,107,108,17,17,109,16,16,16,16,16,16,110,111,16,
>
> -16,16,16,16,112,113,16,16,114,115,116,16,117,118,119,17,17,17,120,121,122,123,
> -124,16,16,16,16,
>
> -16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,254,255,0,252,1,0,0,248,1,
>
> -0,0,120,0,0,0,0,255,251,223,251,0,0,128,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,60,0,252,255,224,175,255,255,255,255,255,255,255,255,
>
> -255,255,223,255,255,255,255,255,32,64,176,0,0,0,0,0,0,0,0,0,0,0,0,0,64,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,252,0,0,0,0,0,230,254,255,255,255,0,64,73,0,0,0,0,0,24,0,255,255,0,216,
> -0,0,0,0,0,0,0,1,0,60,0,0,0,0,0,0,0,0,0,0,0,0,16,224,1,30,0,
>
> -96,255,191,0,0,0,0,0,0,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,248,207,
>
> -227,0,0,0,3,0,32,255,127,0,0,0,78,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,0,7,252,0,0,0,
>
> -0,0,0,0,0,0,16,0,32,30,0,48,0,1,0,0,0,0,0,0,0,0,16,0,32,0,0,0,0,252,111,0,0,0,
>
> -0,0,0,0,16,0,32,0,0,0,0,64,0,0,0,0,0,0,0,0,16,0,32,0,0,0,0,3,224,0,0,0,0,0,0,
>
> -0,16,0,32,0,0,0,0,253,0,0,0,0,0,0,0,0,0,0,32,0,0,0,0,255,7,16,0,0,0,0,0,0,0,0,
>
> -32,0,0,0,0,128,255,16,0,0,0,0,0,0,16,0,32,0,0,0,0,0,0,0,0,0,0,0,0,0,24,0,160,
>
> -0,127,0,0,255,3,0,0,0,0,0,0,0,0,0,4,0,0,0,0,16,0,0,0,0,0,0,128,0,128,192,223,
> -0,12,0,0,0,0,0,0,0,0,0,0,0,4,0,31,0,0,0,0,0,
>
> -0,254,255,255,255,0,252,255,255,0,0,0,0,0,0,0,0,252,0,0,0,0,0,0,192,255,223,
>
> -255,7,0,0,0,0,0,0,0,0,0,0,128,6,0,252,0,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,0,
>
> -0,0,8,0,0,0,0,0,0,0,0,0,0,0,224,255,255,255,31,0,0,255,3,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,96,0,0,1,0,0,24,0,0,0,0,0,0,0,0,0,56,0,0,0,0,16,0,0,0,112,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,48,0,0,254,127,47,0,0,255,3,255,127,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,14,49,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,0,0,0,196,255,255,255,
>
> -255,0,0,0,192,0,0,0,0,0,0,0,0,1,0,224,159,0,0,0,0,127,63,255,127,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,16,0,16,0,0,252,255,255,255,31,0,0,0,0,0,12,0,0,0,0,0,0,64,0,
>
> -12,240,0,0,0,0,0,0,128,248,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,255,0,255,255,
>
> -255,33,144,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,
>
> -127,0,224,251,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,160,3,224,0,224,0,
>
> -224,0,96,128,248,255,255,255,252,255,255,255,255,255,127,223,255,241,127,255,
>
> -127,0,0,255,255,255,255,0,0,255,255,255,255,1,0,123,3,208,193,175,66,0,12,31,
>
> -188,255,255,0,0,0,0,0,14,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,127,0,0,0,255,7,0,0,255,255,255,255,255,255,255,255,255,
> -255,63,0,0,0,0,0,0,252,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,207,255,255,255,
>
> -63,255,255,255,255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,224,135,3,254,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,127,255,255,255,255,0,
>
> -0,0,0,0,0,255,255,255,251,255,255,255,255,255,255,255,255,255,255,15,0,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,63,0,0,0,255,15,30,255,255,255,1,252,193,224,0,0,0,0,
>
> -0,0,0,0,0,0,0,30,1,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -255,255,0,0,0,0,255,255,255,255,15,0,0,0,255,255,255,127,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> -255,255,255,
>
> -255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,
>
> -255,255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,127,0,0,0,
>
> -0,0,0,192,0,224,0,0,0,0,0,0,0,0,0,0,0,128,15,112,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -255,0,255,255,127,0,3,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -64,0,0,0,0,15,255,3,0,0,0,0,0,0,240,0,0,0,0,0,0,0,0,0,16,192,0,0,255,255,3,23,
>
> -0,0,0,0,0,248,0,0,0,0,8,128,0,0,0,0,0,0,0,0,0,0,8,0,255,63,0,192,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,240,0,0,128,3,0,0,0,0,0,0,0,128,2,0,0,192,0,0,67,0,0,0,0,0,
> -0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,56,0,
>
> -0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,2,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,252,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,48,255,255,255,3,255,255,255,255,255,255,247,
>
> -255,127,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,254,255,0,252,1,0,0,248,1,0,
>
> -0,248,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,127,127,0,48,135,255,255,255,255,255,
>
> -143,255,0,0,0,0,0,0,224,255,255,127,255,15,1,0,0,0,0,0,255,255,255,255,255,63,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,
>
> -15,0,0,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -128,255,0,0,128,255,0,0,0,0,128,255,0,0,0,0,0,0,0,0,0,248,0,0,192,143,0,0,0,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,48,255,255,252,255,255,255,255,255,0,0,0,0,
>
> -0,0,0,135,255,1,255,1,0,0,0,224,0,0,0,224,0,0,0,0,0,1,0,0,96,248,127,0,0,0,0,
>
> -0,0,0,0,254,0,0,0,255,0,0,0,255,0,0,0,30,0,254,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,0,0,0,0,0,0,0,0,0,0,0,
> -0,255,255,255,127,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,224,127,0,0,0,192,255,255,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,192,63,252,255,63,0,0,128,3,0,0,0,0,0,0,254,3,32,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,24,0,15,0,0,0,0,0,56,0,0,0,0,0,0,0,0,0,225,63,0,232,254,255,31,0,0,
>
> -0,0,0,0,0,96,63,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,
>
> -24,0,32,0,0,192,31,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,68,
>
> -248,0,104,0,0,0,0,0,0,0,0,0,0,0,0,76,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,128,255,255,255,0,0,0,0,0,0,0,0,0,0,0,0,128,14,0,0,0,255,
> -31,0,0,0,0,0,0,0,0,192,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,8,0,252,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,252,7,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,24,128,255,0,0,0,0,0,
>
> -0,0,0,0,0,223,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,62,0,0,252,255,31,3,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,52,0,0,0,0,0,0,0,0,0,128,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,128,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,
> -255,3,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,192,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,63,0,0,0,0,0,0,0,255,255,48,0,0,248,
>
> -3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,
>
> -255,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,176,15,0,0,0,0,0,0,
>
> -0,0,0,0,0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
> -255,255,255,255,255,255,255,255,255,255,255,255,255,63,
>
> -0,255,255,255,255,127,254,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,1,0,0,255,255,255,255,255,255,255,255,
>
> -63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,255,15,0,255,255,255,255,255,255,
>
> -255,255,255,255,127,0,255,255,255,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,8,0,0,0,8,0,0,32,0,0,0,32,0,0,128,
>
> -0,0,0,128,0,0,0,2,0,0,0,2,0,0,8,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,15,0,248,254,255,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,127,0,0,128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,240,0,
>
> -128,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,128,255,127,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,112,7,0,192,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,254,255,255,255,255,255,255,255,31,0,0,0,0,0,0,0,0,0,254,255,
>
> -255,255,255,255,255,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,255,255,255,255,255,
>
> -15,255,255,255,255,255,255,255,255,255,255,255,255,15,0,255,127,254,255,254,
>
> -255,254,255,255,255,63,0,255,31,255,255,255,255,0,0,0,252,0,0,0,28,0,0,0,252,
>
> -255,255,255,31,0,0,0,0,0,0,192,255,255,255,7,0,255,255,255,255,255,15,255,1,3,
> -0,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
>
> -0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,255,63,0,255,31,255,7,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,255,255,15,0,255,255,255,255,255,255,255,255,255,255,255,1,
>
> -255,15,0,0,255,15,255,255,255,255,255,255,255,0,255,3,255,255,255,255,255,0,
>
> -255,255,255,63,0,0,0,0,0,0,0,0,0,0,255,239,255,255,255,255,255,255,255,255,
>
> -255,255,255,255,123,252,255,255,255,255,231,199,255,255,255,231,255,255,255,
>
> -255,255,255,255,255,255,255,255,255,255,255,255,255,15,0,255,63,15,7,7,0,63,0,
> -0,0,0,0,0,0,0,0,0,0,0,0,
> --
> 2.51.2
>
>
[-- Attachment #2: Type: text/html, Size: 63264 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47%
2026-02-18 1:28 ` Xan Phung
@ 2026-02-25 15:06 ` Rich Felker
2026-03-02 8:57 ` Xan Phung
0 siblings, 1 reply; 6+ messages in thread
From: Rich Felker @ 2026-02-25 15:06 UTC (permalink / raw)
To: Xan Phung; +Cc: Openwall musl
On Wed, Feb 18, 2026 at 12:28:33PM +1100, Xan Phung wrote:
> Hi,
>
> I haven't heard back on where V2 of this patch stands - has it been
> rejected or is it still under evaluation?
It's certainly not rejected. This looks good. I just want to give it
adequate attention to understand it before merging and be confident
that it's keeping or improving performance on most/all archs not just
a couple mainstream ones. When I first saw it I was worried the
popcount thing would be a problem and only fast on archs with native
popcounts, but I saw that you're not even using any native popcount
just portable arithmetic so that seems like a non-issue.
If anyone else has an opportunity to test performance on other archs
and post results I'd love to see them.
I don't expect to get to looking at this in depth until rolling a
release, but I am actively trying to do that again now.
Thanks for working on this and for pinging about it again!
Rich
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47%
2026-02-25 15:06 ` Rich Felker
@ 2026-03-02 8:57 ` Xan Phung
0 siblings, 0 replies; 6+ messages in thread
From: Xan Phung @ 2026-03-02 8:57 UTC (permalink / raw)
To: Rich Felker, Openwall musl
[-- Attachment #1: Type: text/plain, Size: 3310 bytes --]
Hi Rich,
Thanks for the update!
I have also now extended my header file generation tool to create the data
for wcwidth as well.
(My existing 'popcount' data format was flexible enough to easily handle
the two bit data values needed by wcwidth - so the separate 1-bit data sets
of 4kb in wide.h & nonspacing.h, plus the hard-coded '-1' control chars
outside code page 0 are replaced by a unified 1.7kb data set containing
encoded 2 bit data values).
My tool ('gen_wcdata') thus replaces the function of 3 tools (I keep these
original tools only for regression testing):
- gen_ctype
- gen_nonspacing
- gen_wide
I have uploaded my 'gen_wcdata' tool on github, as a fork of your repo at:
https://github.com/nglibc/musl-chartable-tools/tree/master/ctype
It also does verification by testing all 3 functions against 20,000
codepoints. You can run the verification by using 'make test' within the
ctype subdirectory.
I'll also prepare a wcwidth.c patch (for the git.musl-libc.org repo) soon.
P.S. More details about gen_wcdata and test_wcdata are in the ctype/README
file:
The command line syntax below is accepted. The 'a', 'p', 'w' letters
generate iswalpha_table.h, iswpunct_table.h and wcwidth_table.h, and
the upper case letters generate the the _dict.h header files:
gen_wcdata [-v] a|p|w|A|P|W
Also, if compiled together with the functions source, it will do tests
of these wctype/wchar functions against this repo's Unicode data.
If compiled as just the gen_wcdata.c source file alone, the tool tests
against the compiler's C library equivalent functions.
To test against the compiler C library functions, execute the following
(where 'a' tests iswalpha, 'p' tests iswpunct, 'v' tests wcwidth):
gen_wcdata -t a|p|v
Although gen_wcdata is a full replacement for gen_ctype, gen_wide and
gen_nonspacing, these tools are retained for regression testing.
A test recipe has been added to Makefile to generate both new and old
header files, and compile test_wcdata with new & old source files.
To run these tests, execute:
make test
Alternatively, the test_wcdata tool can be run directly using:
test_wcdata -t a|p|v
or (to test older musl functions) use:
test_wcdata_v0 -t a|p|v
Best regards
Xan Phung
On Thu, 26 Feb 2026 at 02:06, Rich Felker <dalias@libc.org> wrote:
> On Wed, Feb 18, 2026 at 12:28:33PM +1100, Xan Phung wrote:
> > Hi,
> >
> > I haven't heard back on where V2 of this patch stands - has it been
> > rejected or is it still under evaluation?
>
> It's certainly not rejected. This looks good. I just want to give it
> adequate attention to understand it before merging and be confident
> that it's keeping or improving performance on most/all archs not just
> a couple mainstream ones. When I first saw it I was worried the
> popcount thing would be a problem and only fast on archs with native
> popcounts, but I saw that you're not even using any native popcount
> just portable arithmetic so that seems like a non-issue.
>
> If anyone else has an opportunity to test performance on other archs
> and post results I'd love to see them.
>
> I don't expect to get to looking at this in depth until rolling a
> release, but I am actively trying to do that again now.
>
> Thanks for working on this and for pinging about it again!
>
> Rich
>
[-- Attachment #2: Type: text/html, Size: 11579 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-03-02 8:57 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-01-23 13:20 [musl][PATCH v2] wctype: reduce size of iswalpha & iswpunct by 47% Xan Phung
2026-01-23 13:40 ` Xan Phung
2026-01-30 12:39 ` Xan Phung
2026-02-18 1:28 ` Xan Phung
2026-02-25 15:06 ` Rich Felker
2026-03-02 8:57 ` Xan Phung
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/musl/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).