From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/749 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Help establishing wctype character classes Date: Sun, 22 Apr 2012 16:41:03 -0400 Message-ID: <20120422204103.GJ14673@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: dough.gmane.org 1335127103 14512 80.91.229.3 (22 Apr 2012 20:38:23 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sun, 22 Apr 2012 20:38:23 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-750-gllmg-musl=m.gmane.org@lists.openwall.com Sun Apr 22 22:38:22 2012 Return-path: Envelope-to: gllmg-musl@plane.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1SM3Y4-0006SL-Ih for gllmg-musl@plane.gmane.org; Sun, 22 Apr 2012 22:38:20 +0200 Original-Received: (qmail 26036 invoked by uid 550); 22 Apr 2012 20:38:20 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 26028 invoked from network); 22 Apr 2012 20:38:19 -0000 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Xref: news.gmane.org gmane.linux.lib.musl.general:749 Archived-At: Hi, I'm looking for some help establishing the proper definitions (in terms of Unicode) for the different wctype character classes (isw*() functions) in musl. Some are relatively easy and already done: - Spaces: Unicode White_Space property, minus non-breaking spaces (which lack "space" semantics; that's their whole point) and script-specific, non-blank-glyph characters that Unicode oddly calls "spaces". - Control: The traditional C0/C1 control codes, plus some bidi and embedding control stuff. (Basically, anything that can't be printed literally in plain text and expected to work.) - Printable: Any valid codepoint that's not control. This broad definition is to avoid the situation where an application refuses to print valid text in the end user's language because it contains characters added to Unicode after the libc/app was published. It's assumed that the vast majority of, if not all, new characters added to Unicode will be non-control characters and non-problematic to print. - Graphic: Any printable character that's not a space. This definition is provided by the C standard. - Digit: Exactly the ASCII decimal digits '0'...'9' and nothing else. This definition is provided by the C standard. If anyone has objections to the above definitions, I'd be happy to hear them, but what I'm most concerned about is the remaining classes, specifically alphabetic and punctuation. Here are some tentative partial definitions: - Alphabetic: Derived property Alphabetic from Unicode's DerivedCoreProperties.txt. Not yet in musl. - Punctutation: Currently defined in musl as Unicode character classes P* and Sm, plus Pattern_Syntax from PropList.txt. However, these definitions omit a number of characters which should probably be in one or the other of these two classes, in particular at least: - All Unicode class N* (numeric) characters, like superscripts, vulgar fractions, Roman numerals, etc. - Combining diacritic characters that are not themselves letters. and I'm unclear on what should be done with them. Especially for numeric/digit-like characters, they cannot be classified as "digits" (per ISO C rules), and calling them letters may make more sense; glibc on the other hand considers them punctuation. As for combining diacritics, considering them letters if their intended usage is to combine onto letters would be convenient for handling NFD text with regular expressions (/[[:alpha:]]+/ will match full words even if the diacritics are decomposed), but I suspect it will also make some cases behave worse. Before anybody suggests just dumping what glibc does and copying it, glibc is horribly incorrect on a number of characters in ways that break internationalized usage. For instance, all Tibetan subjoined letter characters are wrongly classified as punctuation rather than as letters; this will break usage of regular expressions like /[[:alpha:]]+/ with affected scripts/languages. Rich