From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/7905 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: Build option to disable locale [was: Byte-based C locale, draft 1] Date: Sun, 7 Jun 2015 22:36:35 -0400 Message-ID: <20150608023635.GE17573@brightrain.aerifal.cx> References: <20150606214007.GA17398@brightrain.aerifal.cx> <55737684.7020803@gmx.de> <20150606231057.GZ17573@brightrain.aerifal.cx> <55738979.4030809@gmx.de> <20150607002459.GA17573@brightrain.aerifal.cx> <5574DAE7.8040101@gmx.de> <5574F66D.2070803@gmx.de> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1433731014 20830 80.91.229.3 (8 Jun 2015 02:36:54 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 8 Jun 2015 02:36:54 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-7918-gllmg-musl=m.gmane.org@lists.openwall.com Mon Jun 08 04:36:54 2015 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1Z1mvj-0006Zn-EN for gllmg-musl@m.gmane.org; Mon, 08 Jun 2015 04:36:51 +0200 Original-Received: (qmail 30320 invoked by uid 550); 8 Jun 2015 02:36:49 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 30291 invoked from network); 8 Jun 2015 02:36:48 -0000 Content-Disposition: inline In-Reply-To: <5574F66D.2070803@gmx.de> User-Agent: Mutt/1.5.21 (2010-09-15) Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:7905 Archived-At: On Mon, Jun 08, 2015 at 03:57:01AM +0200, Harald Becker wrote: > Hi Josiah ! > > On 08.06.2015 02:28, Josiah Worcester wrote: > >To start with: keep in mind that in the case of static linking most of > >this is not at all pulled in except when strictly necessary. Static > >linking might be more relevant to your needs. > > You missed my lead in message: I don't talk about static linking > (which is what I currently do, replacing other stuff). I'm looking > for a really small shared C library. Do you have an application in mind where saving ~13k in libc.so would make the difference between being able to use it or not? > The problem is: Stripping all the functions on every new release is > too much work, so it needs some minimal support in the library build > process. Which should be pure optional, enabling the full version as > default. Maintaining useless configuration knobs and having to test them all is also too much work, and the main reason why uClibc is dying. > >>>- UTF-8 encoding and decoding > >>May be of use to keep, if on bare minimum. > > > >Seeing as the UTF-8 decoder is very small already, I'd be shocked if > >you could make an argument for removing that. > > That's why I told "keep". I know it to be small, so keep it small > and fast, and UTF-8 only (not supporting other multi byte char > sets). That's how it always has been. You could actually make it smaller by writing the string functions as trivial character-at-a-time loops on top of mbtowc/wctomb, but you'd be sacrificing performance to save a few hundred bytes at most. This does not seem like a worthwhile trade-off. > >>>- Character properties > >> > >>>- Case mappings > >> > >>Keep ASCII, map all none ASCII to a single value. > > > >This would be not-quite-right. Also, the case mapping tables are quite > >small. towctrans.lo which contains the case mappings is 1106 bytes. > > Sorry for my poor English. I mean, let all functions behave as for > locale "C" (ASCII), but don't fail / break when there is an embedded > UTF-8 sequence. Just say "this is not ASCII" or may be: "this is > UTF-8". musl's "C" locale is _not_ ASCII but is UTF-8. Supporting both byte-at-a-time operation in the C locale and proper UTF-8 handling otherwise is the topic of this thread, and it very mildly increases code size. > >>>- Internal message translation (nl_langinfo strings, errors, etc.) > >> > >>>- Message translation API (gettext) > >> > >>No translation at all, keep the English messages (as short as possible). > > > >musl does not have any translations in it at all. It only has a small > >portion of logic able to load external translations. locale_map.lo and > >__mo_lookup.lo which are together responsible for this, are a total of > >1471 bytes. > > I mean drop the portion to load external translations. This probably could be a configurable option at some point, but it's not going to save you any meaningful amount of space. Best case would be saving about 4k. > >>>- Charset conversion (iconv) > >> > >> > >>Copy ASCII / UTF-8, but fail for all other. > > > >Though quite possible, it's worth noting that musl iconv is not very > >large. iconv.lo is 128408 bytes, or 125k. > > A big hunk to kick off. Just keep a stub that allows for ASCII and > UTF-8 and 1:1 copy operation. Should be possible to draw that down > to not more than 1k. 119k of the 128k for iconv is legacy CJK character set tables. The code for them is only 1k or so. Legacy 8bit codepages are another 6k. The easy way to make things optional here would be just dropping tables for configured-out charsets. > >The regex equivalence classes are handled via the isw* functions which > >(as mentioned above) are quite small. > > So handle them as for the "C" locale, but don't fail/break when > someone enters an UTF-8 sequence. "Don't break" means handling UTF-8. Thankfully that's small and changing this code is not i > >In short, it seems like if we made these changes we'd maybe be able to > >trim out 135k and almost all of that would be in iconv. Though I > >appreciate the desire for smaller code, this doesn't quite seem like > >the place to go looking. > > My request is, to get a shared lib which does not need to include > all that other char set and locale code. An optional build option, > but with the need of some minimal support, so it can persist for > future releases. I dislike doing all the strip down work over and > over again for any new release. Is this a practical need or an ideological one? > The resulting library shall be standards conform and fully > operational, as long as the applications only use the bare C locale, > but should allow to pass through embedded UTF-8 sequences. Your goal keeps shifting back between supporting UTF-8 and "passing through sequences" which is _NOT_ supporting UTF-8. If I accidentally type a non-ASCII character on a command line then press backspace and the character disappears but there are still two hidden junk bytes on the command line, that is a broken system. At many levels, you can get by with just treating text as abstract byte strings that can pass through anything, but for any kind of visual presentation (even on a terminal) or entry/editing, you need to know character identity. Fortunately, as I've been trying to say, that's extremely cheap. You could save more space by making the string functions or qsort or something naive and slow... Rich