From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/3825 Path: news.gmane.org!not-for-mail From: Harald Becker Newsgroups: gmane.linux.lib.musl.general Subject: Re: iconv Korean and Traditional Chinese research so far Date: Mon, 5 Aug 2013 09:03:43 +0200 Message-ID: <20130805090343.6d2f9f00@ralda.gmx.de> References: <20130804165152.GA32076@brightrain.aerifal.cx> <20130805003943.050fc58e@ralda.gmx.de> <20130805004420.GL25714@port70.net> <20130805032452.280127fd@ralda.gmx.de> <20130805031322.GM25714@port70.net> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1375686258 16415 80.91.229.3 (5 Aug 2013 07:04:18 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 5 Aug 2013 07:04:18 +0000 (UTC) Cc: musl@lists.openwall.com, nsz@port70.net Original-X-From: musl-return-3829-gllmg-musl=m.gmane.org@lists.openwall.com Mon Aug 05 09:04:21 2013 Return-path: Envelope-to: gllmg-musl@plane.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1V6Eq3-000158-PW for gllmg-musl@plane.gmane.org; Mon, 05 Aug 2013 09:04:19 +0200 Original-Received: (qmail 7479 invoked by uid 550); 5 Aug 2013 07:04:03 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 7403 invoked from network); 5 Aug 2013 07:03:57 -0000 In-Reply-To: <20130805031322.GM25714@port70.net> X-Provags-ID: V03:K0:9ey4CunMW0LFYWcg5YG2LZQ5RXNGljpEqhufwg5zTEX9cIcoJEs QsFKHD22ObBLg/W0VCng30gHdS7g6g/udmJp7FrlqmmJx9GYtbhJNOXmdm62wPy/vzqwP/T 6tXbmx+/JMe7VdxoGFR7zLAnEAtbCzcOLwWK98zR4MbPC9pnAy1ZbLxQekeBEDk71haqU3W jKzOh2Zif8t91LP7z6U2g== Xref: news.gmane.org gmane.linux.lib.musl.general:3825 Archived-At: Hi ! 05-08-2013 05:13 Szabolcs Nagy : > * Harald Becker [2013-08-05 03:24:52 +0200]: > > iconv then shall: > > - look for some fixed charsets like ASCII, Latin-1, UTF-8, > > etc. > > - search table of with libc linked charsets > > - search table of with the program linked charsets > > - search for charset on external search path > > sounds like a lot of extra management cost > (for libc, application writer and user as well) This is not so much work. You already need to search for the character set table to use, that is you need to search at least a table of string values to find the pointer to the conversion table. Searching a table in may statement above, means just waking a pointer chain doing the string compares to find a matching character set. Not much difference to the really required code. Now do this twice to check possible user chain, is just one more helper function call. The only code that get a bit more, is the file system search. This depends if we only try single location or walk through a search path list. But this is the cost of flexibility to dynamically load character set conversions (which I would really prefer for seldom used char sets). ... and for application writer it is only more, if he likes to add some charset tables into his program, which are not in statical libc. The problem is, all tables in libc need to be linked to your program, if you include iconv. So each added charset conversion increases size of your program ... and I definitly won't include Japanese, Chinese or Korean charsets in my program. No that I ignore those peoples need, I just wont need it, so I don't like to add those conversions to programs sitting on my disk. > it would be nice if the compiler could figure out > at build time (eg with lto) which tables are used > but i guess charsets often only known at runtime How do you want to do this? And how shall the compiler know which char sets the user may use during operation? So the only way to select the charset tables to include in your program, is by assuming ahead, which tables might be used. That is part of the configuration of musl build or application program build. > > [Addendum after thinking a bit more: The byte code conversion > > files shall exist of a small statical header, followed by the > > byte code program. The header shall contain the charset name, > > version of required virtual machine and length of byte code. > > So you need only add all such conversion files to a big array > > of bytes and add a Null header to mark the end of table. Then > > you only need the start of the array and you are able to > > search through for a specific charset. The iconv function in > > libc contains a definition for an "unsigned char const > > *iconv_user_charsets = NULL;", which is linked in, when the > > user does not provide it's own definition. So iconv can > > search all linked in charset definitions, and need no code > > changes. Really simple configuration to select charsets to > > build in.] > > > > yes that can work, but it's a musl specific hack > that the application programmer need to take care of Only if application programmer wants to add a char set to the statical build program, which is not in libc, some extra work has to be done. Giving some more flexibility. If you don't care, you get the musl build in list of char sets. > > > if the format changes then dynamic linking is > > > problematic as well: you cannot update libc > > > in a single atomic operation > > > > The byte code shall be independent of dynamic linking. The > > conversion files are only streams of bytes, which shall also > > be architecture independent. So you do only need to update the > > conversion files if the virtual machine definition of iconv > > has been changed (shall not be done much). External files may > > be read into malloc-ed buffers or mmap-ed, not linked in by > > the dynamical linker. > > > > that does not solve the format change problem > you cannot update libc without race > (unless you first replace the .so which supports > the old format as well as the new one, but then > libc has to support all previous formats) If the definition of the iconv virtual state machine is modified, you need to do extra care on update (delete old charset files, install new lib, install new charset files, restart system) ... but this is only required on a major update. As soon as the virtual machine definition gots stabilized you do not need to change charset definition files, or just do update your lib, then update possible new charset files. After an initial phase of testing this shall happen relatively seldom, that the virtual machine definition needs to be changed in an incompatible manner. And simple extending the virtual machine does not invalidate the old charset files. > it's probably easy to design a fixed format to > avoid this A fixed format? For what? Do you know the differences of char sets, especially multi byte char sets? > it seems somewhat similar to the timezone problem > ecxept zoneinfo is maintained outside of libc so > there is not much choice, but there are the same > issues: updating it should be done carefully, > setuid programs must be handled specially etc Again. As soon as the virtual machine definition has reached a stable state, it shall not happen much, that any change invalidates a charset definition file. That is at least old files will continue to work with newer lib versions. So there is no problem on update, just update your lib then update your charset files. The only problem will be, if a still running application uses a new charset file with an old version of the lib. This will be detected and leads to a failure code of iconv. So you need to restart your application ... which is always a good decision as you updated your lib. -- Harald