From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/12556 Path: news.gmane.org!.POSTED!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: Re: iconv failure (ISO-2022-JP) since musl update on AlpineLinux Date: Tue, 27 Feb 2018 15:19:40 -0500 Message-ID: <20180227201940.GR1436@brightrain.aerifal.cx> References: <20180227165704.vyc0m%steffen@sdaoden.eu> <20180227194432.yqIkD%steffen@sdaoden.eu> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Trace: blaine.gmane.org 1519762675 20099 195.159.176.226 (27 Feb 2018 20:17:55 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 27 Feb 2018 20:17:55 +0000 (UTC) User-Agent: Mutt/1.5.21 (2010-09-15) To: musl@lists.openwall.com, Steffen Nurpmeso Original-X-From: musl-return-12572-gllmg-musl=m.gmane.org@lists.openwall.com Tue Feb 27 21:17:51 2018 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1eqlh8-0004iP-DE for gllmg-musl@m.gmane.org; Tue, 27 Feb 2018 21:17:50 +0100 Original-Received: (qmail 15478 invoked by uid 550); 27 Feb 2018 20:19:53 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 15445 invoked from network); 27 Feb 2018 20:19:52 -0000 Content-Disposition: inline In-Reply-To: <20180227194432.yqIkD%steffen@sdaoden.eu> Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:12556 Archived-At: On Tue, Feb 27, 2018 at 08:44:32PM +0100, Steffen Nurpmeso wrote: > Hi. > > Rich Felker wrote: > > sorry i did not get this :) Sorry I neglected to keep you CC'd. > but i wrote: > ||After updating to musl-1.1.19-r0 there i saw test failures for the > ||MUA i maintain, namely regarding the mentioned charset. I will > ||attach a file to reproduce. (Am not subscribed.) > ... > || #?0[steffen@devon steffen]$ cksum in.utf > || 1259742080 686 in.utf > || #?0[steffen@devon steffen]$ iconv -f utf8 -t iso-2022-jp || 2184132317 536 > || #?0[steffen@devon steffen]$ iconv --version > || iconv (GNU libiconv 1.11) > ||.. > || #?0[steffen@essex tmp]$ cksum in.utf > || 1259742080 686 in.utf > || #?0[steffen@essex tmp]$ iconv -f utf8 -t iso-2022-jp || 209789743 1736 > || #?0[steffen@essex tmp]$ apk info --who-owns /usr/bin/iconv > || /usr/bin/iconv is owned by musl-utils-1.1.19-r0 > > |Does the data round-trip correctly? I don't think you can expect > > Ok, i see what you mean, yes, musl iconv(1) can roundtrip. But.. > But for one the error is new (though i actually have forgotten > whether the test ever ran on a musl box or only on BSD and glibc > Linux boxes, but if i recall, it did run, and then it did succeed, > definetely), and then... > > |bitwise match between outputs of different ISO-2022-JP converters, > |unless perhaps they both guarantee minimality, because the ISO-2022-JP > |representation of a string is highly nonunique. > | > |In particular musl's to-ISO-2022-JP converter is stateless and always > |generates shifts in/out around every non-ASCII character. Of course > |this is highly suboptimal, but in the worst case (where the caller > |calls iconv one character at a time) the iconv API can't do any better > |because strings are required to end in the unshifted state, and the > |iconv API doesn't have any method to "finalize" a conversion. This > |implies that every time iconv returns with non-ASCII as the most > |recent output character, it must be followed by a shift back to the > |initial (ASCII) state. > | > |We could improve this in the case of batch conversions by overwriting > |the previous shift-back-to-initial and skipping the next shift if the > |character set of the next character to output matches the previous > |one, but that only works within a single batch call, since iconv can't > |write outside the buffer passed to it for the current call. This is an > |improvement I think I want to make, since it would improve typical > |output size a lot, but the cost is output determinism under different > |chunking by the caller. > > Well... In my cases the MUA fails to convert to ISO-2022-JP at > all, because an iconv(3) error happens. And when i instrument my > code like > > for(;;){ > size_t sz; > > fprintf(stderr, "iconv(3): in %lu out: %lu\n",*inbleft,*outbleft); > fprintf(stderr, " in<%.*s>\n",(int)*inbleft,*inb); > sz = iconv(cd, __INBCAST(inb), inbleft, outb, outbleft); > if(sz > 0 && !(icf & n_ICONV_IGN_NOREVERSE)){ > fprintf(stderr, "iconv(3) returned 0x%lX: %s\n",(ul_i)sz,strerror(errno)); > err = n_ERR_NOENT; > goto jleave; > } > if(sz != (size_t)-1) > break; > > then i get > > #?1[steffen@essex nail.git]$ v mae-test-behave_iconv_mbyte_base64-2 > ICONV 2 > iconv(3): in 220 out: 427 > in<シジュウカラ科（シジュウカラか、学名 Paridae）は、鳥類スズメ目の科である。シジュウカラ（四十雀）と総称されるが、狭義にはこの1種をシジュウカラと呼ぶ。 > > > iconv(3) returned 0xFFFFFFFFFFFFFFFF: Argument list too long > ICONV 2 err: 2 > > And that is somehow ooops? Interestingly if i call iconv(1) only > on these 220 bytes i can roundtrip that, too. Hmmm. ... > I thought maybe it is because of the tcc(1) compiler i use, but > i can reproduce this with AlpineLinux gcc(1), too. I don't know. I think the test is just using an output buffer that's under the worst-case size needed for conversion to ISO-2022-JP. The E2BIG error is specified for "Input conversion stopped due to lack of space in the output buffer" and is not really an error; is just means the conversion stopped before reaching the end and you need to resume with a new buffer for the remainder of the conversion. Rich