From mboxrd@z Thu Jan  1 00:00:00 1970
X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/12555
Path: news.gmane.org!.POSTED!not-for-mail
From: Steffen Nurpmeso <steffen@sdaoden.eu>
Newsgroups: gmane.linux.lib.musl.general
Subject: Re: iconv failure (ISO-2022-JP) since musl update on AlpineLinux
Date: Tue, 27 Feb 2018 20:44:32 +0100
Message-ID: <20180227194432.yqIkD%steffen@sdaoden.eu>
References: <20180227165704.vyc0m%steffen@sdaoden.eu>
Reply-To: musl@lists.openwall.com
NNTP-Posting-Host: blaine.gmane.org
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Trace: blaine.gmane.org 1519760568 13832 195.159.176.226 (27 Feb 2018 19:42:48 GMT)
X-Complaints-To: usenet@blaine.gmane.org
NNTP-Posting-Date: Tue, 27 Feb 2018 19:42:48 +0000 (UTC)
User-Agent: s-nail v14.9.7-9-g06c930cc
To: musl@lists.openwall.com
Original-X-From: musl-return-12571-gllmg-musl=m.gmane.org@lists.openwall.com Tue Feb 27 20:42:44 2018
Return-path: <musl-return-12571-gllmg-musl=m.gmane.org@lists.openwall.com>
Envelope-to: gllmg-musl@m.gmane.org
Original-Received: from mother.openwall.net ([195.42.179.200])
	by blaine.gmane.org with smtp (Exim 4.84_2)
	(envelope-from <musl-return-12571-gllmg-musl=m.gmane.org@lists.openwall.com>)
	id 1eql9A-0003GG-I5
	for gllmg-musl@m.gmane.org; Tue, 27 Feb 2018 20:42:44 +0100
Original-Received: (qmail 17447 invoked by uid 550); 27 Feb 2018 19:44:48 -0000
Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm
Precedence: bulk
List-Post: <mailto:musl@lists.openwall.com>
List-Help: <mailto:musl-help@lists.openwall.com>
List-Unsubscribe: <mailto:musl-unsubscribe@lists.openwall.com>
List-Subscribe: <mailto:musl-subscribe@lists.openwall.com>
List-ID: <musl.lists.openwall.com>
Original-Received: (qmail 17409 invoked from network); 27 Feb 2018 19:44:47 -0000
In-Reply-To: <20180227165704.vyc0m%steffen@sdaoden.eu>
Mail-Followup-To: musl@lists.openwall.com,
 Steffen Nurpmeso <steffen@sdaoden.eu>
OpenPGP: id=EE19E1C1F2F7054F8D3954D8308964B51883A0DD;
 url=https://ftp.sdaoden.eu/steffen.asc
BlahBlahBlah: Any stupid boy can crush a beetle. But all the professors in
 the world can make no bugs.
Xref: news.gmane.org gmane.linux.lib.musl.general:12555
Archived-At: <http://permalink.gmane.org/gmane.linux.lib.musl.general/12555>

Hi.

Rich Felker wrote:

sorry i did not get this :)

but i wrote:
 ||After updating to musl-1.1.19-r0 there i saw test failures for the
 ||MUA i maintain, namely regarding the mentioned charset.  I will
 ||attach a file to reproduce.  (Am not subscribed.)
 ...
 ||  #?0[steffen@devon steffen]$ cksum in.utf=20
 ||  1259742080 686 in.utf
 ||  #?0[steffen@devon steffen]$ iconv -f utf8 -t iso-2022-jp <in.utf|cksum
 ||  2184132317 536
 ||  #?0[steffen@devon steffen]$ iconv --version
 ||  iconv (GNU libiconv 1.11)
 ||..
 ||  #?0[steffen@essex tmp]$ cksum in.utf=20
 ||  1259742080 686 in.utf
 ||  #?0[steffen@essex tmp]$ iconv -f utf8 -t iso-2022-jp <in.utf|cksum=20
 ||  209789743 1736
 ||  #?0[steffen@essex tmp]$ apk info --who-owns /usr/bin/iconv=20
 ||  /usr/bin/iconv is owned by musl-utils-1.1.19-r0

 |Does the data round-trip correctly? I don't think you can expect

Ok, i see what you mean, yes, musl iconv(1) can roundtrip.  But..
But for one the error is new (though i actually have forgotten
whether the test ever ran on a musl box or only on BSD and glibc
Linux boxes, but if i recall, it did run, and then it did succeed,
definetely), and then...

 |bitwise match between outputs of different ISO-2022-JP converters,
 |unless perhaps they both guarantee minimality, because the ISO-2022-JP
 |representation of a string is highly nonunique.
 |
 |In particular musl's to-ISO-2022-JP converter is stateless and always
 |generates shifts in/out around every non-ASCII character. Of course
 |this is highly suboptimal, but in the worst case (where the caller
 |calls iconv one character at a time) the iconv API can't do any better
 |because strings are required to end in the unshifted state, and the
 |iconv API doesn't have any method to "finalize" a conversion. This
 |implies that every time iconv returns with non-ASCII as the most
 |recent output character, it must be followed by a shift back to the
 |initial (ASCII) state.
 |
 |We could improve this in the case of batch conversions by overwriting
 |the previous shift-back-to-initial and skipping the next shift if the
 |character set of the next character to output matches the previous
 |one, but that only works within a single batch call, since iconv can't
 |write outside the buffer passed to it for the current call. This is an
 |improvement I think I want to make, since it would improve typical
 |output size a lot, but the cost is output determinism under different
 |chunking by the caller.

Well...  In my cases the MUA fails to convert to ISO-2022-JP at
all, because an iconv(3) error happens.  And when i instrument my
code like

     for(;;){
        size_t sz;

  fprintf(stderr, "iconv(3): in %lu out: %lu\n",*inbleft,*outbleft);
  fprintf(stderr, "     in<%.*s>\n",(int)*inbleft,*inb);
        sz =3D iconv(cd, __INBCAST(inb), inbleft, outb, outbleft);
        if(sz > 0 && !(icf & n_ICONV_IGN_NOREVERSE)){
  fprintf(stderr, "iconv(3) returned 0x%lX: %s\n",(ul_i)sz,strerror(errno));
           err =3D n_ERR_NOENT;
           goto jleave;
        }
        if(sz !=3D (size_t)-1)
           break;

then i get

  #?1[steffen@essex nail.git]$ v mae-test-behave_iconv_mbyte_base64-2
  ICONV 2
  iconv(3): in 220 out: 427
       in<=E3=82=B7=E3=82=B8=E3=83=A5=E3=82=A6=E3=82=AB=E3=83=A9=E7=A7=91=
=EF=BC=88=E3=82=B7=E3=82=B8=E3=83=A5=E3=82=A6=E3=82=AB=E3=83=A9=E3=81=8B=E3=
=80=81=E5=AD=A6=E5=90=8D Paridae=EF=BC=89=E3=81=AF=E3=80=81=E9=B3=A5=E9=A1=
=9E=E3=82=B9=E3=82=BA=E3=83=A1=E7=9B=AE=E3=81=AE=E7=A7=91=E3=81=A7=E3=81=82=
=E3=82=8B=E3=80=82=E3=82=B7=E3=82=B8=E3=83=A5=E3=82=A6=E3=82=AB=E3=83=A9=EF=
=BC=88=E5=9B=9B=E5=8D=81=E9=9B=80=EF=BC=89=E3=81=A8=E7=B7=8F=E7=A7=B0=E3=81=
=95=E3=82=8C=E3=82=8B=E3=81=8C=E3=80=81=E7=8B=AD=E7=BE=A9=E3=81=AB=E3=81=AF=
=E3=81=93=E3=81=AE1=E7=A8=AE=E3=82=92=E3=82=B7=E3=82=B8=E3=83=A5=E3=82=A6=
=E3=82=AB=E3=83=A9=E3=81=A8=E5=91=BC=E3=81=B6=E3=80=82
  >
  iconv(3) returned 0xFFFFFFFFFFFFFFFF: Argument list too long
  ICONV 2 err: 2

And that is somehow ooops?  Interestingly if i call iconv(1) only
on these 220 bytes i can roundtrip that, too.  Hmmm.  ...
I thought maybe it is because of the tcc(1) compiler i use, but
i can reproduce this with AlpineLinux gcc(1), too.  I don't know.
Ciao,

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)