From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 8969 invoked from network); 25 May 2021 00:46:27 -0000 Received: from mother.openwall.net (195.42.179.200) by inbox.vuxu.org with ESMTPUTF8; 25 May 2021 00:46:27 -0000 Received: (qmail 32008 invoked by uid 550); 25 May 2021 00:46:25 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 31989 invoked from network); 25 May 2021 00:46:24 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JdT5sTkJByDMXaXis2mt0Gqyem881v1OHlcrGLLC9v8=; b=TsywZA2hyHlDW9mAE8mx+EUwnXvTAmJuKG8KZHHlavqkIUIoGJpveqwmnvhrWiPl7f jDdLXX4CAXdD6gjukoH7P0Yzpfw5bVoGPRisGm9oenvLhG4YwaKZPc0iLpjQNMP+ogXC Qnp/EibpX28VMWxwJw3j9DIBvJGaNkDlgOxgzPcQMXhLX1ZVlFcjhVKDYDs4GApm/uHK 26rTudBMw/d2/+fdE+3DwZh/4ABaS3X4HQA7nAdWS9BuOUxQjLzbNtGh0PvMMMneWfQ7 GaC8XK77sXhr9aoVpTsetPlg5U0kWvsQ4BattCQQuw23ybVKK1737iycIjJ+uAOMd02j nGPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JdT5sTkJByDMXaXis2mt0Gqyem881v1OHlcrGLLC9v8=; b=eSTJtugBj9oLGgjbM3HGyYqenp72YXMFj78nF1Aro+rz+w6HTlpVtQ0WLluDRKxhJq o8ovu1BE25YcDyoOu//xVZTxHS7ht7VN8HjIVgfZFenoNsQS525yQTW3Byfgwba0BceH am4NaBbiIR1YW9mf4W0V0SDdpMlNlKmpqdszugrTTsf3BhUNw4kSZeNctLDYnS5OFpE0 ql2WDl+rBqp9NdH3jF4XyxgOWYDjO5TvCyhNKzZopMjjFJL4ONM6NnAcT/Qb31hQtRnQ ct1HEAiGjoZDAbf8baAVa5LQDrFfnJtyANu5psUz/U/GTrCmJR0uLQWoYNVV/vtIHC9m I+yg== X-Gm-Message-State: AOAM532hrXDFpKEfOSupEJSiYb0+Sey+QW2eiKt1tIqs+ZcwOL83ObNt 7lDnVE2cT9n40NooMotYcdavGiy4oMyK00nU5aBTw+J8S4jpmg== X-Google-Smtp-Source: ABdhPJwOWnyzWFA5L0D+oCzGpR5j9JVvheSj24xyvOhIH2WQfIoNUTzgdS4A1xtu/7L7P359nOlDQk+zJBqBQrM0F5M= X-Received: by 2002:a05:6a00:813:b029:27f:fb6a:24b5 with SMTP id m19-20020a056a000813b029027ffb6a24b5mr23469422pfk.18.1621903572501; Mon, 24 May 2021 17:46:12 -0700 (PDT) MIME-Version: 1.0 References: <20210524215021.GC2546@brightrain.aerifal.cx> <20210525003040.GE2546@brightrain.aerifal.cx> In-Reply-To: <20210525003040.GE2546@brightrain.aerifal.cx> From: Konstantin Isakov Date: Mon, 24 May 2021 20:46:01 -0400 Message-ID: To: Rich Felker Cc: musl@lists.openwall.com Content-Type: multipart/alternative; boundary="0000000000004c88ce05c31cdab2" Subject: Re: [musl] [BUG] swprintf() doesn't handle Unicode characters correctly --0000000000004c88ce05c31cdab2 Content-Type: text/plain; charset="UTF-8" Is swprintf() a form of fwprintf() though? fwprintf() and wprintf() output to single-byte streams, so the conversion is necessary there, while swprintf() outputs to a wide buffer. Performing double conversion (to single chars and back) seems like unnecessary work in that case (though, of course, it's less work to implement swprintf() like that). On Mon, May 24, 2021 at 8:30 PM Rich Felker wrote: > On Mon, May 24, 2021 at 08:04:04PM -0400, Konstantin Isakov wrote: > > Thanks for replying! > > > > That fixed it. > > > > I'm surprised, however, that this is required given that in this case > > swprintf() operates on wchars exclusively -- taking wchar arguments and > > producing wchar output. I'd expect that in the worst case scenario it > would > > have to convert from single chars to wide chars, but never the other way > > around, so the representation requirement seems strange. That setlocale() > > step also doesn't seem to be needed with glibc. > > Yes, it's not clear to me whether the glibc behavior is conforming or > not. As specified, > > In addition, all forms of fwprintf() shall fail if: > > [EILSEQ] > A wide-character code that does not correspond > to a valid character has been detected. > > ... > > The "has been detected" wording may allow for the possibility of > ignoring the error, as glibc does, if the function is implemented such > that no conversion takes place (or, for fwprintf, such that conversion > is deferred until flush time) and thus no "detection" takes place. But > it's wrong to assume the operation will succeed. > > In musl, there is no separate wide stdio buffering mode; conversion to > a multibyte sequence happens at (logical) fputwc time, and in the case > of swprintf, conversion (in this case, conversion back) to a wchar_t[] > string occurs at flush time. > > Rich > > > > > > On Mon, May 24, 2021 at 5:50 PM Rich Felker wrote: > > > > > On Mon, May 24, 2021 at 12:39:35AM -0400, Konstantin Isakov wrote: > > > > Hi, > > > > > > > > The following program: > > > > > > > > =================================== > > > > #include > > > > #include > > > > > > > > int main() > > > > { > > > > wchar_t buf[ 32 ]; > > > > > > > > swprintf( buf, sizeof( buf ) / sizeof( *buf ), L"ab\u00E1c" ); > > > > > > > > for ( wchar_t * p = buf; *p; ++p ) > > > > printf( "%u\n", ( unsigned ) *p ); > > > > > > > > return 0; > > > > } > > > > =================================== > > > > > > > > With musl 1.2.2 produces the following output: > > > > 97 > > > > 98 > > > > > > > > The expected output is: > > > > 97 > > > > 98 > > > > 225 > > > > 99 > > > > > > > > With musl, only the first two characters ('a' and 'b') are > processed, and > > > > the string ends on a Unicode character (U+00E1, which is an 'a' with > > > acute > > > > accent), instead of outputting it and the last character, 'c'. > > > > > > > > Please CC me when replying. Thanks! > > > > > > You need to call setlocale(LC_CTYPE, ""). Otherwise the character > > > \u00e1 is unrepresentable, because POSIX requires the C locale be > > > single-byte and you're in the C locale until you call setlocale, and > > > thus produces an encoding error (EILSEQ). > > > > > > Rich > > > > --0000000000004c88ce05c31cdab2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Is swprintf() a form of fwprintf() though? fwprintf() and = wprintf() output to single-byte streams, so the conversion is necessary the= re, while swprintf() outputs to a wide buffer. Performing double conversion= (to single chars and back) seems like unnecessary work in that case (thoug= h, of course, it's less work to implement swprintf() like that).
<= br>
On Mon,= May 24, 2021 at 8:30 PM Rich Felker <dalias@libc.org> wrote:
On Mon, May 24, 2021 at 08:04:04PM -0400, Konstantin Isakov= wrote:
> Thanks for replying!
>
> That fixed it.
>
> I'm surprised, however, that this is required given that in this c= ase
> swprintf() operates on wchars exclusively -- taking wchar arguments an= d
> producing wchar output. I'd expect that in the worst case scenario= it would
> have to convert from single chars to wide chars, but never the other w= ay
> around, so the representation requirement seems strange. That setlocal= e()
> step also doesn't seem to be needed with glibc.

Yes, it's not clear to me whether the glibc behavior is conforming or not. As specified,

=C2=A0 In addition, all forms of fwprintf() shall fail if:

=C2=A0 [EILSEQ]
=C2=A0 =C2=A0 A wide-character code that does not correspond
=C2=A0 =C2=A0 to a valid character has been detected.

=C2=A0 ...

The "has been detected" wording may allow for the possibility of<= br> ignoring the error, as glibc does, if the function is implemented such
that no conversion takes place (or, for fwprintf, such that conversion
is deferred until flush time) and thus no "detection" takes place= . But
it's wrong to assume the operation will succeed.

In musl, there is no separate wide stdio buffering mode; conversion to
a multibyte sequence happens at (logical) fputwc time, and in the case
of swprintf, conversion (in this case, conversion back) to a wchar_t[]
string occurs at flush time.

Rich




> On Mon, May 24, 2021 at 5:50 PM Rich Felker <dalias@libc.org> wrote:
>
> > On Mon, May 24, 2021 at 12:39:35AM -0400, Konstantin Isakov wrote= :
> > > Hi,
> > >
> > > The following program:
> > >
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > #include <stdio.h>
> > > #include <wchar.h>
> > >
> > > int main()
> > > {
> > >=C2=A0 =C2=A0wchar_t buf[ 32 ];
> > >
> > >=C2=A0 =C2=A0swprintf( buf, sizeof( buf ) / sizeof( *buf ), L= "ab\u00E1c" );
> > >
> > >=C2=A0 =C2=A0for ( wchar_t * p =3D buf; *p; ++p )
> > >=C2=A0 =C2=A0 =C2=A0printf( "%u\n", ( unsigned ) *p= );
> > >
> > >=C2=A0 =C2=A0return 0;
> > > }
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > >
> > > With musl 1.2.2 produces the following output:
> > > 97
> > > 98
> > >
> > > The expected output is:
> > > 97
> > > 98
> > > 225
> > > 99
> > >
> > > With musl, only the first two characters ('a' and &#= 39;b') are processed, and
> > > the string ends on a Unicode character (U+00E1, which is an = 'a' with
> > acute
> > > accent), instead of outputting it and the last character, &#= 39;c'.
> > >
> > > Please CC me when replying. Thanks!
> >
> > You need to call setlocale(LC_CTYPE, ""). Otherwise the= character
> > \u00e1 is unrepresentable, because POSIX requires the C locale be=
> > single-byte and you're in the C locale until you call setloca= le, and
> > thus produces an encoding error (EILSEQ).
> >
> > Rich
> >
--0000000000004c88ce05c31cdab2--