From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/11798 Path: news.gmane.org!.POSTED!not-for-mail From: Mikhail Kremnyov Newsgroups: gmane.linux.lib.musl.general Subject: Re: Issues in mbsnrtowcs and wcsnrtombs Date: Wed, 9 Aug 2017 20:57:27 +0300 Message-ID: References: Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------1B62E160D1E5A7123DAD0010" X-Trace: blaine.gmane.org 1502301496 14987 195.159.176.226 (9 Aug 2017 17:58:16 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 9 Aug 2017 17:58:16 +0000 (UTC) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 To: musl@lists.openwall.com Original-X-From: musl-return-11811-gllmg-musl=m.gmane.org@lists.openwall.com Wed Aug 09 19:58:12 2017 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1dfVF7-0002Gl-TX for gllmg-musl@m.gmane.org; Wed, 09 Aug 2017 19:58:09 +0200 Original-Received: (qmail 26136 invoked by uid 550); 9 Aug 2017 17:57:46 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 26103 invoked from network); 9 Aug 2017 17:57:44 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:references:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=/bIJJDFqpg27+zNaN1RD/Estx77N5umPDEKu0Vyjay8=; b=oNaHku7M8QrSYMwtoiMM6w/6MmHtieJbMJO5Bk/giOK3PysE/XGI/eI6nEQkjQgoNT 1n9AshQT/PNhvXSAXr8mWVRW7jOypLSPPDZtjKXEXeKU9+jlrzy2qiOQ2CCJiBFh3Wcl qfzD5SrEIc/Ec/Et3Dh1YlK09cpSG0Pi6Re5TcaX3YYjEk4v8RzSrCEcL/yvZrjfny3W EyRF0xCVFgOAOHqzNJTwaN8kuI/kEh5NOXN1C+RI57zfxljzpp3J1FPvA75SyDIFJyjG Pu1bJ9RD3N8lGdZ4Bz4Evji8QyQ2pmqUfI6r645AzJ9HDEtYSEAacQ+aEOPS5XqESX7c 1jew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=/bIJJDFqpg27+zNaN1RD/Estx77N5umPDEKu0Vyjay8=; b=aSqeQXwebYgz0BY8VTuyeH8rDTxtfJPBgKQ7bBsmK/rPHLUNkwWt78XwCJRJ6ZChV3 DV6v0T05x1k8jO8q3rZXB62DYxeqSNjt4Vc2+TCpnnDKNMIByC6NWgY4DFo7DHpJtxT3 P1itnsqlG0WpoqdA6qTIjuj38AFBhFcXePlRneoDDOr145ge30upbH3JE6a8msMJmqff rDfuXEZOBd6S/AJdHwxA+MwsYtPn0OyuvbhEgffKg8okprp4rQGby80XnQZhfW8g1lGh prCLQz/VqMeIdg6j0ldeZ0eV9D2JEi+LBKlgQEkSu5yfewRHOXz73GO4t0ofw00d/Vku LjZw== X-Gm-Message-State: AHYfb5iu+Z4ojMZm4T41awaSCEDsGe1RD62aWkuKf/rK8MBgCLW+AhHP Fof2o23k3OODakEI X-Received: by 10.25.19.85 with SMTP id j82mr2753018lfi.155.1502301450581; Wed, 09 Aug 2017 10:57:30 -0700 (PDT) In-Reply-To: Content-Language: en-GB Xref: news.gmane.org gmane.linux.lib.musl.general:11798 Archived-At: This is a multi-part message in MIME format. --------------1B62E160D1E5A7123DAD0010 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 SGksCgpIZXJlIGFyZSB0d28gcGF0Y2hlcywgb25lIChmb3IgbGliY190ZXN0KSBhZGRzIGEg Y291cGxlIG9mIHRlc3RzIHRvCnJlcHJvZHVjZSB0aGUgaXNzdWVzLCB0aGUgb3RoZXIgb25l IGZpeGVzIHRoZW0uCgpNaWtoYWlsLgoKT24gMDcvMTgvMjAxNyAxMTowNSBQTSwgTWlraGFp bCBLcmVtbnlvdiB3cm90ZToKPiBIaSwKPgo+IEl0IGxvb2tzIGxpa2UgdGhlcmUgYXJlIHNv bWUgYnVncyBpbiB0aGUgaW1wbGVtZW50YXRpb25zIG9mIG1ic25ydG93Y3MKPiBhbmQgd2Nz bnJ0b21icy4KPiBFLmcuIGluc2lkZSBtYnNucnRvd2NzIHRoZXJlIGlzIHRoaXMgY29kZToK Pgo+ICAgICB3aGlsZSAoIHMgJiYgd24gJiYgKCAobjI9bi80KT49d24gfHwgbjI+MzIgKSAp IHsKPiAgICAgICAgIGlmIChuMj49d24pIG4yPXduOwo+ICAgICAgICAgbiAtPSBuMjsKPiAg ICAgICAgIGwgPSBtYnNydG93Y3Mod3MsICZzLCBuMiwgc3QpOwo+Cj4gSGVyZSAibiIgaXMg dGhlIG51bWJlciBvZiBzb3VyY2UgYnl0ZXMgdG8gY29udmVydCBhbmQgIm4yIiBpcyB0aGUg bnVtYmVyCj4gb2Ygd2lkZSBjaGFycyB0aGF0IG1heSBiZSBwdXQgdG8gdGhlIGRlc3RpbmF0 aW9uLCBzbyBpdCdzIGluY29ycmVjdCB0bwo+IHN1YnRyYWN0IG9uZSBmcm9tIGFub3RoZXIu IEFuZCBpbmRlZWQgYSBzaW1wbGUgdGVzdCBzaG93cyB0aGF0IHRoZQo+IGZ1bmN0aW9uIGRv ZXNuJ3Qgd29yayBjb3JyZWN0bHkgaWYgbG9uZyBlbm91Z2ggbm9uLWFzY2lpIHN0cmluZyBp cwo+IHBhc3NlZCB0byBpdC4gRS5nLjoKPgo+ICAgICBjb25zdCBzdGQ6OnN0cmluZyBvcmln U3RyID0KPiB1OCLQsNCx0LLQs9C00LXRkdC20LfQuNC50LrQu9C80L3QvtC/0YDRgdGC0YPR hNGF0YbRh9GI0YnRitGL0YzRjdGO0Y/QkNCR0JLQk9CU0JXQgdCW0JfQmNCZ0JrQm9Cc0J3Q ntCf0KDQodCi0KPQpNCl0KbQp9Co0KnQqtCr0KzQrdCu0K8iOwo+ICAgICBjb25zdCBzdGQ6 OnN0cmluZyBzcmNTdHIgPSBvcmlnU3RyICsgdTgi0ZfSkdGW0ZQiOwo+Cj4gICAgIHN0ZDo6 bWJzdGF0ZV90IHN0ID0ge307Cj4gICAgIGNvbnN0IGNoYXIqIHNyY1B0ciA9ICZzcmNTdHJb MF07Cj4gICAgIHN0ZDo6d3N0cmluZyBkZXN0KHNyY1N0ci5sZW5ndGgoKSArIDEsIHdjaGFy X3QoMCkpOwo+Cj4gICAgIGF1dG8gcmVzID0gbWJzbnJ0b3djcygmZGVzdFswXSwgJnNyY1B0 ciwgb3JpZ1N0ci5sZW5ndGgoKSwKPiBkZXN0Lmxlbmd0aCgpLCAmc3QpOwo+Cj4gICAgIHN0 ZDo6Y291dCA8PCAicmVzID0gIiA8PCByZXMgPDwgIiwgc3JjUHRyID0gIiA8PCAodm9pZCop c3JjUHRyIDw8Cj4gc3RkOjplbmRsOwo+Cj4gQW5kIHRoZSBvdXRwdXQgaXM6Cj4gICAgIHJl cyA9IDcwLCBzcmNQdHIgPSAwCj4KPiBIZXJlIG1ic25ydG93Y3Mgd2FzIHRvbGQgdG8gY29u dmVydCBvbmx5ICJvcmlnU3RyLmxlbmd0aCgpIiBudW1iZXIgb2YKPiBieXRlcywgd2hpY2gg Y29udGFpbiA2NiAyLWJ5dGUgY2hhcmFjdGVycywgYnV0IGl0IGNvbnZlcnRlZCA3MCwgc3Rv cHBpbmcKPiBvbmx5IGFmdGVyIHRoZSB6ZXJvIGNoYXIgd2FzIG1ldC4KPgo+IEEgc2ltaWxh ciBwcm9ibGVtIGhhcHBlbnMgd2l0aCB3Y3NucnRvbWJzIHVzaW5nIGEgc2xpZ2h0bHkgbG9u Z2VyIHN0cmluZzoKPgo+ICAgICBzdGQ6OndzdHJpbmcgc3JjU3RyID0KPiBMItCw0LHQstCz 0LTQtdGR0LbQt9C40LnQutC70LzQvdC+0L/RgNGB0YLRg9GE0YXRhtGH0YjRidGK0YvRjNGN 0Y7Rj9CQ0JHQktCT0JTQldCB0JbQl9CY0JnQmtCb0JzQndCe0J/QoNCh0KLQo9Ck0KXQptCn 0KjQqdCq0KvQrNCt0K7Qr9Cw0LHQstCz0LTQtdGR0LbQt9C40LnQutC70LzQvdC+0L/RgNGB 0YLRg9GE0YXRhtGH0YjRidGK0YvRjNGN0Y7RjyI7Cj4KPiAgICAgY29uc3Qgd2NoYXJfdCog c3JjUHRyID0gJnNyY1N0clswXTsKPiAgICAgc3RkOjptYnN0YXRlX3Qgc3QgPSB7fTsKPiAg ICAgc3RkOjpzdHJpbmcgZGVzdChzcmNTdHIubGVuZ3RoKCkgKiA0ICsgMSwgY2hhcigwKSk7 Cj4KPiAgICAgYXV0byByZXMgPSB3Y3NucnRvbWJzKCZkZXN0WzBdLCAmc3JjUHRyLCBzcmNT dHIubGVuZ3RoKCksCj4gZGVzdC5sZW5ndGgoKSwgJnN0KTsKPgo+ICAgICBzdGQ6OmNvdXQg PDwgInJlcyA9ICIgPDwgcmVzIDw8ICIsIGRlc3QgPSAiIDw8IGRlc3QgPDwgc3RkOjplbmRs Owo+Cj4gVGhlIG91dHB1dDoKPiAgICAgcmVzID0gOTgsIGRlc3QgPSDQsNCx0LLQs9C00LXR kdC20LfQuNC50LrQu9C80L3QvtC/0YDRgdGC0YPRhNGF0YbRh9GI0YnRitGL0YzRjdGO0Y/Q kNCR0JLQk9CU0JXQgdCW0JfQmNCZ0JrQm9Cc0J3Qngo+ICAgIAo+IEkuZS4gaXQgb25seSBj b252ZXJ0ZWQgNDkgY2hhcmFjdGVycyBpbnN0ZWFkIG9mIDk5Lgo+Cj4KPiBNaWtoYWlsLgo+ Cj4KCg== --------------1B62E160D1E5A7123DAD0010 Content-Type: text/x-patch; name="libc_test.diff" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="libc_test.diff" --- ./src/regression/mbsnrtowcs-overread.c 1970-01-01 03:00:00.000000000 = +0300 +++ ./src/regression/mbsnrtowcs-overread.c 2017-08-09 20:20:29.472003066 = +0300 @@ -0,0 +1,45 @@ +// mbsnrtowcs issue, reported in www.openwall.com/lists/musl/2017/07/18/= 3 +#include +#include +#include +#include +#include "test.h" + +int main(void) +{ + const char *const chr =3D "\u044B"; + const int chr_size =3D strlen(chr); + // The passed length of the source string in bytes should be bigger tha= n + // 32*4 to force mbsnrtowcs to use the optimization based on mbsrtowcs.= + const int chr_count_to_convert =3D 1000; + // Make sure that the source string has more characters after the passe= d + // length. + const int chr_count =3D chr_count_to_convert + 10; +=09 + char src[chr_count * chr_size + 1]; + // dest should also have extra space + wchar_t dest[chr_count + 1]; + size_t r; + const char *str_ptr =3D src; + mbstate_t mbs; + + for (int i =3D 0; i < chr_count; ++i) + { + memcpy(src + i * chr_size, chr, chr_size); + } + src[chr_count * chr_size] =3D 0; +=20 + setlocale(LC_CTYPE, "en_US.UTF-8"); + + memset(&mbs, 0, sizeof(mbs)); + r =3D mbsnrtowcs(dest, &str_ptr, chr_count_to_convert * chr_size, + sizeof(dest)/sizeof(dest[0]), &mbs); + + if (r !=3D chr_count_to_convert) + { + t_error("Expected to convert %d characters, but converted %d\n", + chr_count_to_convert, r); + } + + return t_status; +} --- ./src/regression/wcsnrtombs_underread.c 1970-01-01 03:00:00.000000000= +0300 +++ ./src/regression/wcsnrtombs_underread.c 2017-08-09 20:24:57.575995227= +0300 @@ -0,0 +1,46 @@ +// wcsnrtombs issue, reported in www.openwall.com/lists/musl/2017/07/18/= 3 +#include +#include +#include +#include +#include "test.h" + +#define TEST_CHR "\u044B" + +#define CAT_IMPL(x, y) x##y +#define CAT(x, y) CAT_IMPL(x, y) + +int main(void) +{ + const wchar_t *const chr =3D CAT(L, TEST_CHR); + const int chr_len_in_utf_8 =3D strlen(TEST_CHR); + const int chr_size =3D wcslen(chr); + // The number of characters should be greater than 32 to force wcsnrtom= bs + // to use the optimization based on wcsrtombs. + const int chr_count =3D 1000; + wchar_t src[chr_count]; + char dest[chr_count * 4]; + size_t r; + const wchar_t *str_ptr =3D src; + mbstate_t mbs; + + for (int i =3D 0; i < chr_count; ++i) + { + memcpy(src + i, chr, sizeof(*chr)); + } + src[chr_count] =3D 0; + + setlocale(LC_CTYPE, "en_US.UTF-8"); + + memset(&mbs, 0, sizeof(mbs)); + r =3D wcsnrtombs(dest, &str_ptr, chr_count, + sizeof(dest), &mbs); + + if (r !=3D chr_count * chr_len_in_utf_8) + { + t_error("Expected to fill %d bytes, but filled %d\n", + chr_count * chr_len_in_utf_8, r); + } + + return t_status; +} --------------1B62E160D1E5A7123DAD0010 Content-Type: text/x-patch; name="musl.diff" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="musl.diff" --- ./src/multibyte/mbsnrtowcs.c 2017-08-08 16:19:29.311584832 +0300 +++ ./src/multibyte/mbsnrtowcs.c 2017-08-09 20:33:27.515980317 +0300 @@ -5,6 +5,7 @@ size_t l, cnt=3D0, n2; wchar_t *ws, wbuf[256]; const char *s =3D *src; + const char *tmp_s; =20 if (!wcs) ws =3D wbuf, wn =3D sizeof wbuf / sizeof *wbuf; else ws =3D wcs; @@ -15,7 +16,7 @@ =20 while ( s && wn && ( (n2=3Dn/4)>=3Dwn || n2>32 ) ) { if (n2>=3Dwn) n2=3Dwn; - n -=3D n2; + tmp_s =3D s; l =3D mbsrtowcs(ws, &s, n2, st); if (!(l+1)) { cnt =3D l; @@ -26,6 +27,7 @@ ws +=3D l; wn -=3D l; } + n =3D s ? n - (s - tmp_s) : 0; cnt +=3D l; } if (s) while (wn && n) { --- ./src/multibyte/wcsnrtombs.c 2017-08-08 16:19:29.311584832 +0300 +++ ./src/multibyte/wcsnrtombs.c 2017-08-09 20:34:17.479978856 +0300 @@ -5,13 +5,14 @@ size_t l, cnt=3D0, n2; char *s, buf[256]; const wchar_t *ws =3D *wcs; + const wchar_t *tmp_ws; =20 if (!dst) s =3D buf, n =3D sizeof buf; else s =3D dst; =20 while ( ws && n && ( (n2=3Dwn)>=3Dn || n2>32 ) ) { if (n2>=3Dn) n2=3Dn; - wn -=3D n2; + tmp_ws =3D ws; l =3D wcsrtombs(s, &ws, n2, 0); if (!(l+1)) { cnt =3D l; @@ -22,6 +23,7 @@ s +=3D l; n -=3D l; } + wn =3D ws ? wn - (ws - tmp_ws) : 0; cnt +=3D l; } if (ws) while (n && wn) { --------------1B62E160D1E5A7123DAD0010--