[musl] swprintf possible bug

[musl] swprintf possible bug

[Alexander Vitiuk]

[-- Attachment #1: Type: text/plain, Size: 396 bytes --]

Hello!
It seems, wsprintf() / wprintf() are not working in musl as expected, if uses with cyrillic:

C testcase:
#include <wchar.h>
int main() {
  wprintf(L"[hello]\n");
  wprintf(L"[Привет]\n");
 return 0;
}
on x86_64-linux-gnu prints:
[hello]
[Privet]
and on x86_64-linux-musl prints: [hello]
[

There are other cases described:
https://github.com/emscripten-core/emscripten/issues/11947

[-- Attachment #2: Type: text/html, Size: 3576 bytes --]

Re: [musl] swprintf possible bug

[Érico Nogueira]

On Sun Nov 1, 2020 at 6:06 PM -03, Alexander Vitiuk wrote:
> Hello!
> It seems, wsprintf() / wprintf() are not working in musl as expected, if
> uses with cyrillic:
>
> C testcase:
> #include <wchar.h>
> int main() {
> wprintf(L"[hello]\n");
> wprintf(L"[Привет]\n");
> return 0;
> }
> on x86_64-linux-gnu prints:
> [hello]
> [Privet]
> and on x86_64-linux-musl prints: [hello]
> [
>
> There are other cases described:
> https://github.com/emscripten-core/emscripten/issues/11947

For what it's worth, if this is a bug, it would seem to be in how musl
decides when to print characters (not the formatting functions
themselves), since the below program doesn't print anything:

#include <wchar.h>
#include <stdio.h>

int main() {
  fputws(L"[Привет Василий]\n", stdout);
  // I don't know if I'm accessing a wchar_t appropriately here
  fputwc(L"[Привет Василий]\n"[3], stdout);
  return 0;
}

I tried tracing the execution from fputws, and not printing anything
seems to be caused by the return value of wcsrtombs().

Hope this helps,
Érico

Re: [musl] swprintf possible bug

[Szabolcs Nagy]

* Érico Nogueira <ericonr@disroot.org> [2020-11-01 17:17:49 -0300]:
> On Sun Nov 1, 2020 at 6:06 PM -03, Alexander Vitiuk wrote:
> > It seems, wsprintf() / wprintf() are not working in musl as expected, if
> > uses with cyrillic:
> >
> > C testcase:
> > #include <wchar.h>
> > int main() {
> > wprintf(L"[hello]\n");
> > wprintf(L"[Привет]\n");
> > return 0;
> > }
> > on x86_64-linux-gnu prints:
> > [hello]
> > [Privet]
> > and on x86_64-linux-musl prints: [hello]
> > [
> >
> > There are other cases described:
> > https://github.com/emscripten-core/emscripten/issues/11947
> 
> For what it's worth, if this is a bug, it would seem to be in how musl
> decides when to print characters (not the formatting functions
> themselves), since the below program doesn't print anything:
> 
> #include <wchar.h>
> #include <stdio.h>
> 
> int main() {
>   fputws(L"[Привет Василий]\n", stdout);
>   // I don't know if I'm accessing a wchar_t appropriately here
>   fputwc(L"[Привет Василий]\n"[3], stdout);
>   return 0;
> }
> 
> I tried tracing the execution from fputws, and not printing anything
> seems to be caused by the return value of wcsrtombs().

these functions return an error code..

in this case they must return -1 and set errno to EILSEQ,
since the selected multibyte encoding (LC_CTYPE=C) cannot
represent the printed wide characters.

i think the musl behaviour is correct, you can try adding
setlocale(LC_CTYPE,"") at the start of main to make it work.

Re: [musl] swprintf possible bug

[Wolf]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

On 2020-11-01 17:17:49 -0300, Érico Nogueira wrote:
> For what it's worth, if this is a bug, it would seem to be in how musl
> decides when to print characters (not the formatting functions
> themselves), since the below program doesn't print anything:
> 
> #include <wchar.h>
> #include <stdio.h>
> 
> int main() {
>   fputws(L"[Привет Василий]\n", stdout);
>   // I don't know if I'm accessing a wchar_t appropriately here
>   fputwc(L"[Привет Василий]\n"[3], stdout);
>   return 0;
> }
> 
> I tried tracing the execution from fputws, and not printing anything
> seems to be caused by the return value of wcsrtombs().

That seems to be on the right track, since when you actually check the
return code from fputws:

	#include <wchar.h>
	#include <stdio.h>

	int main(void) {
		if (fputws(L"[Привет Василий]\n", stdout) == -1) {
			perror("fputws");
		}
	}

you get this:

	# ./a
	fputws: Illegal byte sequence

I think it is caused by C (or POSIX) locale being default on entry to
main, so you need to actually activate the system locale by calling
setlocale(LC_ALL, "").

Indeed, if you modify the program to:

	#include <wchar.h>
	#include <stdio.h>
	#include <locale.h>

	int main(void) {
		setlocale(LC_ALL, "");
		if (fputws(L"[Привет Василий]\n", stdout) == -1) {
			perror("fputws");
		}
	}

It starts to work:

	# ./a
	[Привет Василий]


W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [musl] swprintf possible bug

[Érico Nogueira]

On Sun Nov 1, 2020 at 6:40 PM -03, Szabolcs Nagy wrote:
> * Érico Nogueira <ericonr@disroot.org> [2020-11-01 17:17:49 -0300]:
> > On Sun Nov 1, 2020 at 6:06 PM -03, Alexander Vitiuk wrote:
> > > It seems, wsprintf() / wprintf() are not working in musl as expected, if
> > > uses with cyrillic:
> > >
> > > C testcase:
> > > #include <wchar.h>
> > > int main() {
> > > wprintf(L"[hello]\n");
> > > wprintf(L"[Привет]\n");
> > > return 0;
> > > }
> > > on x86_64-linux-gnu prints:
> > > [hello]
> > > [Privet]
> > > and on x86_64-linux-musl prints: [hello]
> > > [
> > >
> > > There are other cases described:
> > > https://github.com/emscripten-core/emscripten/issues/11947
> > 
> > For what it's worth, if this is a bug, it would seem to be in how musl
> > decides when to print characters (not the formatting functions
> > themselves), since the below program doesn't print anything:
> > 
> > #include <wchar.h>
> > #include <stdio.h>
> > 
> > int main() {
> >   fputws(L"[Привет Василий]\n", stdout);
> >   // I don't know if I'm accessing a wchar_t appropriately here
> >   fputwc(L"[Привет Василий]\n"[3], stdout);
> >   return 0;
> > }
> > 
> > I tried tracing the execution from fputws, and not printing anything
> > seems to be caused by the return value of wcsrtombs().
>
> these functions return an error code..
>
> in this case they must return -1 and set errno to EILSEQ,
> since the selected multibyte encoding (LC_CTYPE=C) cannot
> represent the printed wide characters.
>
> i think the musl behaviour is correct, you can try adding
> setlocale(LC_CTYPE,"") at the start of main to make it work.

Thanks, that did fix it. For reference:

#include <wchar.h>
#include <stdio.h>
#include <locale.h>

int main() {
  setlocale(LC_CTYPE, "");
  fputws(L"[Привет Василий]\n", stdout);
  fputwc(L"[Привет Василий]\n"[3], stdout);
  return 0;
}

I wonder what glibc's behavior is that it allows this; and how
emscripten folks can work around the musl behavior as well.

Which environment variables could I set to control this, or is that not
possible?

Thanks,
Érico

Re: [musl] swprintf possible bug

[Rich Felker]

On Sun, Nov 01, 2020 at 05:48:43PM -0300, Érico Nogueira wrote:
> On Sun Nov 1, 2020 at 6:40 PM -03, Szabolcs Nagy wrote:
> > * Érico Nogueira <ericonr@disroot.org> [2020-11-01 17:17:49 -0300]:
> > > On Sun Nov 1, 2020 at 6:06 PM -03, Alexander Vitiuk wrote:
> > > > It seems, wsprintf() / wprintf() are not working in musl as expected, if
> > > > uses with cyrillic:
> > > >
> > > > C testcase:
> > > > #include <wchar.h>
> > > > int main() {
> > > > wprintf(L"[hello]\n");
> > > > wprintf(L"[Привет]\n");
> > > > return 0;
> > > > }
> > > > on x86_64-linux-gnu prints:
> > > > [hello]
> > > > [Privet]
> > > > and on x86_64-linux-musl prints: [hello]
> > > > [
> > > >
> > > > There are other cases described:
> > > > https://github.com/emscripten-core/emscripten/issues/11947
> > > 
> > > For what it's worth, if this is a bug, it would seem to be in how musl
> > > decides when to print characters (not the formatting functions
> > > themselves), since the below program doesn't print anything:
> > > 
> > > #include <wchar.h>
> > > #include <stdio.h>
> > > 
> > > int main() {
> > >   fputws(L"[Привет Василий]\n", stdout);
> > >   // I don't know if I'm accessing a wchar_t appropriately here
> > >   fputwc(L"[Привет Василий]\n"[3], stdout);
> > >   return 0;
> > > }
> > > 
> > > I tried tracing the execution from fputws, and not printing anything
> > > seems to be caused by the return value of wcsrtombs().
> >
> > these functions return an error code..
> >
> > in this case they must return -1 and set errno to EILSEQ,
> > since the selected multibyte encoding (LC_CTYPE=C) cannot
> > represent the printed wide characters.
> >
> > i think the musl behaviour is correct, you can try adding
> > setlocale(LC_CTYPE,"") at the start of main to make it work.
> 
> Thanks, that did fix it. For reference:
> 
> #include <wchar.h>
> #include <stdio.h>
> #include <locale.h>
> 
> int main() {
>   setlocale(LC_CTYPE, "");
>   fputws(L"[Привет Василий]\n", stdout);
>   fputwc(L"[Привет Василий]\n"[3], stdout);
>   return 0;
> }
> 
> I wonder what glibc's behavior is that it allows this; and how
> emscripten folks can work around the musl behavior as well.
> 
> Which environment variables could I set to control this, or is that not
> possible?

There are no environment variables that control the initial state of
the program before setlocale() is called. The environment variables
only affect the behavior of setlocale when it requests the
system's/user's default locale ("").

If you don't want the behavior to be dependent on locale, don't use
the wide functions. fputs("[Привет Василий]\n", stdout) would work
regardless of locale and would simply output the byte sequence. Of
course to be meaningful this depends on both the translation charset
and runtime charset being the same; they should of course both be
UTF-8 anytime in this millennium.

Rich