mailing list of musl libc
 help / color / mirror / Atom feed
* [musl] Question on C++ locale
@ 2020-11-30 10:41 Dong Brett
  2020-11-30 11:31 ` Szabolcs Nagy
                   ` (3 more replies)
  0 siblings, 4 replies; 17+ messages in thread
From: Dong Brett @ 2020-11-30 10:41 UTC (permalink / raw)
  To: musl; +Cc: Binrui Dong

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

Hi all,

I am troubleshooting a locale related issue of our C++ software when building with musl. With some efforts I narrowed our problem down to the inability of setting a UTF-8 locale in C++ standard library.

The following C code prints UTF-8 characters correctly:
#include <ncurses.h>
#include <langinfo.h>
#include <locale.h>

int main()
{
    setlocale(LC_ALL, "");
    initscr();
    printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
    printw("CODESET: %s\n", nl_langinfo(CODESET));
    printw("Hello, world!\n");
    printw("你好,世界!\n");
    refresh();
    getch();
    endwin();
    return 0;
}

Giving the output of
LC_ALL: C.UTF-8;C;C;C;C;C
CODESET: UTF-8
Hello, world!
你好,世界!

However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
#include <langinfo.h>
#include <locale.h>
#include <locale>
using namespace std;
int main()
{
    std::locale::global(locale(""));
    initscr();
    printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
    printw("C++ locale: %s\n", locale().name().c_str());
    printw("CODESET: %s\n", nl_langinfo(CODESET));
    printw("Hello, world!\n");
    printw("你好,世界!\n");
    refresh();
    getch();
    endwin();
    return 0;
}

Giving a corrupted output:
LC_ALL: C
C++ locale: C
CODESET: ASCII
Hello, world!
你好?~L?~V?~U~L!

Seems only ASCII C locale is available in C++. If I run the above C++ code with LANG="C.UTF-8", an exception is thrown and the program is aborted:
terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
Aborted

I also tried LANG="UTF-8”, LANG="en_US.UTF-8" but none of those works. Only LANG="C" could make the program run but then only ASCII characters are supported.

My question is that is there a way to make locale in C++ standard library work with musl? Or had I done anything wrong with it?

Regards,
Brett

[-- Attachment #2: Type: text/html, Size: 12323 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 10:41 [musl] Question on C++ locale Dong Brett
@ 2020-11-30 11:31 ` Szabolcs Nagy
  2020-11-30 11:58   ` Dong Brett
  2020-11-30 11:35 ` Szabolcs Nagy
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 17+ messages in thread
From: Szabolcs Nagy @ 2020-11-30 11:31 UTC (permalink / raw)
  To: Dong Brett; +Cc: musl

* Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33 +0800]:
> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> #include <langinfo.h>
> #include <locale.h>
> #include <locale>
> using namespace std;
> int main()
> {
>     std::locale::global(locale(""));
>     initscr();
>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
>     printw("C++ locale: %s\n", locale().name().c_str());
>     printw("CODESET: %s\n", nl_langinfo(CODESET));
>     printw("Hello, world!\n");
>     printw("你好,世界!\n");
>     refresh();
>     getch();
>     endwin();
>     return 0;
> }

what is initscr, printw, refresh, getch, endwin??

none of these are libc apis.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 10:41 [musl] Question on C++ locale Dong Brett
  2020-11-30 11:31 ` Szabolcs Nagy
@ 2020-11-30 11:35 ` Szabolcs Nagy
  2020-11-30 12:37   ` Érico Nogueira
  2020-11-30 13:44   ` Érico Nogueira
  2020-11-30 14:51 ` Rich Felker
  2020-12-01  3:53 ` [musl] " Dong Brett
  3 siblings, 2 replies; 17+ messages in thread
From: Szabolcs Nagy @ 2020-11-30 11:35 UTC (permalink / raw)
  To: Dong Brett; +Cc: musl

* Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33 +0800]:
> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> #include <langinfo.h>
> #include <locale.h>
> #include <locale>
> using namespace std;
> int main()
> {
>     std::locale::global(locale(""));
>     initscr();
>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
>     printw("C++ locale: %s\n", locale().name().c_str());
>     printw("CODESET: %s\n", nl_langinfo(CODESET));
>     printw("Hello, world!\n");
>     printw("你好,世界!\n");
>     refresh();
>     getch();
>     endwin();
>     return 0;
> }

fwiw for me even the first line fails.
i don't know how c++ locales are supposed to work.

$ cat a.cc
#include <locale>
using namespace std;
int main()
{
    std::locale::global(locale(""));
    return 0;
}

$ g++ a.cc
$ ./a.out 
terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
Aborted

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 11:31 ` Szabolcs Nagy
@ 2020-11-30 11:58   ` Dong Brett
  0 siblings, 0 replies; 17+ messages in thread
From: Dong Brett @ 2020-11-30 11:58 UTC (permalink / raw)
  To: Szabolcs Nagy; +Cc: musl

These are functions from ncurses library.

> On 30 Nov 2020, at 19:31, Szabolcs Nagy <nsz@port70.net> wrote:
> 
> what is initscr, printw, refresh, getch, endwin??
> 
> none of these are libc apis.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 11:35 ` Szabolcs Nagy
@ 2020-11-30 12:37   ` Érico Nogueira
  2020-11-30 13:44   ` Érico Nogueira
  1 sibling, 0 replies; 17+ messages in thread
From: Érico Nogueira @ 2020-11-30 12:37 UTC (permalink / raw)
  To: musl, Dong Brett

On Mon Nov 30, 2020 at 8:35 AM -03, Szabolcs Nagy wrote:
> * Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33
> +0800]:
> > However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> > #include <langinfo.h>
> > #include <locale.h>
> > #include <locale>
> > using namespace std;
> > int main()
> > {
> >     std::locale::global(locale(""));
> >     initscr();
> >     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> >     printw("C++ locale: %s\n", locale().name().c_str());
> >     printw("CODESET: %s\n", nl_langinfo(CODESET));
> >     printw("Hello, world!\n");
> >     printw("你好,世界!\n");
> >     refresh();
> >     getch();
> >     endwin();
> >     return 0;
> > }
>
> fwiw for me even the first line fails.
> i don't know how c++ locales are supposed to work.
>
> $ cat a.cc
> #include <locale>
> using namespace std;
> int main()
> {
> std::locale::global(locale(""));
> return 0;
> }
>
> $ g++ a.cc
> $ ./a.out
> terminate called after throwing an instance of 'std::runtime_error'
> what(): locale::facet::_S_create_c_locale name not valid
> Aborted

For an example of this issue "in the wild", so to speak, I can only
launch supertux2 [1] with LANG=C. Exact same error message otherwise.

- [1] https://github.com/SuperTux/supertux

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 11:35 ` Szabolcs Nagy
  2020-11-30 12:37   ` Érico Nogueira
@ 2020-11-30 13:44   ` Érico Nogueira
  2020-11-30 14:39     ` Samuel Holland
  1 sibling, 1 reply; 17+ messages in thread
From: Érico Nogueira @ 2020-11-30 13:44 UTC (permalink / raw)
  To: musl, Dong Brett

On Mon Nov 30, 2020 at 8:35 AM -03, Szabolcs Nagy wrote:
> * Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33
> +0800]:
> > However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> > #include <langinfo.h>
> > #include <locale.h>
> > #include <locale>
> > using namespace std;
> > int main()
> > {
> >     std::locale::global(locale(""));
> >     initscr();
> >     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> >     printw("C++ locale: %s\n", locale().name().c_str());
> >     printw("CODESET: %s\n", nl_langinfo(CODESET));
> >     printw("Hello, world!\n");
> >     printw("你好,世界!\n");
> >     refresh();
> >     getch();
> >     endwin();
> >     return 0;
> > }
>
> fwiw for me even the first line fails.
> i don't know how c++ locales are supposed to work.

From [1], it seems that C++ locales are supposed to affect the global
locale as well, so they should call setlocale() when appropriate.

- [1] https://www.cplusplus.com/reference/locale/locale/

Unfortunately, I assume libstdc++ uses their generic locale support on
musl...  From gcc-10.2.0/libstdc++-v3/config/locale/generic/c_locale.cc:

  void
  locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
				    __c_locale)
  {
    // Currently, the generic model only supports the "C" locale.
    // See http://gcc.gnu.org/ml/libstdc++/2003-02/msg00345.html
    __cloc = 0;
    if (strcmp(__s, "C"))
      __throw_runtime_error(__N("locale::facet::_S_create_c_locale "
			    "name not valid"));
  }

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 13:44   ` Érico Nogueira
@ 2020-11-30 14:39     ` Samuel Holland
  2020-11-30 14:48       ` Rich Felker
  2020-11-30 15:12       ` Érico Nogueira
  0 siblings, 2 replies; 17+ messages in thread
From: Samuel Holland @ 2020-11-30 14:39 UTC (permalink / raw)
  To: musl, Érico Nogueira, Dong Brett

On 11/30/20 7:44 AM, Érico Nogueira wrote:
> On Mon Nov 30, 2020 at 8:35 AM -03, Szabolcs Nagy wrote:
>> * Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33
>> +0800]:
>>> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
>>> #include <langinfo.h>
>>> #include <locale.h>
>>> #include <locale>
>>> using namespace std;
>>> int main()
>>> {
>>>     std::locale::global(locale(""));
>>>     initscr();
>>>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
>>>     printw("C++ locale: %s\n", locale().name().c_str());
>>>     printw("CODESET: %s\n", nl_langinfo(CODESET));
>>>     printw("Hello, world!\n");
>>>     printw("你好,世界!\n");
>>>     refresh();
>>>     getch();
>>>     endwin();
>>>     return 0;
>>> }
>>
>> fwiw for me even the first line fails.
>> i don't know how c++ locales are supposed to work.
> 
> From [1], it seems that C++ locales are supposed to affect the global
> locale as well, so they should call setlocale() when appropriate.
> 
> - [1] https://www.cplusplus.com/reference/locale/locale/
> 
> Unfortunately, I assume libstdc++ uses their generic locale support on
> musl...  From gcc-10.2.0/libstdc++-v3/config/locale/generic/c_locale.cc:
> 
>   void
>   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> 				    __c_locale)
>   {
>     // Currently, the generic model only supports the "C" locale.
>     // See http://gcc.gnu.org/ml/libstdc++/2003-02/msg00345.html
>     __cloc = 0;
>     if (strcmp(__s, "C"))
>       __throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> 			    "name not valid"));
>   }
> 

I don't know for sure that it's the right thing to do, but I have been patching
out that error for the last several years[1] and so far I have not noticed any
negative effects. Adelie, which is very thorough about testing, has also carried
the patch for a while[2].

Samuel

[1]:
https://github.com/smaeul/portage/blob/c744774a/patches/sys-devel/gcc/gcc-5.4.0-locale.patch
[2]: https://code.foxkit.us/adelie/packages/-/commit/d09b437d

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 14:39     ` Samuel Holland
@ 2020-11-30 14:48       ` Rich Felker
  2020-11-30 15:12       ` Érico Nogueira
  1 sibling, 0 replies; 17+ messages in thread
From: Rich Felker @ 2020-11-30 14:48 UTC (permalink / raw)
  To: Samuel Holland; +Cc: musl, Érico Nogueira, Dong Brett

On Mon, Nov 30, 2020 at 08:39:18AM -0600, Samuel Holland wrote:
> On 11/30/20 7:44 AM, Érico Nogueira wrote:
> > On Mon Nov 30, 2020 at 8:35 AM -03, Szabolcs Nagy wrote:
> >> * Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33
> >> +0800]:
> >>> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> >>> #include <langinfo.h>
> >>> #include <locale.h>
> >>> #include <locale>
> >>> using namespace std;
> >>> int main()
> >>> {
> >>>     std::locale::global(locale(""));
> >>>     initscr();
> >>>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> >>>     printw("C++ locale: %s\n", locale().name().c_str());
> >>>     printw("CODESET: %s\n", nl_langinfo(CODESET));
> >>>     printw("Hello, world!\n");
> >>>     printw("你好,世界!\n");
> >>>     refresh();
> >>>     getch();
> >>>     endwin();
> >>>     return 0;
> >>> }
> >>
> >> fwiw for me even the first line fails.
> >> i don't know how c++ locales are supposed to work.
> > 
> > From [1], it seems that C++ locales are supposed to affect the global
> > locale as well, so they should call setlocale() when appropriate.
> > 
> > - [1] https://www.cplusplus.com/reference/locale/locale/
> > 
> > Unfortunately, I assume libstdc++ uses their generic locale support on
> > musl...  From gcc-10.2.0/libstdc++-v3/config/locale/generic/c_locale.cc:
> > 
> >   void
> >   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> > 				    __c_locale)
> >   {
> >     // Currently, the generic model only supports the "C" locale.
> >     // See http://gcc.gnu.org/ml/libstdc++/2003-02/msg00345.html
> >     __cloc = 0;
> >     if (strcmp(__s, "C"))
> >       __throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> > 			    "name not valid"));
> >   }
> > 
> 
> I don't know for sure that it's the right thing to do, but I have been patching
> out that error for the last several years[1] and so far I have not noticed any
> negative effects. Adelie, which is very thorough about testing, has also carried
> the patch for a while[2].
> 
> Samuel
> 
> [1]:
> https://github.com/smaeul/portage/blob/c744774a/patches/sys-devel/gcc/gcc-5.4.0-locale.patch
> [2]: https://code.foxkit.us/adelie/packages/-/commit/d09b437d

That libstdc++ code is definitely wrong and should be removed, but I'm
not sure what other bugs in libstdc++ might be hiding behind it. Next
time you find something like this please send the patches here for
inclusion in mcm and proper bug reporting to GCC.

Note that the message cited in the comment is just wrong; generic
should work fine with the modern POSIX locale API
(newlocale/uselocale) which I was nearly sure libstdc++ was already
using. If this isn't the case we need to fix that.

In any case, setlocale should be used directly in the above code, not
C++ locale framework, since curses is a C library whose behavior is
defined by the locale set through the C mechanisms not whatever wacky
stuff libstdc++ does.

Rich

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 10:41 [musl] Question on C++ locale Dong Brett
  2020-11-30 11:31 ` Szabolcs Nagy
  2020-11-30 11:35 ` Szabolcs Nagy
@ 2020-11-30 14:51 ` Rich Felker
  2020-12-09 14:35   ` Érico Nogueira
  2020-12-01  3:53 ` [musl] " Dong Brett
  3 siblings, 1 reply; 17+ messages in thread
From: Rich Felker @ 2020-11-30 14:51 UTC (permalink / raw)
  To: Dong Brett; +Cc: musl

On Mon, Nov 30, 2020 at 06:41:33PM +0800, Dong Brett wrote:
> Hi all,
> 
> I am troubleshooting a locale related issue of our C++ software when building with musl. With some efforts I narrowed our problem down to the inability of setting a UTF-8 locale in C++ standard library.
> 
> The following C code prints UTF-8 characters correctly:
> #include <ncurses.h>
> #include <langinfo.h>
> #include <locale.h>
> 
> int main()
> {
>     setlocale(LC_ALL, "");
>     initscr();
>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
>     printw("CODESET: %s\n", nl_langinfo(CODESET));
>     printw("Hello, world!\n");
>     printw("你好,世界!\n");
>     refresh();
>     getch();
>     endwin();
>     return 0;
> }
> 
> Giving the output of
> LC_ALL: C.UTF-8;C;C;C;C;C
> CODESET: UTF-8
> Hello, world!
> 你好,世界!
> 
> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> #include <langinfo.h>
> #include <locale.h>
> #include <locale>
> using namespace std;
> int main()
> {
>     std::locale::global(locale(""));
>     initscr();
>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
>     printw("C++ locale: %s\n", locale().name().c_str());
>     printw("CODESET: %s\n", nl_langinfo(CODESET));
>     printw("Hello, world!\n");
>     printw("你好,世界!\n");
>     refresh();
>     getch();
>     endwin();
>     return 0;
> }
> 
> Giving a corrupted output:
> LC_ALL: C
> C++ locale: C
> CODESET: ASCII
> Hello, world!
> 你好?~L?~V?~U~L!
> 
> Seems only ASCII C locale is available in C++. If I run the above C++ code with LANG="C.UTF-8", an exception is thrown and the program is aborted:
> terminate called after throwing an instance of 'std::runtime_error'
>   what():  locale::facet::_S_create_c_locale name not valid
> Aborted
> 
> I also tried LANG="UTF-8”, LANG="en_US.UTF-8" but none of those
> works. Only LANG="C" could make the program run but then only ASCII
> characters are supported.
> 
> My question is that is there a way to make locale in C++ standard
> library work with musl? Or had I done anything wrong with it?

Thanks for raising this. Indeed you've uncovered a (pile of) bug(s) in
libstdc++, but they don't seem to be relevant to your usage with
ncurses. Being a C library, not a C++ one, curses behavior depends on
the locale as set through the C/POSIX mechanisms, setlocale and/or
newlocale/uselocale. You shouldn't be using C++'s locale framework for
this. Any program using ncurses should start with either
setlocale(LC_ALL,"") or setlocale(LC_CTYPE,"") (depending on whether
you want the behavior of the other categories).

I'll try to figure out what we need to do to get this fixed in
libstdc++. Since it's never been reported before, I suspect just very
few programs are using the C++ locale API so hopefully at least the
problem is low-impact.

Rich

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 14:39     ` Samuel Holland
  2020-11-30 14:48       ` Rich Felker
@ 2020-11-30 15:12       ` Érico Nogueira
  2020-11-30 15:35         ` Rich Felker
  1 sibling, 1 reply; 17+ messages in thread
From: Érico Nogueira @ 2020-11-30 15:12 UTC (permalink / raw)
  To: Samuel Holland, musl, Dong Brett

On Mon Nov 30, 2020 at 11:39 AM -03, Samuel Holland wrote:
> On 11/30/20 7:44 AM, Érico Nogueira wrote:
> > On Mon Nov 30, 2020 at 8:35 AM -03, Szabolcs Nagy wrote:
> >> * Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33
> >> +0800]:
> >>> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> >>> #include <langinfo.h>
> >>> #include <locale.h>
> >>> #include <locale>
> >>> using namespace std;
> >>> int main()
> >>> {
> >>>     std::locale::global(locale(""));
> >>>     initscr();
> >>>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> >>>     printw("C++ locale: %s\n", locale().name().c_str());
> >>>     printw("CODESET: %s\n", nl_langinfo(CODESET));
> >>>     printw("Hello, world!\n");
> >>>     printw("你好,世界!\n");
> >>>     refresh();
> >>>     getch();
> >>>     endwin();
> >>>     return 0;
> >>> }
> >>
> >> fwiw for me even the first line fails.
> >> i don't know how c++ locales are supposed to work.
> > 
> > From [1], it seems that C++ locales are supposed to affect the global
> > locale as well, so they should call setlocale() when appropriate.
> > 
> > - [1] https://www.cplusplus.com/reference/locale/locale/
> > 
> > Unfortunately, I assume libstdc++ uses their generic locale support on
> > musl...  From gcc-10.2.0/libstdc++-v3/config/locale/generic/c_locale.cc:
> > 
> >   void
> >   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> > 				    __c_locale)
> >   {
> >     // Currently, the generic model only supports the "C" locale.
> >     // See http://gcc.gnu.org/ml/libstdc++/2003-02/msg00345.html
> >     __cloc = 0;
> >     if (strcmp(__s, "C"))
> >       __throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> > 			    "name not valid"));
> >   }
> > 
>
> I don't know for sure that it's the right thing to do, but I have been
> patching
> out that error for the last several years[1] and so far I have not
> noticed any
> negative effects. Adelie, which is very thorough about testing, has also
> carried
> the patch for a while[2].
>
> Samuel
>
> [1]:
> https://github.com/smaeul/portage/blob/c744774a/patches/sys-devel/gcc/gcc-5.4.0-locale.patch
> [2]: https://code.foxkit.us/adelie/packages/-/commit/d09b437d

Are those patches correct in functionality? The GNU version is:

  void
  locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
				    __c_locale __old)
  {
    __cloc = __newlocale(1 << LC_ALL, __s, __old);
    if (!__cloc)
      {
	// This named locale is not supported by the underlying OS.
	__throw_runtime_error(__N("locale::facet::_S_create_c_locale "
				  "name not valid"));
      }
  }

It tries to create a locale object, which the generic code doesn't do.
In the generic case, _S_create_c_locale is basically a noop, and I'd
assume localization wouldn't work, even if it does avoid the runtime
abort.

I will try it out locally when I get the time.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 15:12       ` Érico Nogueira
@ 2020-11-30 15:35         ` Rich Felker
  2020-11-30 17:14           ` Érico Nogueira
  0 siblings, 1 reply; 17+ messages in thread
From: Rich Felker @ 2020-11-30 15:35 UTC (permalink / raw)
  To: Érico Nogueira; +Cc: Samuel Holland, musl, Dong Brett

On Mon, Nov 30, 2020 at 12:12:50PM -0300, Érico Nogueira wrote:
> On Mon Nov 30, 2020 at 11:39 AM -03, Samuel Holland wrote:
> > On 11/30/20 7:44 AM, Érico Nogueira wrote:
> > > On Mon Nov 30, 2020 at 8:35 AM -03, Szabolcs Nagy wrote:
> > >> * Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33
> > >> +0800]:
> > >>> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> > >>> #include <langinfo.h>
> > >>> #include <locale.h>
> > >>> #include <locale>
> > >>> using namespace std;
> > >>> int main()
> > >>> {
> > >>>     std::locale::global(locale(""));
> > >>>     initscr();
> > >>>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> > >>>     printw("C++ locale: %s\n", locale().name().c_str());
> > >>>     printw("CODESET: %s\n", nl_langinfo(CODESET));
> > >>>     printw("Hello, world!\n");
> > >>>     printw("你好,世界!\n");
> > >>>     refresh();
> > >>>     getch();
> > >>>     endwin();
> > >>>     return 0;
> > >>> }
> > >>
> > >> fwiw for me even the first line fails.
> > >> i don't know how c++ locales are supposed to work.
> > > 
> > > From [1], it seems that C++ locales are supposed to affect the global
> > > locale as well, so they should call setlocale() when appropriate.
> > > 
> > > - [1] https://www.cplusplus.com/reference/locale/locale/
> > > 
> > > Unfortunately, I assume libstdc++ uses their generic locale support on
> > > musl...  From gcc-10.2.0/libstdc++-v3/config/locale/generic/c_locale.cc:
> > > 
> > >   void
> > >   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> > > 				    __c_locale)
> > >   {
> > >     // Currently, the generic model only supports the "C" locale.
> > >     // See http://gcc.gnu.org/ml/libstdc++/2003-02/msg00345.html
> > >     __cloc = 0;
> > >     if (strcmp(__s, "C"))
> > >       __throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> > > 			    "name not valid"));
> > >   }
> > > 
> >
> > I don't know for sure that it's the right thing to do, but I have been
> > patching
> > out that error for the last several years[1] and so far I have not
> > noticed any
> > negative effects. Adelie, which is very thorough about testing, has also
> > carried
> > the patch for a while[2].
> >
> > Samuel
> >
> > [1]:
> > https://github.com/smaeul/portage/blob/c744774a/patches/sys-devel/gcc/gcc-5.4.0-locale.patch
> > [2]: https://code.foxkit.us/adelie/packages/-/commit/d09b437d
> 
> Are those patches correct in functionality? The GNU version is:
> 
>   void
>   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> 				    __c_locale __old)
>   {
>     __cloc = __newlocale(1 << LC_ALL, __s, __old);
>     if (!__cloc)
>       {
> 	// This named locale is not supported by the underlying OS.
> 	__throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> 				  "name not valid"));
>       }
>   }
> 
> It tries to create a locale object, which the generic code doesn't do.
> In the generic case, _S_create_c_locale is basically a noop, and I'd
> assume localization wouldn't work, even if it does avoid the runtime
> abort.
> 
> I will try it out locally when I get the time.

The code there in the GNU version is correct (the one without
newlocale isn't correct) aside from having the __ prefix, but other
parts of the GNU version are wrong in that they poke at glibc
internals to "optimize" useless byte-based ctype functions (useless
because they can't operate on the only characters whose properties
could vary by locale, the non-ASCII ones). There should probably be a
new "posix" directory here based on the GNU one but with all the
GNUisms removed. If it's not hard to backport that to older GCC
versions maybe we should do that.

One thing: I think in order for std::locale::global to be able to
work, the locale creation code also needs to store the name (string)
passed to locale() constructor, since there's no way to setlocale to a
locale_t. Instead you need to remember the name so you can setlocale()
to the same name. Perhaps NL_LOCALE_NAME would suffice, but I don't
think it can easily give the exact same behavior since it's
per-category.

Rich

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 15:35         ` Rich Felker
@ 2020-11-30 17:14           ` Érico Nogueira
  2020-11-30 18:11             ` Rich Felker
  0 siblings, 1 reply; 17+ messages in thread
From: Érico Nogueira @ 2020-11-30 17:14 UTC (permalink / raw)
  To: musl; +Cc: Samuel Holland, Dong Brett

On Mon Nov 30, 2020 at 12:35 PM -03, Rich Felker wrote:
> On Mon, Nov 30, 2020 at 12:12:50PM -0300, Érico Nogueira wrote:
> > On Mon Nov 30, 2020 at 11:39 AM -03, Samuel Holland wrote:
> > > On 11/30/20 7:44 AM, Érico Nogueira wrote:
> > > > On Mon Nov 30, 2020 at 8:35 AM -03, Szabolcs Nagy wrote:
> > > >> * Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33
> > > >> +0800]:
> > > >>> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> > > >>> #include <langinfo.h>
> > > >>> #include <locale.h>
> > > >>> #include <locale>
> > > >>> using namespace std;
> > > >>> int main()
> > > >>> {
> > > >>>     std::locale::global(locale(""));
> > > >>>     initscr();
> > > >>>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> > > >>>     printw("C++ locale: %s\n", locale().name().c_str());
> > > >>>     printw("CODESET: %s\n", nl_langinfo(CODESET));
> > > >>>     printw("Hello, world!\n");
> > > >>>     printw("你好,世界!\n");
> > > >>>     refresh();
> > > >>>     getch();
> > > >>>     endwin();
> > > >>>     return 0;
> > > >>> }
> > > >>
> > > >> fwiw for me even the first line fails.
> > > >> i don't know how c++ locales are supposed to work.
> > > > 
> > > > From [1], it seems that C++ locales are supposed to affect the global
> > > > locale as well, so they should call setlocale() when appropriate.
> > > > 
> > > > - [1] https://www.cplusplus.com/reference/locale/locale/
> > > > 
> > > > Unfortunately, I assume libstdc++ uses their generic locale support on
> > > > musl...  From gcc-10.2.0/libstdc++-v3/config/locale/generic/c_locale.cc:
> > > > 
> > > >   void
> > > >   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> > > > 				    __c_locale)
> > > >   {
> > > >     // Currently, the generic model only supports the "C" locale.
> > > >     // See http://gcc.gnu.org/ml/libstdc++/2003-02/msg00345.html
> > > >     __cloc = 0;
> > > >     if (strcmp(__s, "C"))
> > > >       __throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> > > > 			    "name not valid"));
> > > >   }
> > > > 
> > >
> > > I don't know for sure that it's the right thing to do, but I have been
> > > patching
> > > out that error for the last several years[1] and so far I have not
> > > noticed any
> > > negative effects. Adelie, which is very thorough about testing, has also
> > > carried
> > > the patch for a while[2].
> > >
> > > Samuel
> > >
> > > [1]:
> > > https://github.com/smaeul/portage/blob/c744774a/patches/sys-devel/gcc/gcc-5.4.0-locale.patch
> > > [2]: https://code.foxkit.us/adelie/packages/-/commit/d09b437d
> > 
> > Are those patches correct in functionality? The GNU version is:
> > 
> >   void
> >   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> > 				    __c_locale __old)
> >   {
> >     __cloc = __newlocale(1 << LC_ALL, __s, __old);
> >     if (!__cloc)
> >       {
> > 	// This named locale is not supported by the underlying OS.
> > 	__throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> > 				  "name not valid"));
> >       }
> >   }
> > 
> > It tries to create a locale object, which the generic code doesn't do.
> > In the generic case, _S_create_c_locale is basically a noop, and I'd
> > assume localization wouldn't work, even if it does avoid the runtime
> > abort.
> > 
> > I will try it out locally when I get the time.
>
> The code there in the GNU version is correct (the one without
> newlocale isn't correct) aside from having the __ prefix, but other
> parts of the GNU version are wrong in that they poke at glibc
> internals to "optimize" useless byte-based ctype functions (useless
> because they can't operate on the only characters whose properties
> could vary by locale, the non-ASCII ones). There should probably be a
> new "posix" directory here based on the GNU one but with all the
> GNUisms removed. If it's not hard to backport that to older GCC
> versions maybe we should do that.

C++ is a bit mysterious to me; do you think there's a chance that
changing the libstdc++ locale implementation could break programs
built for the old version?

I also wonder what the configure script should look for in order to
choose which version to use.

From a really quick look at _S_create_c_locale, the dragonfly version
might be usable for this purpose, although it uses some non-standard
headers.

>
> One thing: I think in order for std::locale::global to be able to
> work, the locale creation code also needs to store the name (string)
> passed to locale() constructor, since there's no way to setlocale to a
> locale_t. Instead you need to remember the name so you can setlocale()
> to the same name. Perhaps NL_LOCALE_NAME would suffice, but I don't
> think it can easily give the exact same behavior since it's
> per-category.
>
> Rich


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 17:14           ` Érico Nogueira
@ 2020-11-30 18:11             ` Rich Felker
  0 siblings, 0 replies; 17+ messages in thread
From: Rich Felker @ 2020-11-30 18:11 UTC (permalink / raw)
  To: Érico Nogueira; +Cc: musl, Samuel Holland, Dong Brett

On Mon, Nov 30, 2020 at 02:14:15PM -0300, Érico Nogueira wrote:
> On Mon Nov 30, 2020 at 12:35 PM -03, Rich Felker wrote:
> > On Mon, Nov 30, 2020 at 12:12:50PM -0300, Érico Nogueira wrote:
> > > On Mon Nov 30, 2020 at 11:39 AM -03, Samuel Holland wrote:
> > > > On 11/30/20 7:44 AM, Érico Nogueira wrote:
> > > > > On Mon Nov 30, 2020 at 8:35 AM -03, Szabolcs Nagy wrote:
> > > > >> * Dong Brett <brett.browning.dong@gmail.com> [2020-11-30 18:41:33
> > > > >> +0800]:
> > > > >>> However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> > > > >>> #include <langinfo.h>
> > > > >>> #include <locale.h>
> > > > >>> #include <locale>
> > > > >>> using namespace std;
> > > > >>> int main()
> > > > >>> {
> > > > >>>     std::locale::global(locale(""));
> > > > >>>     initscr();
> > > > >>>     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> > > > >>>     printw("C++ locale: %s\n", locale().name().c_str());
> > > > >>>     printw("CODESET: %s\n", nl_langinfo(CODESET));
> > > > >>>     printw("Hello, world!\n");
> > > > >>>     printw("你好,世界!\n");
> > > > >>>     refresh();
> > > > >>>     getch();
> > > > >>>     endwin();
> > > > >>>     return 0;
> > > > >>> }
> > > > >>
> > > > >> fwiw for me even the first line fails.
> > > > >> i don't know how c++ locales are supposed to work.
> > > > > 
> > > > > From [1], it seems that C++ locales are supposed to affect the global
> > > > > locale as well, so they should call setlocale() when appropriate.
> > > > > 
> > > > > - [1] https://www.cplusplus.com/reference/locale/locale/
> > > > > 
> > > > > Unfortunately, I assume libstdc++ uses their generic locale support on
> > > > > musl...  From gcc-10.2.0/libstdc++-v3/config/locale/generic/c_locale.cc:
> > > > > 
> > > > >   void
> > > > >   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> > > > > 				    __c_locale)
> > > > >   {
> > > > >     // Currently, the generic model only supports the "C" locale.
> > > > >     // See http://gcc.gnu.org/ml/libstdc++/2003-02/msg00345.html
> > > > >     __cloc = 0;
> > > > >     if (strcmp(__s, "C"))
> > > > >       __throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> > > > > 			    "name not valid"));
> > > > >   }
> > > > > 
> > > >
> > > > I don't know for sure that it's the right thing to do, but I have been
> > > > patching
> > > > out that error for the last several years[1] and so far I have not
> > > > noticed any
> > > > negative effects. Adelie, which is very thorough about testing, has also
> > > > carried
> > > > the patch for a while[2].
> > > >
> > > > Samuel
> > > >
> > > > [1]:
> > > > https://github.com/smaeul/portage/blob/c744774a/patches/sys-devel/gcc/gcc-5.4.0-locale.patch
> > > > [2]: https://code.foxkit.us/adelie/packages/-/commit/d09b437d
> > > 
> > > Are those patches correct in functionality? The GNU version is:
> > > 
> > >   void
> > >   locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
> > > 				    __c_locale __old)
> > >   {
> > >     __cloc = __newlocale(1 << LC_ALL, __s, __old);
> > >     if (!__cloc)
> > >       {
> > > 	// This named locale is not supported by the underlying OS.
> > > 	__throw_runtime_error(__N("locale::facet::_S_create_c_locale "
> > > 				  "name not valid"));
> > >       }
> > >   }
> > > 
> > > It tries to create a locale object, which the generic code doesn't do.
> > > In the generic case, _S_create_c_locale is basically a noop, and I'd
> > > assume localization wouldn't work, even if it does avoid the runtime
> > > abort.
> > > 
> > > I will try it out locally when I get the time.
> >
> > The code there in the GNU version is correct (the one without
> > newlocale isn't correct) aside from having the __ prefix, but other
> > parts of the GNU version are wrong in that they poke at glibc
> > internals to "optimize" useless byte-based ctype functions (useless
> > because they can't operate on the only characters whose properties
> > could vary by locale, the non-ASCII ones). There should probably be a
> > new "posix" directory here based on the GNU one but with all the
> > GNUisms removed. If it's not hard to backport that to older GCC
> > versions maybe we should do that.
> 
> C++ is a bit mysterious to me; do you think there's a chance that
> changing the libstdc++ locale implementation could break programs
> built for the old version?

Hopefully not, but indeed they do awful stuff inlining code. I'm not
sure if any of it gets inlined into applications or only other parts
of libstdc++.

> I also wonder what the configure script should look for in order to
> choose which version to use.

Ideally it should just always use the POSIX one, but it's tagged to
the target tuple anyway.

> From a really quick look at _S_create_c_locale, the dragonfly version
> might be usable for this purpose, although it uses some non-standard
> headers.

Just xlocale.h? It also uses strto*_l etc. where it should just
uselocale(). But indeed it looks like it might be a workable baseline
for fixing this..

Rich

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [musl] Re: Question on C++ locale
  2020-11-30 10:41 [musl] Question on C++ locale Dong Brett
                   ` (2 preceding siblings ...)
  2020-11-30 14:51 ` Rich Felker
@ 2020-12-01  3:53 ` Dong Brett
  2020-12-01 16:21   ` Rich Felker
  3 siblings, 1 reply; 17+ messages in thread
From: Dong Brett @ 2020-12-01  3:53 UTC (permalink / raw)
  To: musl

Hi all,

Thank you all for your replies! I tried Samuel’s patch and it works with our application perfectly!

Regards,
Brett

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Re: Question on C++ locale
  2020-12-01  3:53 ` [musl] " Dong Brett
@ 2020-12-01 16:21   ` Rich Felker
  0 siblings, 0 replies; 17+ messages in thread
From: Rich Felker @ 2020-12-01 16:21 UTC (permalink / raw)
  To: Dong Brett; +Cc: musl

On Tue, Dec 01, 2020 at 11:53:21AM +0800, Dong Brett wrote:
> Hi all,
> 
> Thank you all for your replies! I tried Samuel’s patch and it works
> with our application perfectly!

FYI the patch does not make the C++ locale API work, but it does let
you use std::locale::global(locale("")); as a C++-flavored substitute
for setlocale(LC_ALL,""); Interfaces which takes explicit locale
objects will not work still.

Rich


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-11-30 14:51 ` Rich Felker
@ 2020-12-09 14:35   ` Érico Nogueira
  2020-12-09 16:41     ` Rich Felker
  0 siblings, 1 reply; 17+ messages in thread
From: Érico Nogueira @ 2020-12-09 14:35 UTC (permalink / raw)
  To: musl, Dong Brett

On Mon Nov 30, 2020 at 11:51 AM -03, Rich Felker wrote:
> On Mon, Nov 30, 2020 at 06:41:33PM +0800, Dong Brett wrote:
> > Hi all,
> > 
> > I am troubleshooting a locale related issue of our C++ software when building with musl. With some efforts I narrowed our problem down to the inability of setting a UTF-8 locale in C++ standard library.
> > 
> > The following C code prints UTF-8 characters correctly:
> > #include <ncurses.h>
> > #include <langinfo.h>
> > #include <locale.h>
> > 
> > int main()
> > {
> >     setlocale(LC_ALL, "");
> >     initscr();
> >     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> >     printw("CODESET: %s\n", nl_langinfo(CODESET));
> >     printw("Hello, world!\n");
> >     printw("你好,世界!\n");
> >     refresh();
> >     getch();
> >     endwin();
> >     return 0;
> > }
> > 
> > Giving the output of
> > LC_ALL: C.UTF-8;C;C;C;C;C
> > CODESET: UTF-8
> > Hello, world!
> > 你好,世界!
> > 
> > However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> > #include <langinfo.h>
> > #include <locale.h>
> > #include <locale>
> > using namespace std;
> > int main()
> > {
> >     std::locale::global(locale(""));
> >     initscr();
> >     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> >     printw("C++ locale: %s\n", locale().name().c_str());
> >     printw("CODESET: %s\n", nl_langinfo(CODESET));
> >     printw("Hello, world!\n");
> >     printw("你好,世界!\n");
> >     refresh();
> >     getch();
> >     endwin();
> >     return 0;
> > }
> > 
> > Giving a corrupted output:
> > LC_ALL: C
> > C++ locale: C
> > CODESET: ASCII
> > Hello, world!
> > 你好?~L?~V?~U~L!
> > 
> > Seems only ASCII C locale is available in C++. If I run the above C++ code with LANG="C.UTF-8", an exception is thrown and the program is aborted:
> > terminate called after throwing an instance of 'std::runtime_error'
> >   what():  locale::facet::_S_create_c_locale name not valid
> > Aborted
> > 
> > I also tried LANG="UTF-8”, LANG="en_US.UTF-8" but none of those
> > works. Only LANG="C" could make the program run but then only ASCII
> > characters are supported.
> > 
> > My question is that is there a way to make locale in C++ standard
> > library work with musl? Or had I done anything wrong with it?
>
> Thanks for raising this. Indeed you've uncovered a (pile of) bug(s) in
> libstdc++, but they don't seem to be relevant to your usage with
> ncurses. Being a C library, not a C++ one, curses behavior depends on
> the locale as set through the C/POSIX mechanisms, setlocale and/or
> newlocale/uselocale. You shouldn't be using C++'s locale framework for
> this. Any program using ncurses should start with either
> setlocale(LC_ALL,"") or setlocale(LC_CTYPE,"") (depending on whether
> you want the behavior of the other categories).
>
> I'll try to figure out what we need to do to get this fixed in
> libstdc++. Since it's never been reported before, I suspect just very
> few programs are using the C++ locale API so hopefully at least the
> problem is low-impact.

As another data point for an application that uses C++ locales, there is
snapper. From [1]:

    try
    {
	locale::global(locale(""));
    }
    catch (const runtime_error& e)
    {
	cerr << _("Failed to set locale. Fix your system.") << endl;
    }

Fortunately, they have a try-catch around the call, which will also
catch other errors like bad LANG values, if I understand correctly.  I
wonder if other applications that make use of the API usually have this
block, which can mask the error for the user.

That said, I don't think the project can be built on musl without any
external patches yet (some pieces relied heavily on glibc extensions),
so having locale issues isn't the biggest problem with snapper on musl.

- [1] https://github.com/openSUSE/snapper/blob/9e795ed4f0d87e6afcd5065f26c1350942f8ab38/client/snapper.cc#L126

>
> Rich

Érico

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [musl] Question on C++ locale
  2020-12-09 14:35   ` Érico Nogueira
@ 2020-12-09 16:41     ` Rich Felker
  0 siblings, 0 replies; 17+ messages in thread
From: Rich Felker @ 2020-12-09 16:41 UTC (permalink / raw)
  To: Érico Nogueira; +Cc: musl, Dong Brett

On Wed, Dec 09, 2020 at 11:35:57AM -0300, Érico Nogueira wrote:
> On Mon Nov 30, 2020 at 11:51 AM -03, Rich Felker wrote:
> > On Mon, Nov 30, 2020 at 06:41:33PM +0800, Dong Brett wrote:
> > > Hi all,
> > > 
> > > I am troubleshooting a locale related issue of our C++ software when building with musl. With some efforts I narrowed our problem down to the inability of setting a UTF-8 locale in C++ standard library.
> > > 
> > > The following C code prints UTF-8 characters correctly:
> > > #include <ncurses.h>
> > > #include <langinfo.h>
> > > #include <locale.h>
> > > 
> > > int main()
> > > {
> > >     setlocale(LC_ALL, "");
> > >     initscr();
> > >     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> > >     printw("CODESET: %s\n", nl_langinfo(CODESET));
> > >     printw("Hello, world!\n");
> > >     printw("你好,世界!\n");
> > >     refresh();
> > >     getch();
> > >     endwin();
> > >     return 0;
> > > }
> > > 
> > > Giving the output of
> > > LC_ALL: C.UTF-8;C;C;C;C;C
> > > CODESET: UTF-8
> > > Hello, world!
> > > 你好,世界!
> > > 
> > > However, the following C++ code does not work (our software uses std::locale in C++ standard library for locale related stuff):
> > > #include <langinfo.h>
> > > #include <locale.h>
> > > #include <locale>
> > > using namespace std;
> > > int main()
> > > {
> > >     std::locale::global(locale(""));
> > >     initscr();
> > >     printw("LC_ALL: %s\n", setlocale(LC_ALL, NULL));
> > >     printw("C++ locale: %s\n", locale().name().c_str());
> > >     printw("CODESET: %s\n", nl_langinfo(CODESET));
> > >     printw("Hello, world!\n");
> > >     printw("你好,世界!\n");
> > >     refresh();
> > >     getch();
> > >     endwin();
> > >     return 0;
> > > }
> > > 
> > > Giving a corrupted output:
> > > LC_ALL: C
> > > C++ locale: C
> > > CODESET: ASCII
> > > Hello, world!
> > > 你好?~L?~V?~U~L!
> > > 
> > > Seems only ASCII C locale is available in C++. If I run the above C++ code with LANG="C.UTF-8", an exception is thrown and the program is aborted:
> > > terminate called after throwing an instance of 'std::runtime_error'
> > >   what():  locale::facet::_S_create_c_locale name not valid
> > > Aborted
> > > 
> > > I also tried LANG="UTF-8”, LANG="en_US.UTF-8" but none of those
> > > works. Only LANG="C" could make the program run but then only ASCII
> > > characters are supported.
> > > 
> > > My question is that is there a way to make locale in C++ standard
> > > library work with musl? Or had I done anything wrong with it?
> >
> > Thanks for raising this. Indeed you've uncovered a (pile of) bug(s) in
> > libstdc++, but they don't seem to be relevant to your usage with
> > ncurses. Being a C library, not a C++ one, curses behavior depends on
> > the locale as set through the C/POSIX mechanisms, setlocale and/or
> > newlocale/uselocale. You shouldn't be using C++'s locale framework for
> > this. Any program using ncurses should start with either
> > setlocale(LC_ALL,"") or setlocale(LC_CTYPE,"") (depending on whether
> > you want the behavior of the other categories).
> >
> > I'll try to figure out what we need to do to get this fixed in
> > libstdc++. Since it's never been reported before, I suspect just very
> > few programs are using the C++ locale API so hopefully at least the
> > problem is low-impact.
> 
> As another data point for an application that uses C++ locales, there is
> snapper. From [1]:
> 
>     try
>     {
> 	locale::global(locale(""));
>     }
>     catch (const runtime_error& e)
>     {
> 	cerr << _("Failed to set locale. Fix your system.") << endl;
>     }
> 
> Fortunately, they have a try-catch around the call, which will also
> catch other errors like bad LANG values, if I understand correctly.  I
> wonder if other applications that make use of the API usually have this
> block, which can mask the error for the user.

On musl there are no bad LANG values. setlocale to "" (and likewise
newlocale for LC_ALL/"" or LC_CTYPE/"") can never fail. But this could
matter on other systems.

Rich

^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2020-12-09 16:41 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-30 10:41 [musl] Question on C++ locale Dong Brett
2020-11-30 11:31 ` Szabolcs Nagy
2020-11-30 11:58   ` Dong Brett
2020-11-30 11:35 ` Szabolcs Nagy
2020-11-30 12:37   ` Érico Nogueira
2020-11-30 13:44   ` Érico Nogueira
2020-11-30 14:39     ` Samuel Holland
2020-11-30 14:48       ` Rich Felker
2020-11-30 15:12       ` Érico Nogueira
2020-11-30 15:35         ` Rich Felker
2020-11-30 17:14           ` Érico Nogueira
2020-11-30 18:11             ` Rich Felker
2020-11-30 14:51 ` Rich Felker
2020-12-09 14:35   ` Érico Nogueira
2020-12-09 16:41     ` Rich Felker
2020-12-01  3:53 ` [musl] " Dong Brett
2020-12-01 16:21   ` Rich Felker

mailing list of musl libc

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.vuxu.org/musl

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 musl musl/ http://inbox.vuxu.org/musl \
		musl@inbox.vuxu.org
	public-inbox-index musl

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.archive.musl


code repositories for the project(s) associated with this inbox:

	https://git.vuxu.org/mirror/musl/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git