Re: libintl: stubs or working functions

Re: libintl: stubs or working functions

[Felix Janda]

On Thu, Mar 06, 2015 at 22:24:15PM GMT, Rich Felker wrote:
> On Thu, Mar 05, 2015 at 04:36:49PM +0700, Рысь wrote:
[snip]
> > * Did I understand that right that I do not need GNU gettext anymore and
> >   I can use musl's interface for that?
>
> Yes, modulo some GNU software (coreutils for example) that probes for
> glibc/gnu-libintl internals at configure time and depends on
> poorly-designed and undocumented features (SYSDEP strings). These
> programs will not work without either GNU libintl or patching out the
> bad parts of configure and using a version of msgfmt that works around
> the need for SYSDEP strings. I believe the one from sabotage
> gettext-tiny does.

I would like to see what it takes to fix the autoconf tests. The problem
is the macro AM_GNU_GETTEXT with the check

http://git.savannah.gnu.org/cgit/gettext.git/tree/gettext-runtime/m4/gettext.m4#n159

(It looks for the internal symbols _nl_msg_cat_cntr and _nl_domain_bindings
instead of relying on __GNU_GETTEXT_SUPPORTED_REVISION().)
debian code search suggests that quite a lot of projects use this macro.


https://lists.gnu.org/archive/html/bug-gnu-utils/2006-03/msg00011.html

gives some reasoning for the unportable tests:

* _GNU_GETTEXT_SUPPORTED_REVISION() was only introduced in gettext 0.10.xx
* _GNU_GETTEXT_SUPPORTED_REVISION() of glibc says that it does not support
  major revision 1 although it does

I would like to ask the gettext developers for an additional test
"for GNU gettext in libc", which fails if __GLIBC__ uses
_GNU_GETTEXT_SUPPORTED_REVISION and can only improve the previous test result.

Any comments on this or alternative approaches?

Thanks,
Felix

Re: Re: libintl: stubs or working functions

[Rich Felker]

On Wed, Apr 15, 2015 at 09:18:16PM +0200, Felix Janda wrote:
> On Thu, Mar 06, 2015 at 22:24:15PM GMT, Rich Felker wrote:
> > On Thu, Mar 05, 2015 at 04:36:49PM +0700, Рысь wrote:
> [snip]
> > > * Did I understand that right that I do not need GNU gettext anymore and
> > >   I can use musl's interface for that?
> >
> > Yes, modulo some GNU software (coreutils for example) that probes for
> > glibc/gnu-libintl internals at configure time and depends on
> > poorly-designed and undocumented features (SYSDEP strings). These
> > programs will not work without either GNU libintl or patching out the
> > bad parts of configure and using a version of msgfmt that works around
> > the need for SYSDEP strings. I believe the one from sabotage
> > gettext-tiny does.
> 
> I would like to see what it takes to fix the autoconf tests. The problem
> is the macro AM_GNU_GETTEXT with the check
> 
> http://git.savannah.gnu.org/cgit/gettext.git/tree/gettext-runtime/m4/gettext.m4#n159

Sadly m4 gives me a headache, so I haven't yet figured out what the
above code is doing.

> (It looks for the internal symbols _nl_msg_cat_cntr and _nl_domain_bindings
> instead of relying on __GNU_GETTEXT_SUPPORTED_REVISION().)
> debian code search suggests that quite a lot of projects use this macro.
> 
> 
> https://lists.gnu.org/archive/html/bug-gnu-utils/2006-03/msg00011.html
> 
> gives some reasoning for the unportable tests:
> 
> * _GNU_GETTEXT_SUPPORTED_REVISION() was only introduced in gettext 0.10.xx
> * _GNU_GETTEXT_SUPPORTED_REVISION() of glibc says that it does not support
>   major revision 1 although it does
> 
> I would like to ask the gettext developers for an additional test
> "for GNU gettext in libc", which fails if __GLIBC__ uses
> _GNU_GETTEXT_SUPPORTED_REVISION and can only improve the previous test result.
> 
> Any comments on this or alternative approaches?

If the program actually needs a specific version of GNU gettext, then
it almost certainly does not (fully) work with musl's without also
using a fixed msgfmt utility.. The offending functionality is GNU
gettext's "SYSDEP strings" which conflict with the whole design of
gettext to use immutable memory-mapped strings in-place.

The problem they're trying to solve is that some translatable strings
contain formats from <inttypes.h> like PRIx64, etc. whose definitions
may vary from one system to another.

The recent GNU/glibc gettext these projects are trying to depend on
has a hideous hack to process variable substitutions in strings at
message catalog load time and produce per-process copies of the
patched-up strings. This is a waste of code size, runtime, and memory,
and it's quite simply an offense against any rational sensibilities.

For programs using this feature, the version of the msgfmt utility
from Sabotage Linux simply performs the mappings (all possible ones,
so the resulting .mo file works on any system) at po->mo compile time,
and then standard gettext behavior in musl and other non-GNU
implementations can handle the application's needs.

I'm not sure what's the right way to get rid of the assumption on the
app configure side that it needs GNU gettext, though.

Rich

Re: Re: libintl: stubs or working functions

[stephen Turner]

[-- Attachment #1: Type: text/plain, Size: 196 bytes --]

> Sadly m4 gives me a headache, so I haven't yet figured out what the
> above code is doing.
>

Have you tried the converted bsd m4? It plays nice with pcc and musl.
http://haddonthethird.net/m4/

[-- Attachment #2: Type: text/html, Size: 306 bytes --]

Re: Re: libintl: stubs or working functions

[Felix Janda]

Rich Felker wrote:
> On Wed, Apr 15, 2015 at 09:18:16PM +0200, Felix Janda wrote:
> > On Thu, Mar 06, 2015 at 22:24:15PM GMT, Rich Felker wrote:
> > > On Thu, Mar 05, 2015 at 04:36:49PM +0700, Рысь wrote:
> > [snip]
> > > > * Did I understand that right that I do not need GNU gettext anymore and
> > > >   I can use musl's interface for that?
> > >
> > > Yes, modulo some GNU software (coreutils for example) that probes for
> > > glibc/gnu-libintl internals at configure time and depends on
> > > poorly-designed and undocumented features (SYSDEP strings). These
> > > programs will not work without either GNU libintl or patching out the
> > > bad parts of configure and using a version of msgfmt that works around
> > > the need for SYSDEP strings. I believe the one from sabotage
> > > gettext-tiny does.
> > 
> > I would like to see what it takes to fix the autoconf tests. The problem
> > is the macro AM_GNU_GETTEXT with the check
> > 
> > http://git.savannah.gnu.org/cgit/gettext.git/tree/gettext-runtime/m4/gettext.m4#n159
> 
> Sadly m4 gives me a headache, so I haven't yet figured out what the
> above code is doing.
> 
> > (It looks for the internal symbols _nl_msg_cat_cntr and _nl_domain_bindings
> > instead of relying on __GNU_GETTEXT_SUPPORTED_REVISION().)
> > debian code search suggests that quite a lot of projects use this macro.
> > 
> > 
> > https://lists.gnu.org/archive/html/bug-gnu-utils/2006-03/msg00011.html
> > 
> > gives some reasoning for the unportable tests:
> > 
> > * _GNU_GETTEXT_SUPPORTED_REVISION() was only introduced in gettext 0.10.xx
> > * _GNU_GETTEXT_SUPPORTED_REVISION() of glibc says that it does not support
> >   major revision 1 although it does
> > 
> > I would like to ask the gettext developers for an additional test
> > "for GNU gettext in libc", which fails if __GLIBC__ uses
> > _GNU_GETTEXT_SUPPORTED_REVISION and can only improve the previous test result.
> > 
> > Any comments on this or alternative approaches?
> 
> If the program actually needs a specific version of GNU gettext, then
> it almost certainly does not (fully) work with musl's without also
> using a fixed msgfmt utility.. The offending functionality is GNU
> gettext's "SYSDEP strings" which conflict with the whole design of
> gettext to use immutable memory-mapped strings in-place.
> 
> The problem they're trying to solve is that some translatable strings
> contain formats from <inttypes.h> like PRIx64, etc. whose definitions
> may vary from one system to another.
> 
> The recent GNU/glibc gettext these projects are trying to depend on
> has a hideous hack to process variable substitutions in strings at
> message catalog load time and produce per-process copies of the
> patched-up strings. This is a waste of code size, runtime, and memory,
> and it's quite simply an offense against any rational sensibilities.
> 
> For programs using this feature, the version of the msgfmt utility
> from Sabotage Linux simply performs the mappings (all possible ones,
> so the resulting .mo file works on any system) at po->mo compile time,
> and then standard gettext behavior in musl and other non-GNU
> implementations can handle the application's needs.
> 
> I'm not sure what's the right way to get rid of the assumption on the
> app configure side that it needs GNU gettext, though.

As I understand applications need to pass 'need-formatstring-macros' to
the AM_GNU_GETTEXT macro to request this functionality. But with debian
code search I couldn't find any program doing that...

The macro distinguishes three gettext apis. gt_api_version is 1 for the
basic version. For the ngettext functions gt_api_version>=2 is necessary
and for these "SYSDEP strings" gt_api_version>=3 is necessary.

As I understand musl has gt_api_version=2. Is that right?

To test for the api version AM_GNU_GETTEXT checks in all cases for
bindtextdomain, gettext, _nl_msg_cat_cntr and _nl_domain_bindings. For
gt_api_version>=2 it checks for ngettext and for gt_api_version>=3 it
does something with __GNU_GETTEXT_SUPPORTED_REVISION we won't need to
care about.

So for musl the test gives obviously the wrong result.

I guess that we don't want to export symbols _nl_*.

What I would now like to ask upstream is to put the _nl_* stuff behind
an #ifdef __GLIBC__ ...

Felix

Re: Re: libintl: stubs or working functions

[Rich Felker]

On Thu, Apr 16, 2015 at 07:15:51PM +0200, Felix Janda wrote:
> As I understand applications need to pass 'need-formatstring-macros' to
> the AM_GNU_GETTEXT macro to request this functionality. But with debian
> code search I couldn't find any program doing that...

OK. AFAIK it's only GNU software like coreutils using this
functionality.

> The macro distinguishes three gettext apis. gt_api_version is 1 for the
> basic version. For the ngettext functions gt_api_version>=2 is necessary
> and for these "SYSDEP strings" gt_api_version>=3 is necessary.
> 
> As I understand musl has gt_api_version=2. Is that right?

Based on your understanding of the versions, yes, that sounds right.

> To test for the api version AM_GNU_GETTEXT checks in all cases for
> bindtextdomain, gettext, _nl_msg_cat_cntr and _nl_domain_bindings. For
> gt_api_version>=2 it checks for ngettext and for gt_api_version>=3 it
> does something with __GNU_GETTEXT_SUPPORTED_REVISION we won't need to
> care about.
> 
> So for musl the test gives obviously the wrong result.
> 
> I guess that we don't want to export symbols _nl_*.
> 
> What I would now like to ask upstream is to put the _nl_* stuff behind
> an #ifdef __GLIBC__ ...

I think we should ask why they're doing it. It sounds to me like they
should just be checking for the APIs they want to use.

Rich

Re: Re: libintl: stubs or working functions

[Felix Janda]

On Thu, Apr 16, 2015, Rich Felker wrote:
> On Thu, Apr 16, 2015 at 07:15:51PM +0200, Felix Janda wrote:
> > As I understand applications need to pass 'need-formatstring-macros' to
> > the AM_GNU_GETTEXT macro to request this functionality. But with debian
> > code search I couldn't find any program doing that...
> 
> OK. AFAIK it's only GNU software like coreutils using this
> functionality.
> 
> > The macro distinguishes three gettext apis. gt_api_version is 1 for the
> > basic version. For the ngettext functions gt_api_version>=2 is necessary
> > and for these "SYSDEP strings" gt_api_version>=3 is necessary.
> > 
> > As I understand musl has gt_api_version=2. Is that right?
> 
> Based on your understanding of the versions, yes, that sounds right.
> 
> > To test for the api version AM_GNU_GETTEXT checks in all cases for
> > bindtextdomain, gettext, _nl_msg_cat_cntr and _nl_domain_bindings. For
> > gt_api_version>=2 it checks for ngettext and for gt_api_version>=3 it
> > does something with __GNU_GETTEXT_SUPPORTED_REVISION we won't need to
> > care about.
> > 
> > So for musl the test gives obviously the wrong result.
> > 
> > I guess that we don't want to export symbols _nl_*.
> > 
> > What I would now like to ask upstream is to put the _nl_* stuff behind
> > an #ifdef __GLIBC__ ...
> 
> I think we should ask why they're doing it. It sounds to me like they
> should just be checking for the APIs they want to use.

Thanks for your advice.

I've sent something [1] to their mailing list a while ago but got no
reply so far.

It might be difficult to find a cross-compile friendly test for the API.

Felix

[1]: http://lists.gnu.org/archive/html/bug-gettext/2015-04/msg00002.html