Progress since 1.1.9

Progress since 1.1.9

[Rich Felker]

Fixed regressions from 1.1.9:
- Inability to build as thumb2 for ARM
- i386 thread-pointer-init breakage under qmeu-user
- Broken copy relocations on PowerPC 
- Problems with crt_arch.h on some archs/toolchains
- PIC violations in crt_arch.h based Scrt1.o with vis.h in use

Other fixes:
- Null deref in gettext
- Broken stack alignment in mips Scrt1.o and dynamic linker

Locale work:
- Removing TLS avoidance hacks
- Removing unecessary use of low-level atomics
- Static fail-safe C locales for newlocale to return
- Saving locale name for all categories

New features/enhancements:
- Production of rcrt1.o start file for static PIE
- "CHAR" and "" support in iconv_open
- SH4a inline atomics

Out-of-tree work:
- Refactoring atomics for archs that provide only LLSC or CAS
- New internal-lock/normal-type-mutex design

The one roadmap item I don't have any progress on is what to do with
the libgcc_s symbol version mess, and I think we should probably just
hold off until the next release cycle for that now.

Please test and report any new regressions -- see the above list for
areas that might have regressions. I'll try to keep making a few more
improvements over the next few days and try to get another release
packaged soon!

Rich

Re: Progress since 1.1.9

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2015-05-27 19:13:08 -0400]:
> The one roadmap item I don't have any progress on is what to do with
> the libgcc_s symbol version mess, and I think we should probably just
> hold off until the next release cycle for that now.

in case anyone wonders about the issue:

libgcc_s has a symbolic reference to an internal deprecated
symbol with no default version on x86 (in gcc trunk).
(so no @version only @@version which makes it invisible to
musl and ld).

the deprecated function is a constructor in libgcc_s that
initializes an unused struct in libgcc_s.

the only reason the symbol got versioned this way is because
they want old binaries to work and remove the symbol from
libgcc_s for new binaries, while still using the same symbol
name in new binaries but with different abi behaviour: now
it's only available in libgcc.a to fix their ifunc hack for
multi-versioning.

since it's a symbol internal to libgcc and the semantics
of the symbol changed they could have just used a new name
and left the old one alone (so old stuff is guaranteed to not
break including musl and new binaries use the new symbol with
new semantics).

so i still think my proposed libgcc patch makes more sense
than versioning:
https://gcc.gnu.org/ml/gcc-patches/2015-05/msg00899.html

gcc is unlikely to fix this, but this is a nonsense usecase
(and there is no other known use of versioning that is broken
with musl.. in theory other libs may deprecate symbols in a
similar way while still keeping symbolic references to them,
but such use was not yet observed).

in musl-gcc it can be worked around by preloading a noop
__cpu_indicator_init and in a musl based gcc it can be
patched out.

Re: Progress since 1.1.9

[Rich Felker]

On Thu, May 28, 2015 at 01:45:15PM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2015-05-27 19:13:08 -0400]:
> > The one roadmap item I don't have any progress on is what to do with
> > the libgcc_s symbol version mess, and I think we should probably just
> > hold off until the next release cycle for that now.
> 
> in case anyone wonders about the issue:
> 
> libgcc_s has a symbolic reference to an internal deprecated
> symbol with no default version on x86 (in gcc trunk).
> (so no @version only @@version which makes it invisible to
> musl and ld).
> 
> the deprecated function is a constructor in libgcc_s that
> initializes an unused struct in libgcc_s.
> 
> the only reason the symbol got versioned this way is because
> they want old binaries to work and remove the symbol from
> libgcc_s for new binaries, while still using the same symbol
> name in new binaries but with different abi behaviour: now
> it's only available in libgcc.a to fix their ifunc hack for
> multi-versioning.
> 
> since it's a symbol internal to libgcc and the semantics
> of the symbol changed they could have just used a new name
> and left the old one alone (so old stuff is guaranteed to not
> break including musl and new binaries use the new symbol with
> new semantics).

Yes but if the new symbol is not linkable like they made it, then old
musl-linked binaries depending on getting it from libgcc_s would fail
to find it at dynamic-link time and error out. I don't know whether
this can happen without use of the multiversioning feature, though,
which would not have worked with musl anyway since we don't have
ifunc.

> so i still think my proposed libgcc patch makes more sense
> than versioning:
> https://gcc.gnu.org/ml/gcc-patches/2015-05/msg00899.html
> 
> gcc is unlikely to fix this, but this is a nonsense usecase
> (and there is no other known use of versioning that is broken
> with musl.. in theory other libs may deprecate symbols in a
> similar way while still keeping symbolic references to them,
> but such use was not yet observed).

Are you sure? have you tried building C++ programs with gcc 3.x or 4.2
then using a libstdc++ from recent gcc? Unless the soname is
different, I suspect they're using symbol versions to make it "work"
and it will probably break catastrophically.

> in musl-gcc it can be worked around by preloading a noop
> __cpu_indicator_init and in a musl based gcc it can be
> patched out.

But it sounds like that patch will be unacceptable for upstream. There
are alternatives we could do, like providing in musl a list of useless
symbols to ignore (resolve to a nop func or a dummy data symbol) if
they're not found, but these are all a bit hackish and I worry more
stuff with symbol versioning will come back to bite us in the future.
The other option would be to teach gcc and binutils that musl does not
support symbol versions, but I worry that might break even more things
that expect versions to be available on Linux but that work fine with
musl's current dummying-out of version matching.

Rich

Re: Progress since 1.1.9

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2015-05-28 13:12:41 -0400]:

> On Thu, May 28, 2015 at 01:45:15PM +0200, Szabolcs Nagy wrote:
> > * Rich Felker <dalias@libc.org> [2015-05-27 19:13:08 -0400]:
> > > The one roadmap item I don't have any progress on is what to do with
> > > the libgcc_s symbol version mess, and I think we should probably just
> > > hold off until the next release cycle for that now.
> > 
> > in case anyone wonders about the issue:
> > 
> > libgcc_s has a symbolic reference to an internal deprecated
> > symbol with no default version on x86 (in gcc trunk).
> > (so no @version only @@version which makes it invisible to
> > musl and ld).
> > 
> > the deprecated function is a constructor in libgcc_s that
> > initializes an unused struct in libgcc_s.
> > 
> > the only reason the symbol got versioned this way is because
> > they want old binaries to work and remove the symbol from
> > libgcc_s for new binaries, while still using the same symbol
> > name in new binaries but with different abi behaviour: now
> > it's only available in libgcc.a to fix their ifunc hack for
> > multi-versioning.
> > 
> > since it's a symbol internal to libgcc and the semantics
> > of the symbol changed they could have just used a new name
> > and left the old one alone (so old stuff is guaranteed to not
> > break including musl and new binaries use the new symbol with
> > new semantics).
> 
> Yes but if the new symbol is not linkable like they made it, then old
> musl-linked binaries depending on getting it from libgcc_s would fail
> to find it at dynamic-link time and error out. I don't know whether
> this can happen without use of the multiversioning feature, though,
> which would not have worked with musl anyway since we don't have
> ifunc.

old binaries don't reference the new name unless new name == old name.

there is no reason to make the old name unlinkable in libgcc_s.so if
the new symbol has new name.  (old binaries only reference the old
name which is available in libgcc_s.so and libgcc.a like before, new
binaries only reference the new name that is only in libgcc.a).
(this is what my patch does).

(it was possible to reference __cpu_indicator_init in libgcc_s.so without
using ifunc based multi-versioning: the x86 specific __builtin_cpu_init
can be called directly to make __builtin_cpu_is and __builtin_cpu_supports
work, but i think only c++ code used -lgcc_s).

> > so i still think my proposed libgcc patch makes more sense
> > than versioning:
> > https://gcc.gnu.org/ml/gcc-patches/2015-05/msg00899.html
> > 
> > gcc is unlikely to fix this, but this is a nonsense usecase
> > (and there is no other known use of versioning that is broken
> > with musl.. in theory other libs may deprecate symbols in a
> > similar way while still keeping symbolic references to them,
> > but such use was not yet observed).
> 
> Are you sure? have you tried building C++ programs with gcc 3.x or 4.2
> then using a libstdc++ from recent gcc? Unless the soname is
> different, I suspect they're using symbol versions to make it "work"
> and it will probably break catastrophically.

ok that may cause problems (but won't make the dynamic linking fail:
all versioned symbols in libstdc++ has default version).

but mixing libs linked with different versions of libstdc++ is
broken anyway if any two libs happen to use the same symbol with
different versions.

> > in musl-gcc it can be worked around by preloading a noop
> > __cpu_indicator_init and in a musl based gcc it can be
> > patched out.
> 
> But it sounds like that patch will be unacceptable for upstream. There
> are alternatives we could do, like providing in musl a list of useless
> symbols to ignore (resolve to a nop func or a dummy data symbol) if
> they're not found, but these are all a bit hackish and I worry more
> stuff with symbol versioning will come back to bite us in the future.
> The other option would be to teach gcc and binutils that musl does not
> support symbol versions, but I worry that might break even more things
> that expect versions to be available on Linux but that work fine with
> musl's current dummying-out of version matching.

if things use symbol versioning then musl will have problems.

but is that really widespread?

Re: Progress since 1.1.9

[Rich Felker]

On Thu, May 28, 2015 at 09:47:34PM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2015-05-28 13:12:41 -0400]:
> 
> > On Thu, May 28, 2015 at 01:45:15PM +0200, Szabolcs Nagy wrote:
> > > * Rich Felker <dalias@libc.org> [2015-05-27 19:13:08 -0400]:
> > > > The one roadmap item I don't have any progress on is what to do with
> > > > the libgcc_s symbol version mess, and I think we should probably just
> > > > hold off until the next release cycle for that now.
> > > 
> > > in case anyone wonders about the issue:
> > > 
> > > libgcc_s has a symbolic reference to an internal deprecated
> > > symbol with no default version on x86 (in gcc trunk).
> > > (so no @version only @@version which makes it invisible to
> > > musl and ld).
> > > 
> > > the deprecated function is a constructor in libgcc_s that
> > > initializes an unused struct in libgcc_s.
> > > 
> > > the only reason the symbol got versioned this way is because
> > > they want old binaries to work and remove the symbol from
> > > libgcc_s for new binaries, while still using the same symbol
> > > name in new binaries but with different abi behaviour: now
> > > it's only available in libgcc.a to fix their ifunc hack for
> > > multi-versioning.
> > > 
> > > since it's a symbol internal to libgcc and the semantics
> > > of the symbol changed they could have just used a new name
> > > and left the old one alone (so old stuff is guaranteed to not
> > > break including musl and new binaries use the new symbol with
> > > new semantics).
> > 
> > Yes but if the new symbol is not linkable like they made it, then old
> > musl-linked binaries depending on getting it from libgcc_s would fail
> > to find it at dynamic-link time and error out. I don't know whether
> > this can happen without use of the multiversioning feature, though,
> > which would not have worked with musl anyway since we don't have
> > ifunc.
> 
> old binaries don't reference the new name unless new name == old name.
> 
> there is no reason to make the old name unlinkable in libgcc_s.so if
> the new symbol has new name.  (old binaries only reference the old
> name which is available in libgcc_s.so and libgcc.a like before, new
> binaries only reference the new name that is only in libgcc.a).
> (this is what my patch does).

Yes, that would have been preferable, but it looks hard to get gcc to
change this... :(

> (it was possible to reference __cpu_indicator_init in libgcc_s.so without
> using ifunc based multi-versioning: the x86 specific __builtin_cpu_init
> can be called directly to make __builtin_cpu_is and __builtin_cpu_supports
> work, but i think only c++ code used -lgcc_s).

Bleh.

> > > so i still think my proposed libgcc patch makes more sense
> > > than versioning:
> > > https://gcc.gnu.org/ml/gcc-patches/2015-05/msg00899.html
> > > 
> > > gcc is unlikely to fix this, but this is a nonsense usecase
> > > (and there is no other known use of versioning that is broken
> > > with musl.. in theory other libs may deprecate symbols in a
> > > similar way while still keeping symbolic references to them,
> > > but such use was not yet observed).
> > 
> > Are you sure? have you tried building C++ programs with gcc 3.x or 4.2
> > then using a libstdc++ from recent gcc? Unless the soname is
> > different, I suspect they're using symbol versions to make it "work"
> > and it will probably break catastrophically.
> 
> ok that may cause problems (but won't make the dynamic linking fail:
> all versioned symbols in libstdc++ has default version).

If so, it's worse than making dynamic linking fail -- it presumably
causes silent runtime memory corruption due to using wrong data
structures.

> but mixing libs linked with different versions of libstdc++ is
> broken anyway if any two libs happen to use the same symbol with
> different versions.

I agree completely that symbol versioning is a horrible idea and does
not properly solve the problem it was invented to solve. (And this
problem can be addressed much better, though still not perfectly, with
macro-based, inline-function-based, or asm-label-based redirects in
header files.) But I believe the way gcc is doing things _does_ work
reliably when you do not have C++ types crossing DSO interface
boundaries, and that's a fairly large portion of real-world usage.
(And when C++ types do cross DSO boundaries, it's usually
intra-project DSOs, not system-wide ones in the public lib dirs,
because people generally know C++ ABI is continually broken for public
library interfaces.)

> > > in musl-gcc it can be worked around by preloading a noop
> > > __cpu_indicator_init and in a musl based gcc it can be
> > > patched out.
> > 
> > But it sounds like that patch will be unacceptable for upstream. There
> > are alternatives we could do, like providing in musl a list of useless
> > symbols to ignore (resolve to a nop func or a dummy data symbol) if
> > they're not found, but these are all a bit hackish and I worry more
> > stuff with symbol versioning will come back to bite us in the future.
> > The other option would be to teach gcc and binutils that musl does not
> > support symbol versions, but I worry that might break even more things
> > that expect versions to be available on Linux but that work fine with
> > musl's current dummying-out of version matching.
> 
> if things use symbol versioning then musl will have problems.
> 
> but is that really widespread?

I recall ALSA libs were using it; this is why we added the hack that
we have now, always using the version ld would choose instead of the
first-found version.

Rich