gcc'c crtstuff.c: a musl-related experience

gcc'c crtstuff.c: a musl-related experience

[writeonce]

Greetings,

I thought I should share a recent experience that I had while building a 
cross compiler for i686-unknown-linux with x86_64 as a host.  The 
problem I encountered (and now solved) is yet another "penalty" for me 
deviating from the norm, which in this case is gcc's way of hard-coding 
libc-specific features into the toolchain (mglibc, muclibc, etc.)  As my 
toolchain does not provide such options, the compiler was mistakenly 
thinking that glibc was the default, which led to the "incident" I'd 
like to describe.

The two files at stake are gcc/linux.h, and libgcc/crtstuff.c.  In the 
former, a built-in macro named __gnu_linux__ is defined when glibc is 
the toolchain's default libc (OPTION_GLIBC).  In the latter, a macro 
named USE_PT_GNU_EH_FRAME is defined when __gnu_linux__ is defined, 
which accordingly prevents USE_EH_FRAME_REGISTRY from being defined 
shortly thereafter.  The absence of this last macro results in a 
crtbegin.o that is incompatible with musl, at least in some cases (a 
simple c++ application crashes after an exception has been thrown, which 
is how I came to notice it).

The resulting file crtbegin.o is installed in one of the toolchain's 
directories, and is thus shared between all the libc's targeted by the 
toolchain.  This means that when the toolchain's default libc is glibc, 
specifying -mmusl would result in a binary that has an incompatible 
crtbegin.o, at least in the above environment (i686 running on x86_64).

To complete the picture: in the musl-cross toolchains, glibc is _not_ 
the default libc, and __gnu_linux_ is accordingly not defined.  All you 
need to do in order to reproduce the above bug is moving around the 
options so that glibc becomes the default.  With the current code of 
crtstuff.c, one easy way to avoid the bug while still keeping -mglibc, 
-mmusl, etc. is to "manually" define USE_EH_FRAME_REGISTRY.  Not ideal, 
but still works.

zg

Re: gcc'c crtstuff.c: a musl-related experience

[Rich Felker]

On Sun, Jun 15, 2014 at 05:36:37PM -0400, writeonce@midipix.org wrote:
> The two files at stake are gcc/linux.h, and libgcc/crtstuff.c.  In
> the former, a built-in macro named __gnu_linux__ is defined when
> glibc is the toolchain's default libc (OPTION_GLIBC).  In the
> latter, a macro named USE_PT_GNU_EH_FRAME is defined when
> __gnu_linux__ is defined, which accordingly prevents
> USE_EH_FRAME_REGISTRY from being defined shortly thereafter.  The
> absence of this last macro results in a crtbegin.o that is
> incompatible with musl, at least in some cases (a simple c++
> application crashes after an exception has been thrown, which is how
> I came to notice it).

I'm not an expert on gcc internals, but I'm doubtful of your
explanation of the problem. From just the macro names,
USE_PT_GNU_EH_FRAME is the one you want, and USE_EH_FRAME_REGISTRY
sounds like the nasty old fallback that's incompatible with static
linking of libgcc_eh.a.

However I'm confused what this has to do with crtbegin.o/crtend.o.
Perhaps someone who's worked with toolchain stuff could comment.

The reason I'm bringing this up is that there might really be some bug
in musl or in your toolchain setup, where switching to the old code is
just covering up the bug.

Rich

Re: gcc'c crtstuff.c: a musl-related experience

[Rich Felker]

On Sun, Jun 15, 2014 at 09:41:31PM -0400, Rich Felker wrote:
> On Sun, Jun 15, 2014 at 05:36:37PM -0400, writeonce@midipix.org wrote:
> > The two files at stake are gcc/linux.h, and libgcc/crtstuff.c.  In
> > the former, a built-in macro named __gnu_linux__ is defined when
> > glibc is the toolchain's default libc (OPTION_GLIBC).  In the
> > latter, a macro named USE_PT_GNU_EH_FRAME is defined when
> > __gnu_linux__ is defined, which accordingly prevents
> > USE_EH_FRAME_REGISTRY from being defined shortly thereafter.  The
> > absence of this last macro results in a crtbegin.o that is
> > incompatible with musl, at least in some cases (a simple c++
> > application crashes after an exception has been thrown, which is how
> > I came to notice it).
> 
> I'm not an expert on gcc internals, but I'm doubtful of your
> explanation of the problem. From just the macro names,
> USE_PT_GNU_EH_FRAME is the one you want, and USE_EH_FRAME_REGISTRY
> sounds like the nasty old fallback that's incompatible with static
> linking of libgcc_eh.a.
> 
> However I'm confused what this has to do with crtbegin.o/crtend.o.
> Perhaps someone who's worked with toolchain stuff could comment.
> 
> The reason I'm bringing this up is that there might really be some bug
> in musl or in your toolchain setup, where switching to the old code is
> just covering up the bug.

I followed up on this and it's entirely a toolchain issue. Both
crtstuff and libgcc are involved:

crtstuff can be built in two different ways. One way is for supporting
the old EH_FRAME_REGISTRY stuff, whereby the global ctors/dtors for a
shared library or the main program must register/unregister their
unwind tables with libgcc_s (or libgcc_eh, but this has serious
problems in this configuration except when static linking the whole
program) and thus the registration code is required in crtstart.o and
crtend.o. The other way omits this code, assuming the modern system
based on dl_iterate_phdr will be used lib libgcc.

The libgcc unwind code can also be built in two ways. One has support
for the new PT_GNU_EH_FRAME system, which requires additional code in
libgcc_eh and support from libc. The other only suppors the old way,
where registration of unwind tables by individual modules is required.

In short, crtstuff can be built with or without support for the old
junk (the new way works regardless of how crtstuff was built, as long
as libgcc supports it), and libgcc can be built with or without
support for the new stuff (the old way work regardless of how libgcc
was built).

The current musl-cross patches fix libgcc to support the new way, but
they don't fix crtstuff to drop the old bloated code. If crtstuff is
fixed (or if you copy crtbegin.o and crtend.o from a toolchain built
for glibc), then static-linked programs using exceptions start
crashing. And here's why:

The default GCC specs have:

#define LINK_EH_SPEC "%{!static:--eh-frame-hdr} "

This inhibits the linker from emitting a PT_GNU_EH_FRAME program
header when static linking. I really have no idea how this is supposed
to work, but somehow it does work for glibc. (Perhaps their
dl_iterate_phdr is patching up a fake PT_GNU_EH_FRAME to pass back to
the application?)

So, here's what should be changed in musl-cross, in my opinion:

1. The crtstuff.c preprocessor checks for whether support for the old
   bloated/buggy registration system is needed should be fixed to
   align precisely with the checks used in libgcc unwind code to
   determine whether the latter uses dl_iterate_phdr.

2. Ether the specfile should be changed to provide --eh-frame-hdr to
   the linker even when static-linking, or we should figure out how
   glibc manages to get by without this program header and do the
   same (not sure if this would be on the musl side or the toolchain
   side).

Please note that these changes are not mandatory; at present,
everything works, but crtbegin.o and crtend.o are unnecessarily
bloated and this affects static binary size.

Rich