Linking musl with ld.gold

Linking musl with ld.gold

[Stephen Thomas]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

I have come across a problem and it only appears when the gold linker is used. I am using the latest release of binutils, binutils-2.24.51.0.3.
I discovered that busybox was not flushing stdout (as there was no prompt appearing) when using musl. Busybox calls fflush(NULL) which should flush stdout as done in src/stdio/fflush.c. 
In that file I checked the value for __stdout_used and it came back as 0. So I changed the declaration of the weak symbol to an extern FILE* __stdout_used and stdout was being flushed.
Has anyone else seen this and have they reported this apparent bug in binutils?
Thomo 		 	   		  

[-- Attachment #2: Type: text/html, Size: 955 bytes --]

Re: Linking musl with ld.gold

[Szabolcs Nagy]

* Stephen Thomas <scjthm@live.com> [2014-05-06 10:07:59 +0100]:
> I have come across a problem and it only appears when the gold linker is used. I am using the latest release of binutils, binutils-2.24.51.0.3.
> I discovered that busybox was not flushing stdout (as there was no prompt appearing) when using musl. Busybox calls fflush(NULL) which should flush stdout as done in src/stdio/fflush.c. 
> In that file I checked the value for __stdout_used and it came back as 0. So I changed the declaration of the weak symbol to an extern FILE* __stdout_used and stdout was being flushed.
> Has anyone else seen this and have they reported this apparent bug in binutils?


i think we only reported a broken tls visibility issue against gold
https://sourceware.org/bugzilla/show_bug.cgi?id=16728

you should try to reproduce the bug on a minimal example, eg. the
following code works here with gold (binutils 2.22)

// a.c
struct foo {int i;};
static struct foo *const dummy = 0;
extern struct foo *const hasfoo __attribute__((weak, alias("dummy")));
int f(void)
{
        return hasfoo ? hasfoo->i : 0;
}
// b.c
struct foo {int i;};
static struct foo foo = {42};
struct foo *const hasfoo = &foo;
// main.c
int f(void);
int main()
{
        return f();
}

gcc main.o a.o -o t1
gcc main.o a.o b.o -o t2
./t1 returns 0
./t2 returns 42

RE: Linking musl with ld.gold

[Stephen Thomas]

[-- Attachment #1.1: Type: text/plain, Size: 5657 bytes --]

I think that I have managed to reproduce the problem, in as much isolation as possible. I believe that there is a missing step in your test, which is included in the build, and that is the creation of the archive. I may have done something incorrect also, which is also very likely.
I attach a simple script for running the tests on my machine. Unfortunately, the system I use doesn't display indicate where ld.gold was being used as it chooses to do this by changing the symlink to which ld points to.
To summari(s/z)e when using ar to create an archive the weak symbol doesn't appear to be overridden, using either ld.bfd nor ld.gold. The script that was used to generate the output is attached and this should describe the 4 test cases. Test case #2 is probably irrelevant, and the first two are simply reproducing the results of your tests.
Thomo
gt-bld / # binutils-config --linker ld.bfd * Setting default linker to ld.bfd for x86_64-pc-linux-gnu-2.24.51.0.3 ...     gt-bld / # /tmp/badar.sh ************************************************************Running with ...gcc (Gentoo 4.8.2-r1 p1.4-ssptest, pie-0.5.9-ssptest) 4.8.2Copyright (C) 2013 Free Software Foundation, Inc.This is free software; see the source for copying conditions.  There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
GNU ld (Linux/GNU Binutils) 2.24.51.0.3.20140127Copyright 2014 Free Software Foundation, Inc.This program is free software; you may redistribute it under the terms ofthe GNU General Public License version 3 or (at your option) a later version.This program has absolutely no warranty.************************************************************Case 0: 0************************************************************************************************************************Case 1: 42************************************************************ar: creating a.a
a.o:0000000000000000 r dummy0000000000000000 T f0000000000000000 V hasfoo************************************************************Case 2: 0************************************************************ar: creating a.a
a.o:0000000000000000 r dummy0000000000000000 T f0000000000000000 V hasfoo
b.o:0000000000000000 d foo0000000000000000 R hasfoo************************************************************Case 3: 0************************************************************gt-bld / # binutils-config --linker ld.gold * Setting default linker to ld.gold for x86_64-pc-linux-gnu-2.24.51.0.3 ...                                                                                                                                                            [ ok ]gt-bld / # /tmp/badar.sh ************************************************************Running with ...gcc (Gentoo 4.8.2-r1 p1.4-ssptest, pie-0.5.9-ssptest) 4.8.2Copyright (C) 2013 Free Software Foundation, Inc.This is free software; see the source for copying conditions.  There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
GNU gold (Linux/GNU Binutils 2.24.51.0.3.20140127) 1.11Copyright 2014 Free Software Foundation, Inc.This program is free software; you may redistribute it under the terms ofthe GNU General Public License version 3 or (at your option) a later version.This program has absolutely no warranty.************************************************************Case 0: 0************************************************************************************************************************Case 1: 42************************************************************ar: creating a.a
a.o:0000000000000000 r dummy0000000000000000 T f0000000000000000 V hasfoo************************************************************Case 2: 0************************************************************ar: creating a.a
a.o:0000000000000000 r dummy0000000000000000 T f0000000000000000 V hasfoo
b.o:0000000000000000 d foo0000000000000000 R hasfoo************************************************************Case 3: 0************************************************************ 

> Date: Tue, 6 May 2014 12:14:10 +0200
> From: nsz@port70.net
> To: musl@lists.openwall.com
> Subject: Re: [musl] Linking musl with ld.gold
> 
> * Stephen Thomas <scjthm@live.com> [2014-05-06 10:07:59 +0100]:
> > I have come across a problem and it only appears when the gold linker is used. I am using the latest release of binutils, binutils-2.24.51.0.3.
> > I discovered that busybox was not flushing stdout (as there was no prompt appearing) when using musl. Busybox calls fflush(NULL) which should flush stdout as done in src/stdio/fflush.c. 
> > In that file I checked the value for __stdout_used and it came back as 0. So I changed the declaration of the weak symbol to an extern FILE* __stdout_used and stdout was being flushed.
> > Has anyone else seen this and have they reported this apparent bug in binutils?
> 
> 
> i think we only reported a broken tls visibility issue against gold
> https://sourceware.org/bugzilla/show_bug.cgi?id=16728
> 
> you should try to reproduce the bug on a minimal example, eg. the
> following code works here with gold (binutils 2.22)
> 
> // a.c
> struct foo {int i;};
> static struct foo *const dummy = 0;
> extern struct foo *const hasfoo __attribute__((weak, alias("dummy")));
> int f(void)
> {
>         return hasfoo ? hasfoo->i : 0;
> }
> // b.c
> struct foo {int i;};
> static struct foo foo = {42};
> struct foo *const hasfoo = &foo;
> // main.c
> int f(void);
> int main()
> {
>         return f();
> }
> 
> gcc main.o a.o -o t1
> gcc main.o a.o b.o -o t2
> ./t1 returns 0
> ./t2 returns 42
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 7833 bytes --]

[-- Attachment #2: badar.sh --]
[-- Type: application/x-sh, Size: 1898 bytes --]

Re: Linking musl with ld.gold

[Szabolcs Nagy]

* Stephen Thomas <scjthm@live.com> [2014-05-06 22:11:32 +0100]:
> I think that I have managed to reproduce the problem, in as much isolation as possible. I believe that there is a missing step in your test, which is included in the build, and that is the creation of the archive. I may have done something incorrect also, which is also very likely.
> I attach a simple script for running the tests on my machine. Unfortunately, the system I use doesn't display indicate where ld.gold was being used as it chooses to do this by changing the symlink to which ld points to.
> To summari(s/z)e when using ar to create an archive the weak symbol doesn't appear to be overridden, using either ld.bfd nor ld.gold. The script that was used to generate the output is attached and this should describe the 4 test cases. Test case #2 is probably irrelevant, and the first two are simply reproducing the results of your tests.


only the object files with referenced symbols are linked from an archive

so only a.o with the given main.o because of the symbol f

now if you make some reference in main.c such that b.o should
be included but main still returns 0 that would be a bug

eg. add a void g(void){} to b.c and call it from main.c

RE: Linking musl with ld.gold

[Stephen Thomas]

[-- Attachment #1.1: Type: text/plain, Size: 1938 bytes --]

> 
> only the object files with referenced symbols are linked from an archive
> 
> so only a.o with the given main.o because of the symbol f
> 
> now if you make some reference in main.c such that b.o should
> be included but main still returns 0 that would be a bug
> 
> eg. add a void g(void){} to b.c and call it from main.c

Ok, thanks for that info. It appears that there is a problem in gcc 4.9 and not 4.8.3. Come to think of it this I only noticed the prompt being wrong when using  gcc 4.9. I will check this with the git version of gcc also. I attach the code also.
This is from 4.8************************************************************Running with ...gcc (Gentoo 4.8.2-r1 p1.4-ssptest, pie-0.5.9-ssptest) 4.8.2Copyright (C) 2013 Free Software Foundation, Inc.This is free software; see the source for copying conditions.  There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Case A: 42************************************************************ar: creating a.a
a.o:0000000000000000 r dummy0000000000000000 T f0000000000000000 V hasfoo
b.o:0000000000000000 d foo0000000000000000 T g0000000000000000 R hasfooCase B: 42************************************************************
and this is from 4.9************************************************************Running with ...x86_64-buildroot-linux-musl-gcc (Buildroot 2014.05-git-00965-gf077df0-dirty) 4.9.0Copyright (C) 2014 Free Software Foundation, Inc.This is free software; see the source for copying conditions.  There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Case A: 0************************************************************ar: creating a.a
a.o:0000000000000000 r dummy0000000000000000 T f0000000000000000 V hasfoo
b.o:0000000000000000 d foo0000000000000000 T g0000000000000000 R hasfooCase B: 0************************************************************
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 2829 bytes --]

[-- Attachment #2: badgcc.sh --]
[-- Type: application/x-sh, Size: 1283 bytes --]

Re: Linking musl with ld.gold

[Timo Teras]

On Wed, 7 May 2014 10:04:24 +0100
Stephen Thomas <scjthm@live.com> wrote:

> > only the object files with referenced symbols are linked from an
> > archive
> > 
> > so only a.o with the given main.o because of the symbol f
> > 
> > now if you make some reference in main.c such that b.o should
> > be included but main still returns 0 that would be a bug
> > 
> > eg. add a void g(void){} to b.c and call it from main.c  
> 
> Ok, thanks for that info. It appears that there is a problem in gcc
> 4.9 and not 4.8.3.

Is perhaps -ffunction-sections and/or -fdata-sections added
automatically? Those would break musl like experienced.

- Timo

RE: Linking musl with ld.gold

[Stephen Thomas]

[-- Attachment #1.1: Type: text/plain, Size: 1118 bytes --]



> > > only the object files with referenced symbols are linked from an
> > > archive
> > > 
> > > so only a.o with the given main.o because of the symbol f
> > > 
> > > now if you make some reference in main.c such that b.o should
> > > be included but main still returns 0 that would be a bug
> > > 
> > > eg. add a void g(void){} to b.c and call it from main.c  
> > 
> > Ok, thanks for that info. It appears that there is a problem in gcc
> > 4.9 and not 4.8.3.
> 
> Is perhaps -ffunction-sections and/or -fdata-sections added
> automatically? Those would break musl like experienced.

Thanks for that tip, but I didn't change the sections in the linker script. I didn't add any flags either. I attach the single test case cleaned up a bit. I ran this on the buildroot toolchain (standard config with uclibc and with 2.24.0 binutils) with 4.8.2 and the result is 42. I rebuilt the toolchain only changing the compiler version to 4.9.2 and the result is 0.
This corroborates what I was seeing (or not as in the case of the prompt not appearing in busybox) with my builds also.
Thomo 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 1549 bytes --]

[-- Attachment #2: badgcc.sh --]
[-- Type: application/x-sh, Size: 1133 bytes --]

Re: Linking musl with ld.gold

[Rich Felker]

On Wed, May 07, 2014 at 01:04:43PM +0300, Timo Teras wrote:
> On Wed, 7 May 2014 10:04:24 +0100
> Stephen Thomas <scjthm@live.com> wrote:
> 
> > > only the object files with referenced symbols are linked from an
> > > archive
> > > 
> > > so only a.o with the given main.o because of the symbol f
> > > 
> > > now if you make some reference in main.c such that b.o should
> > > be included but main still returns 0 that would be a bug
> > > 
> > > eg. add a void g(void){} to b.c and call it from main.c  
> > 
> > Ok, thanks for that info. It appears that there is a problem in gcc
> > 4.9 and not 4.8.3.
> 
> Is perhaps -ffunction-sections and/or -fdata-sections added
> automatically? Those would break musl like experienced.

They should not break musl; if they do, it's a compiler bug. The
strong symbol that overrides the weak symbol elsewhere is not unused
and available for garbage collection because it's referenced.

I suspect your claim is just wrong, since IIRC people have
successfully used these options with musl.

Rich

RE: Linking musl with ld.gold

[Stephen Thomas]

[-- Attachment #1: Type: text/plain, Size: 2473 bytes --]



> On Wed, May 07, 2014 at 01:04:43PM +0300, Timo Teras wrote:
> > On Wed, 7 May 2014 10:04:24 +0100
> > Stephen Thomas <scjthm@live.com> wrote:
> > 
> > > > only the object files with referenced symbols are linked from an
> > > > archive
> > > > 
> > > > so only a.o with the given main.o because of the symbol f
> > > > 
> > > > now if you make some reference in main.c such that b.o should
> > > > be included but main still returns 0 that would be a bug
> > > > 
> > > > eg. add a void g(void){} to b.c and call it from main.c  
> > > 
> > > Ok, thanks for that info. It appears that there is a problem in gcc
> > > 4.9 and not 4.8.3.
> > 
> > Is perhaps -ffunction-sections and/or -fdata-sections added
> > automatically? Those would break musl like experienced.
> 
> They should not break musl; if they do, it's a compiler bug. The
> strong symbol that overrides the weak symbol elsewhere is not unused
> and available for garbage collection because it's referenced.
> 
> I suspect your claim is just wrong, since IIRC people have
> successfully used these options with musl.

I will raise an issue with the gcc team (but I don't really want to build from git, as it takes too long). I am not saying that the library doesn't work, I am merely saying that there was a bug where stdout was not being flushed, and this in my opinion was due to the weak symbol in fflush not being overridden. I did run that single test case on the two different builds and the result was different. This was on two clean buildroot branches based on uclibc. 
I don't understand what you mean by garbage collection. Basically, if I understand this correctly, there is a weak symbol defined in fflush.c for the purpose of allowing this file to run without the an implementation that initialises the real symbol with the internal implementation of stdout (which is just basically a manufactured wrapper around file descriptor number 1). This is good as far as unit testing goes, doesn't use #defines but the linker -- good stuff. However,  when  I noticed was that the prompt on busybox using musl was not appearing until after a new line was entered, I added a discovered that the value of the pointer was 0 (NULL). If you still suspect that my claim is wrong, then I believe you are saying that this is not the case. It would be nice if someone who has gcc 4.9 installed could run either run the test or check that busybox is working. 
Thomo

 		 	   		  

[-- Attachment #2: Type: text/html, Size: 3103 bytes --]

Re: Linking musl with ld.gold

[Rich Felker]

On Thu, May 08, 2014 at 03:08:41AM +0100, Stephen Thomas wrote:
> > On Wed, May 07, 2014 at 01:04:43PM +0300, Timo Teras wrote:
> > > On Wed, 7 May 2014 10:04:24 +0100
> > > Stephen Thomas <scjthm@live.com> wrote:
> > > 
> > > > > only the object files with referenced symbols are linked from an
> > > > > archive
> > > > > 
> > > > > so only a.o with the given main.o because of the symbol f
> > > > > 
> > > > > now if you make some reference in main.c such that b.o should
> > > > > be included but main still returns 0 that would be a bug
> > > > > 
> > > > > eg. add a void g(void){} to b.c and call it from main.c  
> > > > 
> > > > Ok, thanks for that info. It appears that there is a problem in gcc
> > > > 4.9 and not 4.8.3.
> > > 
> > > Is perhaps -ffunction-sections and/or -fdata-sections added
> > > automatically? Those would break musl like experienced.
> > 
> > They should not break musl; if they do, it's a compiler bug. The
> > strong symbol that overrides the weak symbol elsewhere is not unused
> > and available for garbage collection because it's referenced.
> > 
> > I suspect your claim is just wrong, since IIRC people have
> > successfully used these options with musl.
> 
> I will raise an issue with the gcc team (but I don't really want to
> build from git, as it takes too long). I am not saying that the
> library doesn't work, I am merely saying that there was a bug where
> stdout was not being flushed, and this in my opinion was due to the
> weak symbol in fflush not being overridden. I did run that single
> test case on the two different builds and the result was different.
> This was on two clean buildroot branches based on uclibc.

My reply to Timo was not in doubt that you observed such an issue, but
rather expressing doubt about his possible explanation. I don't think
it has anything to do with -ffunction-sections or -fdata-sections.

> I don't understand what you mean by garbage collection. Basically,

Those two options are meant to be used with the linker option
--gc-sections, which performs garbage collection to remove any
sections which are not referenced.

> if I understand this correctly, there is a weak symbol defined in
> fflush.c for the purpose of allowing this file to run without the an
> implementation that initialises the real symbol with the internal
> implementation of stdout (which is just basically a manufactured
> wrapper around file descriptor number 1). This is good as far as
> unit testing goes, doesn't use #defines but the linker -- good
> stuff. However, when I noticed was that the prompt on busybox using
> musl was not appearing until after a new line was entered, I added a
> discovered that the value of the pointer was 0 (NULL). If you still
> suspect that my claim is wrong, then I believe you are saying that

I'm just saying Timo's claim about the mechanism of what you observed
is probably wrong. The way the 'magic' with weak symbols work is that
the weak definition satisfies the reference, so that the linker does
not need to pull in additional object files to satisfy it. But once an
additional file which has a strong definition is pulled in (as a
result of another undefined symbol reference), the strong definition
is _referenced_ (because it overrides any weak definitions) and thus
its section is not a legal candidate for garbage collection via
--gc-sections.

> this is not the case. It would be nice if someone who has gcc 4.9
> installed could run either run the test or check that busybox is
> working.

The bug is almost surely in the linker, not gcc, unless it's a matter
of gcc passing bad options to the linker. You can add -v to the gcc
command line for linking to see exactly how it invokes the linker.

Rich

Re: Linking musl with ld.gold

[Timo Teras]

On Wed, 7 May 2014 21:06:05 -0400
Rich Felker <dalias@libc.org> wrote:

> On Wed, May 07, 2014 at 01:04:43PM +0300, Timo Teras wrote:
> > Is perhaps -ffunction-sections and/or -fdata-sections added
> > automatically? Those would break musl like experienced.
> 
> They should not break musl; if they do, it's a compiler bug. The
> strong symbol that overrides the weak symbol elsewhere is not unused
> and available for garbage collection because it's referenced.
> 
> I suspect your claim is just wrong, since IIRC people have
> successfully used these options with musl.

-fdata-sections breaks things to my understanding.

stdin.c (and others), have:

FILE *const stdin = &f;
FILE *const __stdin_used = &f;

And __stdin_used is only ever referenced via weak alias. -fdata-section
makes each symbol go to different section, and if linker has
gc-sections, it'll remove any unreferenced section. This means
__stdin_used will never get pulled in causing problems like described
in the original mail.

With -ffunction-sections/-fdata-sections/-Wl,--gc-sections it is no
longer valid assumption that if compilation unit gets pulled in, all of
the symbols defined in it will get pulled in.

Of course this does not affect .so as visible exported symbols are
reachable and not GCd. But static build would be effected.

- Timo

Re: Linking musl with ld.gold

[Rich Felker]

On Thu, May 08, 2014 at 08:11:26AM +0300, Timo Teras wrote:
> On Wed, 7 May 2014 21:06:05 -0400
> Rich Felker <dalias@libc.org> wrote:
> 
> > On Wed, May 07, 2014 at 01:04:43PM +0300, Timo Teras wrote:
> > > Is perhaps -ffunction-sections and/or -fdata-sections added
> > > automatically? Those would break musl like experienced.
> > 
> > They should not break musl; if they do, it's a compiler bug. The
> > strong symbol that overrides the weak symbol elsewhere is not unused
> > and available for garbage collection because it's referenced.
> > 
> > I suspect your claim is just wrong, since IIRC people have
> > successfully used these options with musl.
> 
> -fdata-sections breaks things to my understanding.
> 
> stdin.c (and others), have:
> 
> FILE *const stdin = &f;
> FILE *const __stdin_used = &f;
> 
> And __stdin_used is only ever referenced via weak alias. -fdata-section
> makes each symbol go to different section, and if linker has
> gc-sections, it'll remove any unreferenced section. This means

But the section is not unreferenced. There's a reference to
__stdin_used from __stdio_exit.o. Ignoring this reference just because
it could have been satisfied by a weak symbol is wrong.

> __stdin_used will never get pulled in causing problems like described
> in the original mail.

I don't see anywhere it's documented to behave this way, and in fact
making it behave this way would not only be harmful but more difficult
than making it behave correctly. The basic way a linker works is to
pull in all objects to satisfy undefined symbols; performing GC on
sections is a separate task that takes place (or at least should)
after all symbols have been resolved, and thus once it's possible to
know which symbols are referenced and which are not.

Rich