[PATCH] force LTO to be disabled when compiling dlstart.lo

[PATCH] force LTO to be disabled when compiling dlstart.lo

[Andre McCurdy]

When LTO is enabled, the _dlstart_c symbol seems to get removed before
the crt_arch.h assembler fragment which references it has been linked.

Disabling LTO for dlstart.lo is a less intrusive workaround than making
_dlstart_c globally visible.

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
---
 Makefile  | 4 ++++
 configure | 8 ++++++++
 2 files changed, 12 insertions(+)

diff --git a/Makefile b/Makefile
index 6559295..48580fc 100644
--- a/Makefile
+++ b/Makefile
@@ -89,6 +89,10 @@ src/ldso/dlstart.lo src/ldso/dynlink.lo: src/internal/dynlink.h arch/$(ARCH)/rel
 
 crt/crt1.o crt/Scrt1.o src/ldso/dlstart.lo: $(wildcard arch/$(ARCH)/crt_arch.h)
 
+# Disable LTO as a workaround for the _dlstart_c symbol being removed
+# before the crt_arch.h assembler fragment which references it is linked.
+src/ldso/dlstart.lo: CFLAGS += $(CFLAGS_NOLTO)
+
 crt/Scrt1.o: CFLAGS += -fPIC
 
 OPTIMIZE_SRCS = $(wildcard $(OPTIMIZE_GLOBS:%=src/%))
diff --git a/configure b/configure
index 143dc92..69c6dab 100755
--- a/configure
+++ b/configure
@@ -112,6 +112,7 @@ fi
 CFLAGS_C99FSE=
 CFLAGS_AUTO=
 CFLAGS_MEMOPS=
+CFLAGS_NOLTO=
 CFLAGS_NOSSP=
 LDFLAGS_AUTO=
 OPTIMIZE_GLOBS=
@@ -296,6 +297,12 @@ CFLAGS_C99FSE="$CFLAGS_C99FSE -D__may_alias__="
 fi
 
 #
+# Check for options to disable LTO. If not found, this is not an
+# error; we assume the toolchain does not support LTO.
+#
+tryflag CFLAGS_NOLTO -fno-lto
+
+#
 # Check for options to disable stack protector, which needs to be
 # disabled for a few early-bootstrap translation units. If not found,
 # this is not an error; we assume the toolchain does not do ssp.
@@ -564,6 +571,7 @@ CC = $CC
 CFLAGS = $CFLAGS_AUTO $CFLAGS
 CFLAGS_C99FSE = $CFLAGS_C99FSE
 CFLAGS_MEMOPS = $CFLAGS_MEMOPS
+CFLAGS_NOLTO = $CFLAGS_NOLTO
 CFLAGS_NOSSP = $CFLAGS_NOSSP
 CPPFLAGS = $CPPFLAGS
 LDFLAGS = $LDFLAGS_AUTO $LDFLAGS
-- 
1.9.1

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Alexander Monakov]

Hello,

Sorry for not joining the discussion earlier.

Andre, can you specify your GCC and Binutils version?  The reason I ask, with
modern toolchain you shouldn't be seeing the error you reported.  The fact
that _dlstart_c function is used from assembly should have been communicated
from the linker to the compiler via the "linker plugin".  If linker plugin was
not used, that would explan the problem.

Can you also check if adding '-fuse-linker-plugin' to '-flto' works for you?

For reference, with GCC 4.9 that uses linker plugin for LTO automatically, I
get the following diagnostics:

/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.2/../../../../x86_64-pc-linux-gnu/bin/ld:
error: /tmp/ccxxkwJ8.ltrans0.ltrans.o: requires dynamic R_X86_64_PC32 reloc
against '_dlstart_c' which may overflow at runtime; recompile with -fPIC
/tmp/ccxxkwJ8.ltrans0.ltrans.o(.text+0x12): error: undefined reference to
'_dlstart_c'

Not saying the patch can't go in -- just want to make sure everyone on the
same page regarding the origin of the problem and GCC LTO capabilities.

Alexander

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Alexander Monakov]

On Tue, 28 Apr 2015, Alexander Monakov wrote:
> For reference, with GCC 4.9 that uses linker plugin for LTO automatically, I
> get the following diagnostics:
> 
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.2/../../../../x86_64-pc-linux-gnu/bin/ld:
> error: /tmp/ccxxkwJ8.ltrans0.ltrans.o: requires dynamic R_X86_64_PC32 reloc
> against '_dlstart_c' which may overflow at runtime; recompile with -fPIC
> /tmp/ccxxkwJ8.ltrans0.ltrans.o(.text+0x12): error: undefined reference to
> '_dlstart_c'

Hm, I've just noticed Andre was using gcc 4.9.2 as well.  I get the same
diagnostics with BFD linker -- the diagnostics above are produced when using
the Gold linker.

Rich, do gold diagnostics help to see what musl might be doing to confuse
symbols dependencies resolution?

Alexander

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Rich Felker]

On Tue, Apr 28, 2015 at 11:35:18AM +0300, Alexander Monakov wrote:
> Hello,
> 
> Sorry for not joining the discussion earlier.
> 
> Andre, can you specify your GCC and Binutils version?  The reason I ask, with
> modern toolchain you shouldn't be seeing the error you reported.  The fact
> that _dlstart_c function is used from assembly should have been communicated
> from the linker to the compiler via the "linker plugin".  If linker plugin was
> not used, that would explan the problem.
> 
> Can you also check if adding '-fuse-linker-plugin' to '-flto' works for you?
> 
> For reference, with GCC 4.9 that uses linker plugin for LTO automatically, I
> get the following diagnostics:
> 
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.2/../../../../x86_64-pc-linux-gnu/bin/ld:
> error: /tmp/ccxxkwJ8.ltrans0.ltrans.o: requires dynamic R_X86_64_PC32 reloc
> against '_dlstart_c' which may overflow at runtime; recompile with -fPIC
> /tmp/ccxxkwJ8.ltrans0.ltrans.o(.text+0x12): error: undefined reference to
> '_dlstart_c'
> 
> Not saying the patch can't go in -- just want to make sure everyone on the
> same page regarding the origin of the problem and GCC LTO capabilities.

This seems to be a common problem then. I helped someone on #gcc with
almost the exact same issue doing freestanding work making a
kernel/bare-metal app using LTO a week or so ago. I'm not sure the
linker plugin can solve the problem since it seems to happen for
symbol references *within* a single translation unit (or a combined .o
file produced by ld -r, as in the case of the person on #gcc) which
the linker plugin probably does not track.

Even if the problem is missing linker plugin though, I think we want
to avoid LTO on these files. It's likely to be very risky since the
code is running in a situation where no function calls, global data
accesses, or symbolic references are possible. Here we really are
asking the compiler to produce asm for us, rather than asking it to
produce an optimized way to get an abstract job done.

Rich

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Khem Raj]

On Tue, Apr 28, 2015 at 6:43 AM, Rich Felker <dalias@libc.org> wrote:
> On Tue, Apr 28, 2015 at 11:35:18AM +0300, Alexander Monakov wrote:
>> Hello,
>>
>> Sorry for not joining the discussion earlier.
>>
>> Andre, can you specify your GCC and Binutils version?  The reason I ask, with
>> modern toolchain you shouldn't be seeing the error you reported.  The fact
>> that _dlstart_c function is used from assembly should have been communicated
>> from the linker to the compiler via the "linker plugin".  If linker plugin was
>> not used, that would explan the problem.
>>
>> Can you also check if adding '-fuse-linker-plugin' to '-flto' works for you?
>>
>> For reference, with GCC 4.9 that uses linker plugin for LTO automatically, I
>> get the following diagnostics:
>>
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.2/../../../../x86_64-pc-linux-gnu/bin/ld:
>> error: /tmp/ccxxkwJ8.ltrans0.ltrans.o: requires dynamic R_X86_64_PC32 reloc
>> against '_dlstart_c' which may overflow at runtime; recompile with -fPIC
>> /tmp/ccxxkwJ8.ltrans0.ltrans.o(.text+0x12): error: undefined reference to
>> '_dlstart_c'
>>
>> Not saying the patch can't go in -- just want to make sure everyone on the
>> same page regarding the origin of the problem and GCC LTO capabilities.
>

 with -flto you have to make sure that you use a version of binutils
that supports gcc's liblto_plugin. Since version 4.9 gcc produces slim
object files that only contain the intermediate representation. In
order to handle archives of these objects you have to use the gcc
wrappers: gcc-ar, gcc-nm and gcc-ranlib

> This seems to be a common problem then. I helped someone on #gcc with
> almost the exact same issue doing freestanding work making a
> kernel/bare-metal app using LTO a week or so ago. I'm not sure the
> linker plugin can solve the problem since it seems to happen for
> symbol references *within* a single translation unit (or a combined .o
> file produced by ld -r, as in the case of the person on #gcc) which
> the linker plugin probably does not track.
>
> Even if the problem is missing linker plugin though, I think we want
> to avoid LTO on these files. It's likely to be very risky since the
> code is running in a situation where no function calls, global data
> accesses, or symbolic references are possible. Here we really are
> asking the compiler to produce asm for us, rather than asking it to
> produce an optimized way to get an abstract job done.
>
> Rich

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Alexander Monakov]

> This seems to be a common problem then. I helped someone on #gcc with
> almost the exact same issue doing freestanding work making a
> kernel/bare-metal app using LTO a week or so ago. I'm not sure the
> linker plugin can solve the problem since it seems to happen for
> symbol references *within* a single translation unit (or a combined .o
> file produced by ld -r, as in the case of the person on #gcc) which
> the linker plugin probably does not track.

I think I see the root cause now.  The reference to _dlstart_c is from a GCC
asm statement (toplevel in this case), not from a separate assembly file.  The
toplevel asm is part of LTO "bytecode", so when linker symbol resolution runs,
it did not yet have a chance to see the reference from the asm, and GCC is
oblivious to the fact (it would need to parse the asm to notice).

If the person on #gcc with ld -r was using gcc asm statements as well, that
would explain their problem too.

So moving the assembly into its own .s file should avoid the issue; adding
__attribute__((used)) to _dlstart_c fixes it too; but even then ...

> Even if the problem is missing linker plugin though, I think we want
> to avoid LTO on these files. It's likely to be very risky since the
> code is running in a situation where no function calls, global data
> accesses, or symbolic references are possible. Here we really are
> asking the compiler to produce asm for us, rather than asking it to
> produce an optimized way to get an abstract job done.

acknowledged.  Your call :)

Thanks.
Alexander

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Alexander Monakov]

> If the person on #gcc with ld -r was using gcc asm statements as well, that
> would explain their problem too.

Let me add that there's another class of problems with toplevel asms and LTO:
as a result of LTO partitioning, toplevel asm may be moved into a different
partition from the symbol it's referencing, breaking the build (when toplevel
asm is referencing a static function).  GCC bug report for that issue is here:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=57703

Alexander

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Rich Felker]

On Tue, Apr 28, 2015 at 09:50:59PM +0300, Alexander Monakov wrote:
> > If the person on #gcc with ld -r was using gcc asm statements as well, that
> > would explain their problem too.
> 
> Let me add that there's another class of problems with toplevel asms and LTO:
> as a result of LTO partitioning, toplevel asm may be moved into a different
> partition from the symbol it's referencing, breaking the build (when toplevel
> asm is referencing a static function).  GCC bug report for that issue is here:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=57703

I have an interesting solution to this problem: get rid of the
top-level asm and instead put it inside the function with 2 additional
lines:

- At the top, "jmp 9f" or equivalent.
- At the bottom, "9:".

Now the asm can't be separated from the function because it's part of
it. Would that work?

Rich

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Alexander Monakov]

> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=57703
> 
> I have an interesting solution to this problem: get rid of the
> top-level asm and instead put it inside the function with 2 additional
> lines:
> 
> - At the top, "jmp 9f" or equivalent.
> - At the bottom, "9:".
> 
> Now the asm can't be separated from the function because it's part of
> it. Would that work?

I think it would avoid the LTO partitioning issue, but it would also be a step
back in terms of solving their original problem, which was to avoid duplicates
of syscall entry sequence being potentially emitted.  As another GCC
discussion mentions, they started using toplevel asm there after they found
they couldn't instruct Clang not do duplicate that function, unlike they could,
with function attributes, tame GCC.

Alexander

Re: [PATCH] force LTO to be disabled when compiling dlstart.lo

[Rich Felker]

On Tue, Apr 28, 2015 at 10:23:37PM +0300, Alexander Monakov wrote:
> > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=57703
> > 
> > I have an interesting solution to this problem: get rid of the
> > top-level asm and instead put it inside the function with 2 additional
> > lines:
> > 
> > - At the top, "jmp 9f" or equivalent.
> > - At the bottom, "9:".
> > 
> > Now the asm can't be separated from the function because it's part of
> > it. Would that work?
> 
> I think it would avoid the LTO partitioning issue, but it would also be a step
> back in terms of solving their original problem, which was to avoid duplicates
> of syscall entry sequence being potentially emitted.

Syscall entry sequences? I don't follow. Even if you meant "ELF entry
points" which sounds more plausible I'm still not sure how duplicates
would be emitted. I guess the concern is that if it's in a function
the asm block, even if it's volatile, could be emitted in more than
one place (e.g. a ptr-is-aligned branch and a ptr-is-misaligned
branch); it just has to execute the right number of times. So yes, if
this is what you mean, then I agree my solution is problematic and
probably best avoided.

> As another GCC
> discussion mentions, they started using toplevel asm there after they found
> they couldn't instruct Clang not do duplicate that function, unlike they could,
> with function attributes, tame GCC.

Interesting.

Rich