[musl] [PATCH] force -fno-lto for CRT files

[musl] [PATCH] force -fno-lto for CRT files

[Slava Barinov]

If the library is built using -flto flag the symbol only referenced in inline
assembly is considered unused and therefore is removed.

Zig compiler has the same issue: https://github.com/ziglang/zig/issues/10364
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 3ad88b35..709c5860 100644
--- a/Makefile
+++ b/Makefile
@@ -127,7 +127,7 @@ NOSSP_OBJS = $(CRT_OBJS) $(LDSO_OBJS) $(filter \
 	, $(LIBC_OBJS))
 $(NOSSP_OBJS) $(NOSSP_OBJS:%.o=%.lo): CFLAGS_ALL += $(CFLAGS_NOSSP)
 
-$(CRT_OBJS): CFLAGS_ALL += -DCRT
+$(CRT_OBJS): CFLAGS_ALL += -DCRT -fno-lto
 
 $(LOBJS) $(LDSO_OBJS): CFLAGS_ALL += -fPIC
 
-- 
2.49.0

Re: [musl] [PATCH] force -fno-lto for CRT files

[Szabolcs Nagy]

* Slava Barinov <rayslava@gmail.com> [2025-04-26 00:38:18 +0900]:
> If the library is built using -flto flag the symbol only referenced in inline
> assembly is considered unused and therefore is removed.
> 
> Zig compiler has the same issue: https://github.com/ziglang/zig/issues/10364

cflags like -fno-lto require configure check, see CFLAGS_NOSSP

> ---
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 3ad88b35..709c5860 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -127,7 +127,7 @@ NOSSP_OBJS = $(CRT_OBJS) $(LDSO_OBJS) $(filter \
>  	, $(LIBC_OBJS))
>  $(NOSSP_OBJS) $(NOSSP_OBJS:%.o=%.lo): CFLAGS_ALL += $(CFLAGS_NOSSP)
>  
> -$(CRT_OBJS): CFLAGS_ALL += -DCRT
> +$(CRT_OBJS): CFLAGS_ALL += -DCRT -fno-lto
>  
>  $(LOBJS) $(LDSO_OBJS): CFLAGS_ALL += -fPIC
>  
> -- 
> 2.49.0

Re: [musl] [PATCH] force -fno-lto for CRT files

[Slava Barinov]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

Hello,

Okay, I've rechecked the build and it seems that the check should be applied to
exactly the same objects as NO_SSP flags.

Please find the updated patch.

Best Regards,
Slava Barinov.

On Sat, Apr 26, 2025 at 3:28 AM Szabolcs Nagy <nsz@port70.net> wrote:
>
> * Slava Barinov <rayslava@gmail.com> [2025-04-26 00:38:18 +0900]:
> > If the library is built using -flto flag the symbol only referenced in inline
> > assembly is considered unused and therefore is removed.
> >
> > Zig compiler has the same issue: https://github.com/ziglang/zig/issues/10364
>
> cflags like -fno-lto require configure check, see CFLAGS_NOSSP
>
> > ---
> >  Makefile | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/Makefile b/Makefile
> > index 3ad88b35..709c5860 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -127,7 +127,7 @@ NOSSP_OBJS = $(CRT_OBJS) $(LDSO_OBJS) $(filter \
> >       , $(LIBC_OBJS))
> >  $(NOSSP_OBJS) $(NOSSP_OBJS:%.o=%.lo): CFLAGS_ALL += $(CFLAGS_NOSSP)
> >
> > -$(CRT_OBJS): CFLAGS_ALL += -DCRT
> > +$(CRT_OBJS): CFLAGS_ALL += -DCRT -fno-lto
> >
> >  $(LOBJS) $(LDSO_OBJS): CFLAGS_ALL += -fPIC
> >
> > --
> > 2.49.0

[-- Attachment #2: 0001-force-fno-lto-for-CRT-files.patch --]
[-- Type: text/x-patch, Size: 1860 bytes --]

From d6b93c6cdbcdfb5d0322904ae3c943911e0f171b Mon Sep 17 00:00:00 2001
From: Slava Barinov <rayslava@gmail.com>
Date: Thu, 24 Apr 2025 10:40:26 +0900
Subject: [PATCH] force -fno-lto for CRT files

If the library is built using -flto flag the symbol only referenced in inline
assembly is considered unused and therefore is removed.
---
 Makefile  |  3 +++
 configure | 10 ++++++++++
 2 files changed, 13 insertions(+)

diff --git a/Makefile b/Makefile
index 3ad88b35..27d70c3c 100644
--- a/Makefile
+++ b/Makefile
@@ -127,6 +127,9 @@ NOSSP_OBJS = $(CRT_OBJS) $(LDSO_OBJS) $(filter \
 	, $(LIBC_OBJS))
 $(NOSSP_OBJS) $(NOSSP_OBJS:%.o=%.lo): CFLAGS_ALL += $(CFLAGS_NOSSP)
 
+NOLTO_OBJS = $(NOSSP_OBJS)
+$(NOLTO_OBJS) $(NOLTO_OBJS:%.o=%.lo): CFLAGS_ALL += $(CFLAGS_NOLTO)
+
 $(CRT_OBJS): CFLAGS_ALL += -DCRT
 
 $(LOBJS) $(LDSO_OBJS): CFLAGS_ALL += -fPIC
diff --git a/configure b/configure
index bc9fbe48..12628cb8 100755
--- a/configure
+++ b/configure
@@ -119,6 +119,7 @@ CFLAGS_C99FSE=
 CFLAGS_AUTO=
 CFLAGS_MEMOPS=
 CFLAGS_NOSSP=
+CFLAGS_NOLTO=
 CFLAGS_TRY=
 LDFLAGS_AUTO=
 LDFLAGS_TRY=
@@ -398,6 +399,14 @@ tryflag CFLAGS_C99FSE -Wa,--noexecstack
 #
 tryflag CFLAGS_NOSSP -fno-stack-protector
 
+#
+# Check for options to disable link-time optimization, which needs
+# to be disabled for the same translation units since it affects
+# symbols set. If not found, this is not an error; we assume the
+# toolchain does not do lto.
+#
+tryflag CFLAGS_NOLTO -fno-lto
+
 #
 # Check for options that may be needed to prevent the compiler from
 # generating self-referential versions of memcpy,, memmove, memcmp,
@@ -820,6 +829,7 @@ CFLAGS_AUTO = $CFLAGS_AUTO
 CFLAGS_C99FSE = $CFLAGS_C99FSE
 CFLAGS_MEMOPS = $CFLAGS_MEMOPS
 CFLAGS_NOSSP = $CFLAGS_NOSSP
+CFLAGS_NOLTO = $CFLAGS_NOLTO
 CPPFLAGS = $CPPFLAGS
 LDFLAGS = $LDFLAGS
 LDFLAGS_AUTO = $LDFLAGS_AUTO
-- 
2.44.1

Re: [musl] [PATCH] force -fno-lto for CRT files

[Andy Caldwell]

[-- Attachment #1: Type: text/plain, Size: 1842 bytes --]

Hey,

An alternative approach would be to mark the function as `used` which the
attached patch does.  Sadly this attribute appears to maybe be a GNU-ism (also
supported in all versions of clang I could find) so maybe it would need a
feature check too to handle other compilers, but this feels like the correct
fix - telling the compiler that the symbol is used rather than disabling
compiler features that misbehave because the compiler can't see the indirect
calls.

Andy


On Mon, May 12, 2025 at 3:14 AM Slava Barinov <rayslava@gmail.com> wrote:
>
> Hello,
>
> Okay, I've rechecked the build and it seems that the check should be applied to
> exactly the same objects as NO_SSP flags.
>
> Please find the updated patch.
>
> Best Regards,
> Slava Barinov.
>
> On Sat, Apr 26, 2025 at 3:28 AM Szabolcs Nagy <nsz@port70.net> wrote:
> >
> > * Slava Barinov <rayslava@gmail.com> [2025-04-26 00:38:18 +0900]:
> > > If the library is built using -flto flag the symbol only referenced in inline
> > > assembly is considered unused and therefore is removed.
> > >
> > > Zig compiler has the same issue: https://github.com/ziglang/zig/issues/10364
> >
> > cflags like -fno-lto require configure check, see CFLAGS_NOSSP
> >
> > > ---
> > >  Makefile | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/Makefile b/Makefile
> > > index 3ad88b35..709c5860 100644
> > > --- a/Makefile
> > > +++ b/Makefile
> > > @@ -127,7 +127,7 @@ NOSSP_OBJS = $(CRT_OBJS) $(LDSO_OBJS) $(filter \
> > >       , $(LIBC_OBJS))
> > >  $(NOSSP_OBJS) $(NOSSP_OBJS:%.o=%.lo): CFLAGS_ALL += $(CFLAGS_NOSSP)
> > >
> > > -$(CRT_OBJS): CFLAGS_ALL += -DCRT
> > > +$(CRT_OBJS): CFLAGS_ALL += -DCRT -fno-lto
> > >
> > >  $(LOBJS) $(LDSO_OBJS): CFLAGS_ALL += -fPIC
> > >
> > > --
> > > 2.49.0

[-- Attachment #2: 0001-Mark-__dls2-as-__attribute__-used.patch --]
[-- Type: application/octet-stream, Size: 1119 bytes --]

From 988328716d269005ebb55095d74576bf88f5154f Mon Sep 17 00:00:00 2001
From: Andy Caldwell <andy.m.caldwell@googlemail.com>
Date: Mon, 12 May 2025 16:01:52 +0100
Subject: [PATCH] Mark `__dls2` as `__attribute__((used))`

This function is only referenced indirectly (though an explicit symbol
lookup on inline assembly) which causes LTO to strip it, causing linker
errors.  The `used` attribute tells the compiler (including the LTO
phase) that the function is referenced, even if the compiler can't see
that, preventing the function getting dropped.
---
 ldso/dynlink.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ldso/dynlink.c b/ldso/dynlink.c
index 715948f4..ab78274e 100644
--- a/ldso/dynlink.c
+++ b/ldso/dynlink.c
@@ -1704,7 +1704,7 @@ static void install_new_tls(void)
  * symbols. Its job is to perform symbolic relocations on the dynamic
  * linker itself, but some of the relocations performed may need to be
  * replaced later due to copy relocations in the main program. */
-
+__attribute__((used))
 hidden void __dls2(unsigned char *base, size_t *sp)
 {
 	size_t *auxv;
-- 
2.49.0

Re: [musl] [PATCH] force -fno-lto for CRT files

[Rich Felker]

On Mon, May 12, 2025 at 04:15:34PM +0100, Andy Caldwell wrote:
> Hey,
> 
> An alternative approach would be to mark the function as `used` which the
> attached patch does.  Sadly this attribute appears to maybe be a GNU-ism (also
> supported in all versions of clang I could find) so maybe it would need a
> feature check too to handle other compilers, but this feels like the correct
> fix - telling the compiler that the symbol is used rather than disabling
> compiler features that misbehave because the compiler can't see the indirect
> calls.
> 
> 
> From 988328716d269005ebb55095d74576bf88f5154f Mon Sep 17 00:00:00 2001
> From: Andy Caldwell <andy.m.caldwell@googlemail.com>
> Date: Mon, 12 May 2025 16:01:52 +0100
> Subject: [PATCH] Mark `__dls2` as `__attribute__((used))`
> 
> This function is only referenced indirectly (though an explicit symbol
> lookup on inline assembly) which causes LTO to strip it, causing linker
> errors.  The `used` attribute tells the compiler (including the LTO
> phase) that the function is referenced, even if the compiler can't see
> that, preventing the function getting dropped.
> ---
>  ldso/dynlink.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/ldso/dynlink.c b/ldso/dynlink.c
> index 715948f4..ab78274e 100644
> --- a/ldso/dynlink.c
> +++ b/ldso/dynlink.c
> @@ -1704,7 +1704,7 @@ static void install_new_tls(void)
>   * symbols. Its job is to perform symbolic relocations on the dynamic
>   * linker itself, but some of the relocations performed may need to be
>   * replaced later due to copy relocations in the main program. */
> -
> +__attribute__((used))
>  hidden void __dls2(unsigned char *base, size_t *sp)
>  {
>  	size_t *auxv;
> -- 
> 2.49.0

Is that even the only problem? My understanding is that this bug in
LTO affects all the crt start files as well, and in those it's more
explicit: it's ignoring a top-level asm root not just a reference via
inline asm inside a function.

On top of that, LTO-type transformations are wrong/unsafe for code
that executes prior to relocations and passing control to the entry
point, since at least in theory they could lift things without side
effects to somehow run before relocations or other initialization they
depend upon has happened. In some sense, this code runs in a separate,
lower-level execution environment operating *on* the memory of the
progam that will run rather than *as part of* the program. So I think
it makes sense to suppress LTO.

I kinda don't like introducing a second parallel set of "NOLTO" rules
that are identical to the "NOSSP" ones though, when the motivation for
both (that the code runs prior to the program/runtime being ready) is
the same. Maybe we could rename "NOSSP" to something more descriptive
and use the same CFLAGS & makefile rules for both? OTOH, I'd kinda
rather not change/remove CFLAGS_NOSSP in case it breaks folks' custom
config.mak's.. but maybe I'm the only one with one of those anyway.

Rich

Re: [musl] [PATCH] force -fno-lto for CRT files

[Andy Caldwell]

On Mon, May 12, 2025 at 6:47 PM Rich Felker <dalias@libc.org> wrote:
>
> On Mon, May 12, 2025 at 04:15:34PM +0100, Andy Caldwell wrote:
> > Hey,
> >
> > An alternative approach would be to mark the function as `used` which the
> > attached patch does.  Sadly this attribute appears to maybe be a GNU-ism (also
> > supported in all versions of clang I could find) so maybe it would need a
> > feature check too to handle other compilers, but this feels like the correct
> > fix - telling the compiler that the symbol is used rather than disabling
> > compiler features that misbehave because the compiler can't see the indirect
> > calls.
> >
> >
> > From 988328716d269005ebb55095d74576bf88f5154f Mon Sep 17 00:00:00 2001
> > From: Andy Caldwell <andy.m.caldwell@googlemail.com>
> > Date: Mon, 12 May 2025 16:01:52 +0100
> > Subject: [PATCH] Mark `__dls2` as `__attribute__((used))`
> >
> > This function is only referenced indirectly (though an explicit symbol
> > lookup on inline assembly) which causes LTO to strip it, causing linker
> > errors.  The `used` attribute tells the compiler (including the LTO
> > phase) that the function is referenced, even if the compiler can't see
> > that, preventing the function getting dropped.
> > ---
> >  ldso/dynlink.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/ldso/dynlink.c b/ldso/dynlink.c
> > index 715948f4..ab78274e 100644
> > --- a/ldso/dynlink.c
> > +++ b/ldso/dynlink.c
> > @@ -1704,7 +1704,7 @@ static void install_new_tls(void)
> >   * symbols. Its job is to perform symbolic relocations on the dynamic
> >   * linker itself, but some of the relocations performed may need to be
> >   * replaced later due to copy relocations in the main program. */
> > -
> > +__attribute__((used))
> >  hidden void __dls2(unsigned char *base, size_t *sp)
> >  {
> >       size_t *auxv;
> > --
> > 2.49.0
>
> Is that even the only problem? My understanding is that this bug in
> LTO affects all the crt start files as well, and in those it's more
> explicit: it's ignoring a top-level asm root not just a reference via
> inline asm inside a function.

If the goal is to build `libc.so` with artifact-wide optimisations enabled,
then this is the only change I needed to get that working (including
in its `ld-musl` and `ldd` modes).

Note that naively passing `-flto` in `CFLAGS also causes `rcrti.o` and
`Scrti.o` to be LLVM bitcode rather than object files (since they don't
get re-compiled after their initial LTO-compatible compilation) so at
least _some_ amount of filtering makes sense here (though see below).

> On top of that, LTO-type transformations are wrong/unsafe for code
> that executes prior to relocations and passing control to the entry
> point, since at least in theory they could lift things without side
> effects to somehow run before relocations or other initialization they
> depend upon has happened. In some sense, this code runs in a separate,
> lower-level execution environment operating *on* the memory of the
> program that will run rather than *as part of* the program. So I think
> it makes sense to suppress LTO.

Is this true?  My understanding is that LTO optimizations follow the same
rules as compiler optimizations (this specific case feels special but it's just
dead-code-elimination that the per-unit compilations can't do since they
can't see the whole binary).  Currently start-files are compiled with `-O2`
so any problematic lifting etc could already happen.

Out of perverse interest, I compiled `libc.a`, `Scrti.o` and `rcrti.o`
with `-flto`
and built some dynamic and static(-pie) binaries with `-flto` which all just
worked out of the box (the main complexity was convincing `musl-clang`
to use `lld` or `gold`).  If anyone wants to explore this further,
`rcrti.o` also
needs another `((used))` on it's version of `__dls2` for exactly the same
reason `dynlink` needs one.

> I kinda don't like introducing a second parallel set of "NOLTO" rules
> that are identical to the "NOSSP" ones though, when the motivation for
> both (that the code runs prior to the program/runtime being ready) is
> the same. Maybe we could rename "NOSSP" to something more descriptive
> and use the same CFLAGS & makefile rules for both? OTOH, I'd kinda
> rather not change/remove CFLAGS_NOSSP in case it breaks folks' custom
> config.mak's.. but maybe I'm the only one with one of those anyway.
>
> Rich

Re: [musl] [PATCH] force -fno-lto for CRT files

[Thorsten Glaser]

On Mon, 12 May 2025, Andy Caldwell wrote:
>On Mon, May 12, 2025 at 6:47 PM Rich Felker <dalias@libc.org> wrote:

>> On top of that, LTO-type transformations are wrong/unsafe for code
>> that executes prior to relocations and passing control to the entry
>> point, since at least in theory they could lift things without side
>> effects to somehow run before relocations or other initialization they
>> depend upon has happened. In some sense, this code runs in a separate,
>> lower-level execution environment operating *on* the memory of the
>> program that will run rather than *as part of* the program. So I think
>> it makes sense to suppress LTO.

ACK.

>Is this true?  My understanding is that LTO optimizations follow the same
>rules as compiler optimizations (this specific case feels special but it's just
>dead-code-elimination that the per-unit compilations can't do since they
>can't see the whole binary).  Currently start-files are compiled with `-O2`
>so any problematic lifting etc could already happen.

Only within the unit, i.e. the .o file, not across execution contexts
(pre-program vs. program itself).

LTO (at least in GCC) is also subtly buggy (and the bugreports get
mostly ignored).

>Out of perverse interest, I compiled `libc.a`, `Scrti.o` and `rcrti.o`
>with `-flto` and built some dynamic and static(-pie) binaries with
>`-flto` which all just worked out of the box (the main complexity was

It worked today, in your specific setup, with the toolchain you use,
but there is no guarantee it will continue working.

While whole-program optimisation sounds nice at first, the compiler
will assume that normal C rules for applications apply; the libc, in
application context already, isn’t part of the application but “the
implementation” and therefore allowed to do things application code
isn’t by the standard, and the startup files add initialisation context
to the mix. This will break subtly over time.

>> I kinda don't like introducing a second parallel set of "NOLTO" rules
>> that are identical to the "NOSSP" ones though, when the motivation for

Yes, that looked awful.

Keep CFLAGS_NOSSP and do:

CFLAGS_CSU = $(CFLAGS_NOSSP) $(CFLAGS_NOLTO)

Then use CFLAGS_CSU for the C startup objects.

(At least this is what I’d probably do.)

bye,
//mirabilos
-- 
<igli> exceptions: a truly awful implementation of quite a nice idea.
<igli> just about the worst way you could do something like that, afaic.
<igli> it's like anti-design.  <mirabilos> that too… may I quote you on that?
<igli> sure, tho i doubt anyone will listen ;)