[PATCH v3 1/3] build: overhaul wrapper script system for multiple wrapper support

[PATCH v3 1/3] build: overhaul wrapper script system for multiple wrapper support

[Shiz]

this overhauls part of the build system in order to support multiple
toolchain wrapper scripts, as opposed to solely the musl-gcc wrapper as
before. it thereby replaces --enable-gcc-wrapper with --enable-wrapper=...,
which has the options 'auto' (the default, detect whether to use wrappers),
'all' (build and install all wrappers), 'no' (don't build any) and finally
the options named after the individual compiler scripts (currently only
'gcc' is available) to build and install only that wrapper.
the old --enable-gcc-wrapper is removed from --help, but still available.

it also modifies the wrappers to use the C compiler specified to the build
system as 'inner' compiler, when applicable. as wrapper detection works by
probing this compiler, it may not work with any other.
---
 Makefile  |  4 +++-
 configure | 58 +++++++++++++++++++++++++++++++++++++---------------------
 2 files changed, 40 insertions(+), 22 deletions(-)

diff --git a/Makefile b/Makefile
index 2eb7b30..a1c1928 100644
--- a/Makefile
+++ b/Makefile
@@ -51,6 +51,8 @@ TOOL_LIBS = lib/musl-gcc.specs
 ALL_LIBS = $(CRT_LIBS) $(STATIC_LIBS) $(SHARED_LIBS) $(EMPTY_LIBS) $(TOOL_LIBS)
 ALL_TOOLS = tools/musl-gcc
 
+WRAPCC_GCC = gcc
+
 LDSO_PATHNAME = $(syslibdir)/ld-musl-$(ARCH)$(SUBARCH).so.1
 
 -include config.mak
@@ -155,7 +157,7 @@ lib/musl-gcc.specs: tools/musl-gcc.specs.sh config.mak
 	sh $< "$(includedir)" "$(libdir)" "$(LDSO_PATHNAME)" > $@
 
 tools/musl-gcc: config.mak
-	printf '#!/bin/sh\nexec "$${REALGCC:-gcc}" "$$@" -specs "%s/musl-gcc.specs"\n' "$(libdir)" > $@
+	printf '#!/bin/sh\nexec "$${REALGCC:-$(WRAPCC_GCC)}" "$$@" -specs "%s/musl-gcc.specs"\n' "$(libdir)" > $@
 	chmod +x $@
 
 $(DESTDIR)$(bindir)/%: tools/%
diff --git a/configure b/configure
index 7b29ae4..e3d4ab4 100755
--- a/configure
+++ b/configure
@@ -28,7 +28,7 @@ Optional features:
   --enable-debug          build with debugging information [disabled]
   --enable-warnings       build with recommended warnings flags [disabled]
   --enable-visibility     use global visibility options to optimize PIC [auto]
-  --enable-gcc-wrapper    build musl-gcc toolchain wrapper [auto]
+  --enable-wrapper=...    build given musl toolchain wrapper [auto]
   --disable-shared        inhibit building shared library [enabled]
   --disable-static        inhibit building static library [enabled]
 
@@ -123,6 +123,8 @@ bindir='$(exec_prefix)/bin'
 libdir='$(prefix)/lib'
 includedir='$(prefix)/include'
 syslibdir='/lib'
+tools=
+tool_libs=
 target=
 optimize=auto
 debug=no
@@ -131,6 +133,7 @@ visibility=auto
 shared=auto
 static=yes
 wrapper=auto
+gcc_wrapper=no
 
 for arg ; do
 case "$arg" in
@@ -154,7 +157,11 @@ case "$arg" in
 --disable-warnings|--enable-warnings=no) warnings=no ;;
 --enable-visibility|--enable-visibility=yes) visibility=yes ;;
 --disable-visibility|--enable-visibility=no) visibility=no ;;
---enable-gcc-wrapper|--enable-gcc-wrapper=yes) wrapper=yes ;;
+--enable-wrapper|--enable-wrapper=yes) wrapper=detect ;;
+--enable-wrapper=all) wrapper=yes ; gcc_wrapper=yes ;;
+--enable-wrapper=gcc) wrapper=yes ; gcc_wrapper=yes ;;
+--disable-wrapper|--enable-wrapper=no) wrapper=no ;;
+--enable-gcc-wrapper|--enable-gcc-wrapper=yes) wrapper=yes ; gcc_wrapper=yes ;;
 --disable-gcc-wrapper|--enable-gcc-wrapper=no) wrapper=no ;;
 --enable-*|--disable-*|--with-*|--without-*|--*dir=*|--build=*) ;;
 --host=*|--target=*) target=${arg#*=} ;;
@@ -207,7 +214,7 @@ exit 1
 fi
 
 #
-# Figure out options to force errors on unknown flags.
+# Figure out options to force errors on unknown flags
 #
 tryflag   CFLAGS_TRY  -Werror=unknown-warning-option
 tryflag   CFLAGS_TRY  -Werror=unused-command-line-argument
@@ -215,36 +222,44 @@ tryldflag LDFLAGS_TRY -Werror=unknown-warning-option
 tryldflag LDFLAGS_TRY -Werror=unused-command-line-argument
 
 #
-# Need to know if the compiler is gcc to decide whether to build the
-# musl-gcc wrapper, and for critical bug detection in some gcc versions.
+# Need to know if the compiler is gcc or clang to decide which toolchain
+# wrappers to build.
 #
-printf "checking whether compiler is gcc... "
-if fnmatch '*gcc\ version*' "$(LC_ALL=C $CC -v 2>&1)" ; then
-cc_is_gcc=yes
-else
-cc_is_gcc=no
+printf "checking for C compiler family... "
+cc_ver="$(LC_ALL=C $CC -v 2>&1)"
+cc_family=unknown
+if fnmatch '*gcc\ version*' "$cc_ver" ; then
+cc_family=gcc
 fi
-echo "$cc_is_gcc"
+echo "$cc_family"
 
 #
-# Only build musl-gcc wrapper if toolchain does not already target musl
+# Figure out toolchain wrapper to build
 #
+if test "$wrapper" = auto -o "$wrapper" = detect ; then
+printf "checking for toolchain wrapper to build... "
+if test "$cc_family" = gcc ; then
+gcc_wrapper=yes
 if test "$wrapper" = auto ; then
-printf "checking whether to build musl-gcc wrapper... "
-if test "$cc_is_gcc" = yes ; then
-wrapper=yes
 while read line ; do
-case "$line" in */ld-musl-*) wrapper=no ;; esac
+case "$line" in */ld-musl-*) gcc_wrapper=no ;; esac
 done <<EOF
 $($CC -dumpspecs)
 EOF
+fi
+test "$gcc_wrapper" = yes && echo "gcc"
 else
-wrapper=no
+echo "none"
+if test "$wrapper" = detect ; then
+fail "$0: could not find an appropriate toolchain wrapper"
+fi
 fi
-echo "$wrapper"
 fi
 
-
+if test "$gcc_wrapper" = yes ; then
+tools="$tools tools/musl-gcc"
+tool_libs="$tool_libs lib/musl-gcc.specs"
+fi
 
 #
 # Find the target architecture
@@ -580,11 +595,12 @@ LDFLAGS = $LDFLAGS_AUTO $LDFLAGS
 CROSS_COMPILE = $CROSS_COMPILE
 LIBCC = $LIBCC
 OPTIMIZE_GLOBS = $OPTIMIZE_GLOBS
+ALL_TOOLS = $tools
+TOOL_LIBS = $tool_libs
 EOF
 test "x$static" = xno && echo "STATIC_LIBS ="
 test "x$shared" = xno && echo "SHARED_LIBS ="
-test "x$wrapper" = xno && echo "ALL_TOOLS ="
-test "x$wrapper" = xno && echo "TOOL_LIBS ="
+test "x$cc_family" = xgcc && echo 'WRAPCC_GCC = $(CC)'
 exec 1>&3 3>&-
 
 printf "done\n"
-- 
2.3.6

[PATCH v3 2/3] build: fix musl-targeting toolchain test

[Shiz]

the old test was broken in that it would never fail on a toolchains built
without dynamic linking support, leading to the wrapper script possibly being
installed on compilers that do not support it. in addition, the new test is
portable across compilers: the old test only worked on GCC.

the new test works by testing whether the toolchain libc defines __GLIBC__:
most non-musl Linux libc's do define this for compatibility even when they
are not glibc, so this is a safe bet to check for musl. in addition, the
compiler runtime would need to have a somewhat glibc-compatible ABI in the
first place, so any non-glibc compatible libc's compiler runtime might not
work. it is safer to disable these cases by default and have the user enable
the wrappers manually there using --enable-wrapper if they certain it works.
---
 configure | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/configure b/configure
index e3d4ab4..b07375a 100755
--- a/configure
+++ b/configure
@@ -237,17 +237,16 @@ echo "$cc_family"
 # Figure out toolchain wrapper to build
 #
 if test "$wrapper" = auto -o "$wrapper" = detect ; then
+echo "#include <stdlib.h>" > "$tmpc"
+echo "#if ! __GLIBC__" >> "$tmpc"
+echo "#error no" >> "$tmpc"
+echo "#endif" >> "$tmpc"
 printf "checking for toolchain wrapper to build... "
-if test "$cc_family" = gcc ; then
+if test "$wrapper" = auto && ! $CC -c -o /dev/null "$tmpc" >/dev/null 2>&1 ; then
+echo "none"
+elif test "$cc_family" = gcc ; then
 gcc_wrapper=yes
-if test "$wrapper" = auto ; then
-while read line ; do
-case "$line" in */ld-musl-*) gcc_wrapper=no ;; esac
-done <<EOF
-$($CC -dumpspecs)
-EOF
-fi
-test "$gcc_wrapper" = yes && echo "gcc"
+echo "gcc"
 else
 echo "none"
 if test "$wrapper" = detect ; then
-- 
2.3.6

[PATCH v3 3/3] add musl-clang, a wrapper for system clang installs

[Shiz]

musl-clang allows the user to compile musl-powered programs using their
already existent clang install, without the need of a special cross compiler.
it achieves this by wrapping around both the system clang install and the
linker and passing them special flags to re-target musl at runtime.
it does only affect invocations done through the special musl-clang wrapper
script, so that the user setup remains fully intact otherwise.

the clang wrapper consists of the compiler frontend wrapper script,
musl-clang, and the linker wrapper script, ld.musl-clang.
musl-clang makes sure clang invokes ld.musl-clang to link objects; neither
script needs to be in PATH for the wrapper to work.
---
 .gitignore             |  2 ++
 Makefile               |  5 +++++
 configure              | 13 ++++++++++++-
 tools/ld.musl-clang.in | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 tools/musl-clang.in    | 36 ++++++++++++++++++++++++++++++++++++
 5 files changed, 104 insertions(+), 1 deletion(-)
 create mode 100644 tools/ld.musl-clang.in
 create mode 100644 tools/musl-clang.in

diff --git a/.gitignore b/.gitignore
index 070f549..c5d5c46 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,5 +7,7 @@ arch/*/bits/alltypes.h
 config.mak
 include/bits
 tools/musl-gcc
+tools/musl-clang
+tools/ld.musl-clang
 lib/musl-gcc.specs
 src/internal/version.h
diff --git a/Makefile b/Makefile
index a1c1928..5f74005 100644
--- a/Makefile
+++ b/Makefile
@@ -52,6 +52,7 @@ ALL_LIBS = $(CRT_LIBS) $(STATIC_LIBS) $(SHARED_LIBS) $(EMPTY_LIBS) $(TOOL_LIBS)
 ALL_TOOLS = tools/musl-gcc
 
 WRAPCC_GCC = gcc
+WRAPCC_CLANG = clang
 
 LDSO_PATHNAME = $(syslibdir)/ld-musl-$(ARCH)$(SUBARCH).so.1
 
@@ -160,6 +161,10 @@ tools/musl-gcc: config.mak
 	printf '#!/bin/sh\nexec "$${REALGCC:-$(WRAPCC_GCC)}" "$$@" -specs "%s/musl-gcc.specs"\n' "$(libdir)" > $@
 	chmod +x $@
 
+tools/%-clang: tools/%-clang.in config.mak
+	sed -e 's!@CC@!$(WRAPCC_CLANG)!g' -e 's!@PREFIX@!$(prefix)!g' -e 's!@INCDIR@!$(includedir)!g' -e 's!@LIBDIR@!$(libdir)!g' -e 's!@LDSO@!$(LDSO_PATHNAME)!g' $< > $@
+	chmod +x $@
+
 $(DESTDIR)$(bindir)/%: tools/%
 	$(INSTALL) -D $< $@
 
diff --git a/configure b/configure
index b07375a..a4c9236 100755
--- a/configure
+++ b/configure
@@ -134,6 +134,7 @@ shared=auto
 static=yes
 wrapper=auto
 gcc_wrapper=no
+clang_wrapper=no
 
 for arg ; do
 case "$arg" in
@@ -158,8 +159,9 @@ case "$arg" in
 --enable-visibility|--enable-visibility=yes) visibility=yes ;;
 --disable-visibility|--enable-visibility=no) visibility=no ;;
 --enable-wrapper|--enable-wrapper=yes) wrapper=detect ;;
---enable-wrapper=all) wrapper=yes ; gcc_wrapper=yes ;;
+--enable-wrapper=all) wrapper=yes ; gcc_wrapper=yes ; clang_wrapper=yes ;;
 --enable-wrapper=gcc) wrapper=yes ; gcc_wrapper=yes ;;
+--enable-wrapper=clang) wrapper=yes ; clang_wrapper=yes ;;
 --disable-wrapper|--enable-wrapper=no) wrapper=no ;;
 --enable-gcc-wrapper|--enable-gcc-wrapper=yes) wrapper=yes ; gcc_wrapper=yes ;;
 --disable-gcc-wrapper|--enable-gcc-wrapper=no) wrapper=no ;;
@@ -230,6 +232,8 @@ cc_ver="$(LC_ALL=C $CC -v 2>&1)"
 cc_family=unknown
 if fnmatch '*gcc\ version*' "$cc_ver" ; then
 cc_family=gcc
+elif fnmatch '*clang\ version*' "$cc_ver" ; then
+cc_family=clang
 fi
 echo "$cc_family"
 
@@ -247,6 +251,9 @@ echo "none"
 elif test "$cc_family" = gcc ; then
 gcc_wrapper=yes
 echo "gcc"
+elif test "$cc_family" = clang ; then
+clang_wrapper=yes
+echo "clang"
 else
 echo "none"
 if test "$wrapper" = detect ; then
@@ -259,6 +266,9 @@ if test "$gcc_wrapper" = yes ; then
 tools="$tools tools/musl-gcc"
 tool_libs="$tool_libs lib/musl-gcc.specs"
 fi
+if test "$clang_wrapper" = yes ; then
+tools="$tools tools/musl-clang tools/ld.musl-clang"
+fi
 
 #
 # Find the target architecture
@@ -600,6 +610,7 @@ EOF
 test "x$static" = xno && echo "STATIC_LIBS ="
 test "x$shared" = xno && echo "SHARED_LIBS ="
 test "x$cc_family" = xgcc && echo 'WRAPCC_GCC = $(CC)'
+test "x$cc_family" = xclang && echo 'WRAPCC_CLANG = $(CC)'
 exec 1>&3 3>&-
 
 printf "done\n"
diff --git a/tools/ld.musl-clang.in b/tools/ld.musl-clang.in
new file mode 100644
index 0000000..cac3c0b
--- /dev/null
+++ b/tools/ld.musl-clang.in
@@ -0,0 +1,49 @@
+#!/bin/sh
+cc="@CC@"
+libc_lib="@LIBDIR@"
+ldso="@LDSO@"
+cleared=
+shared=
+userlinkdir=
+userlink=
+
+for x ; do
+    test "$cleared" || set -- ; cleared=1
+
+    case "$x" in
+        -L-user-start)
+            userlinkdir=1
+            ;;
+        -L-user-end)
+            userlinkdir=
+            ;;
+        -L*)
+            test "$userlinkdir" && set -- "$@" "$x"
+            ;;
+        -l-user-start)
+            userlink=1
+            ;;
+        -l-user-end)
+            userlink=
+            ;;
+        -l*)
+            test "$userlink" && set -- "$@" "$x"
+            ;;
+        -shared)
+            shared=1
+            set -- "$@" -shared
+            ;;
+        -sysroot=*|--sysroot=*)
+            ;;
+        *)
+            set -- "$@" "$x"
+            ;;
+    esac
+done
+
+set -- "$@" -lc
+if ! test "$shared"; then
+    set -- "$libc_lib/Scrt1.o" "$libc_lib/crti.o" "$@" "$libc_lib/crtn.o"
+fi
+
+exec $($cc -print-prog-name=ld) -nostdlib "$@" -dynamic-linker "$ldso"
diff --git a/tools/musl-clang.in b/tools/musl-clang.in
new file mode 100644
index 0000000..d815081
--- /dev/null
+++ b/tools/musl-clang.in
@@ -0,0 +1,36 @@
+#!/bin/sh
+cc="@CC@"
+libc="@PREFIX@"
+libc_inc="@INCDIR@"
+libc_lib="@LIBDIR@"
+thisdir="`cd "$(dirname "$0")"; pwd`"
+
+# prevent clang from running the linker (and erroring) on no input.
+sflags=
+eflags=
+for x ; do
+    case "$x" in
+        -l*) input=1 ;;
+        *) input= ;;
+    esac
+    if test "$input" ; then
+        sflags="-l-user-start"
+        eflags="-l-user-end"
+        break
+    fi
+done
+
+exec $cc \
+    -B"$thisdir" \
+    -fuse-ld=musl-clang \
+    -rtlib=compiler-rt \
+    -nostdinc \
+    -nostartfiles \
+    --sysroot "$libc" \
+    -isystem "$libc_inc" \
+    -L-user-start \
+    $sflags \
+    "$@" \
+    $eflags \
+    -L"$libc_lib" \
+    -L-user-end
-- 
2.3.6

Re: [PATCH v3 2/3] build: fix musl-targeting toolchain test

[Isaac Dunham]

On Sun, Jun 28, 2015 at 11:08:20PM +0200, Shiz wrote:
> the old test was broken in that it would never fail on a toolchains built
> without dynamic linking support, leading to the wrapper script possibly being
> installed on compilers that do not support it. in addition, the new test is
> portable across compilers: the old test only worked on GCC.
> 
> the new test works by testing whether the toolchain libc defines __GLIBC__:
> most non-musl Linux libc's do define this for compatibility even when they
> are not glibc, so this is a safe bet to check for musl. in addition, the
> compiler runtime would need to have a somewhat glibc-compatible ABI in the
> first place, so any non-glibc compatible libc's compiler runtime might not
> work. it is safer to disable these cases by default and have the user enable
> the wrappers manually there using --enable-wrapper if they certain it works.

A long while back, someone reported successfully using a FreeBSD gcc
to build musl, then using that + musl-gcc to build programs that
worked using the Linux emulation layer on FreeBSD.
They submitted a patch with a BSD makefile at that point.

Thanks,
Isaac Dunham

Re: [PATCH v3 2/3] build: fix musl-targeting toolchain test

[Rich Felker]

On Mon, Jun 29, 2015 at 07:35:21AM -0700, Isaac Dunham wrote:
> On Sun, Jun 28, 2015 at 11:08:20PM +0200, Shiz wrote:
> > the old test was broken in that it would never fail on a toolchains built
> > without dynamic linking support, leading to the wrapper script possibly being
> > installed on compilers that do not support it. in addition, the new test is
> > portable across compilers: the old test only worked on GCC.
> > 
> > the new test works by testing whether the toolchain libc defines __GLIBC__:
> > most non-musl Linux libc's do define this for compatibility even when they
> > are not glibc, so this is a safe bet to check for musl. in addition, the
> > compiler runtime would need to have a somewhat glibc-compatible ABI in the
> > first place, so any non-glibc compatible libc's compiler runtime might not
> > work. it is safer to disable these cases by default and have the user enable
> > the wrappers manually there using --enable-wrapper if they certain it works.
> 
> A long while back, someone reported successfully using a FreeBSD gcc
> to build musl, then using that + musl-gcc to build programs that
> worked using the Linux emulation layer on FreeBSD.
> They submitted a patch with a BSD makefile at that point.

This is probably a sufficiently unusual setup to warrant manually
enabling the wrapper. I suspect additional work is needed to get the
linker to output the right OS type in the ELF header. Unless a FreeBSD
GCC toolchain wrapped by musl-gcc can automatically generate working
binaries with no further modification or options, I don't think it's
appropriate to auto-enable the wrapper there.

Rich

Re: [PATCH v3 1/3] build: overhaul wrapper script system for multiple wrapper support

[Rich Felker]

On Sun, Jun 28, 2015 at 11:08:19PM +0200, Shiz wrote:
> this overhauls part of the build system in order to support multiple
> toolchain wrapper scripts, as opposed to solely the musl-gcc wrapper as
> [...]

I've committed all 3 patches with the changes we discussed on IRC. It
Works For Me with clang on Alpine (albeit being a useless NOP since
the host clang is already musl-targeting) and doesn't break anything
with gcc.

If anyone gets around to trying this before release, let me know if
any issues come up and we can fix them.

Rich

Re: [PATCH v3 3/3] add musl-clang, a wrapper for system clang installs

[Matias A. Fonzo]

El Sun, 28 Jun 2015 23:08:21 +0200
Shiz <hi@shiz.me> escribió:
> musl-clang allows the user to compile musl-powered programs using
> their already existent clang install, without the need of a special
> cross compiler. it achieves this by wrapping around both the system
> clang install and the linker and passing them special flags to
> re-target musl at runtime. it does only affect invocations done
> through the special musl-clang wrapper script, so that the user setup
> remains fully intact otherwise.
> 
> the clang wrapper consists of the compiler frontend wrapper script,
> musl-clang, and the linker wrapper script, ld.musl-clang.
> musl-clang makes sure clang invokes ld.musl-clang to link objects;
> neither script needs to be in PATH for the wrapper to work.
> [..]
> +thisdir="`cd "$(dirname "$0")"; pwd`"
> [..]

A better version of "thisdir=" to prevent directories containing spaces
and weird characters is:

  thisdir="`cd "$(dirname -- "$0" 2> /dev/null)" ; printf %s "$PWD"`"