configure script for musl (?!)

configure script for musl (?!)

[Rich Felker]

OK before anybody flips out, no, there's no autoconf. :-)

Partly as an exercise in learning the details of how users/systems
integrators expect "./configure" to work, and partly in the interest
of simplifying build/installation of musl, I've been playing around
with writing a configure script. It's a flat shell script written in
portable (POSIX) sh, and depending on no external programs except the
compiler whose characteristics are being tested.

What does this buy us?

1. Automatic selection of ARCH as long as the compiler supports the
   GCC -dumpmachine option. This works even if $CC is set to a
   cross-compiler.

2. Automatic selection of compiler options that can fix standards
   conformance issues, like -ffloat-store or
   -fexcess-precision=standard. I believe the latter is default with
   -std=c99 for all compilers that support it, but testing for it
   allows us to fallback to (better than nothing) -ffloat-store on
   ones that don't. We could also add things like -frounding-math if
   they're helpful without worrying about breaking build on compilers
   that don't support them.

3. Automatic enabling of preferred warning options without breaking
   build on older compilers that don't support them all, or becoming
   spamming on newer compilers that need -Wno-* options to disable
   some bogus ones.

4. Automatic enabling of debloating options when supported/possible
   (like omitting DWARF2 stuff when debugging is disabled).

I'm going to start by committing some changes to Makefile with the
minimum invasiveness to support configure-like semantics in
config.mak. The key change needed is that we have to split up CFLAGS
so that overriding CFLAGS from the make command line does not remove
the essential options (like -std=c99, -ffreestanding,
-D_XOPEN_SOURCE=700, etc.) and break things horribly. I'm also making
some changes that make it easier to disable shared OR STATIC libs from
the build process if desired, and some general cleanup. My intent is
that these changes not break existing manually-written config.mak
usage at all.

After that, I'll post a draft configure script here, and if all goes
well, it can be committed and can serve as an alternative to the
manual configuration method (which will continue to work).

Rich

Re: configure script for musl (?!)

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 35 bytes --]

Here's the configure draft..

Rich

[-- Attachment #2: configure --]
[-- Type: text/plain, Size: 5235 bytes --]

#!/bin/sh

usage () {
cat <<EOF
Usage: $0 [OPTION]... [VAR=VALUE]... [BUILDTYPE]

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Installation directories:
  --prefix=PREFIX         main installation prefix [/usr/local/musl]
  --exec-prefix=EPREFIX   installation prefix for tools [/usr/local]

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --libdir=DIR            library files for the linker [PREFIX/lib]
  --includedir=DIR        include files for the C compiler [PREFIX/include]
  --syslibdir=DIR         location for the dynamic linker [/lib]

System types:
  --target=TARGET         configure to run on target TARGET [detected]
  --build=BUILD           same as --target

Optional features:
  --enable-debug          build libraries with debugging information
  --disable-shared        inhibit building shared library
  --disable-static        inhibit building static library

Some influential environment variables:
  CC                      C compiler command [guessed]
  CFLAGS                  C compiler flags [-O2 ...]

Use these variables to override the choices made by configure.

EOF
exit 0
}

# Helper functions

fnmatch () { eval "case \"\$2\" in $1) return 0 ;; *) return 1 ;; esac" ; }
cmdexists () { type "$1" >/dev/null 2>&1 ; }
trycc () { test -z "$CC" && cmdexists "$1" && CC=$1 ; }

setdir () {
if eval "test -z \"\${$1}\"" ; then eval "$1=\$2"
else eval "fnmatch '*/' \"\${$1}\"" && eval "$1=\${$1%/}" ; fi
}

tryflag () {
printf "checking whether %s %s works... " "$CC" "$2"
echo "typedef int x;" > "$tmpc"
if "$CC" "$2" -c -o /dev/null "$tmpc" 2>/dev/null ; then
printf "yes\n"
eval "$1=\"\${$1} \$2\""
eval "$1=\${$1# }"
else
printf "no\n"
fi
}



# Beginning of actual script

prefix=
exec_prefix=
bindir=
libdir=
includedir=
syslibdir=
build=
debug=
shared=
static=

for arg ; do
case "$arg" in
--help) usage ;;
--prefix=*) prefix=${arg#*=} ;;
--exec-prefix=*) exec_prefix=${arg#*=} ;;
--bindir=*) bindir=${arg#*=} ;;
--libdir=*) libdir=${arg#*=} ;;
--includedir=*) includedir=${arg#*=} ;;
--syslibdir=*) syslibdir=${arg#*=} ;;
--enable-shared|--enable-shared=yes) shared=yes ;;
--disable-shared|--enable-shared=no) shared=no ;;
--enable-static|--enable-static=yes) static=yes ;;
--disable-static|--enable-static=no) static=no ;;
--enable-debug|--enable-debug=yes) debug=yes ;;
--disable-debug|--enable-debug=no) debug=no ;;
--enable-*|--disable-*|--with-*|--without-*|--*dir=*|--host=*) ;;
--build=*|--target=*) build=${arg#*=} ;;
-* ) echo "$0: unknown option $arg" ;;
CC=*) CC=${arg#*=} ;;
CFLAGS=*) CFLAGS=${arg#*=} ;;
CPPFLAGS=*) CPPFLAGS=${arg#*=} ;;
LDFLAGS=*) LDFLAGS=${arg#*=} ;;
*) build=$arg ;;
esac
done

setdir prefix /usr/local/musl
setdir exec_prefix '/usr/local'
setdir bindir '$(exec_prefix)/bin'
setdir libdir '$(prefix)/lib'
setdir includedir '$(prefix)/include'
setdir syslibdir '/lib'

# Get a temp filename we can use
test -n "$TMPDIR" && tmpc="$TMPDIR/conf$$.c" || tmpc="/tmp/conf$$.c"
set -C
> "$tmpc" || {
echo "$0: cannot create temporary file $tmpc"
exit 1
}
set +C
trap 'rm "$tmpc"' EXIT

# Find a C compiler to use
printf "checking for C compiler... "
trycc gcc
trycc c99
trycc cc
printf "%s\n" "$CC"
test -n "$CC" || { echo "$0: cannot find a C compiler" ; exit 1 ; }

# Find the build architecture
printf "checking build system type... "
test -n "$build" || build=$("$CC" -dumpmachine 2>/dev/null) || build=unknown
printf "%s\n" "$build"

# Convert to just ARCH
case "$build" in
arm*) ARCH=arm ;;
i?86*) ARCH=i386 ;;
x86_64*) ARCH=x86_64 ;;
unknown) echo "$0: unable to detect built target; try $0 --build=..." ; exit 1 ;;
*) echo "$0: unknown or unsupported build target \"$build\"" ; exit 1 ;;
esac

# Try to get a conforming C99 freestanding environment
tryflag CFLAGS_C99FSE -std=c99
tryflag CFLAGS_C99FSE -nostdinc
tryflag CFLAGS_C99FSE -ffreestanding \
|| tryflag CFLAGS_C99FSE -fno-builtin
tryflag CFLAGS_C99FSE -fexcess-precision=standard \
|| tryflag CFLAGS_C99FSE -ffloat-store

# Default CFLAGS
test -z "$CFLAGS" && CFLAGS=-O2
test "x$debug" = xyes && CFLAGS="$CFLAGS -g"
tryflag CFLAGS -pipe

# If debugging is disabled, omit bloated DWARF2 unwind tables & frame ptr
if fnmatch '-g*|*\ -g*' "$CFLAGS" ; then :
else 
tryflag CFLAGS -fno-asynchronous-unwind-tables
tryflag CFLAGS -fomit-frame-pointer
fi

# Some optimization levels add bloated alignment that hurt performance
tryflag CFLAGS -falign-functions=1
tryflag CFLAGS -falign-labels=1
tryflag CFLAGS -falign-loops=1
tryflag CFLAGS -falign-jumps=1

if test "$ARCH" = "i386" ; then
fnmatch '-march=*|*\ -march=*' "$CFLAGS" || tryflag CFLAGS -march=i486
fnmatch '-mtune=*|*\ -mtune=*' "$CFLAGS" || tryflag CFLAGS -mtune=generic
fi

exec 1>config.mak

cat << EOF

ARCH = $ARCH

prefix = $prefix
exec_prefix = $exec_prefix
bindir = $bindir
libdir = $libdir
includedir = $includedir
syslibdir = $syslibdir

CC = $CC
CFLAGS = $CFLAGS
CFLAGS_C99FSE = $CFLAGS_C99FSE

CPPFLAGS = $CPPFLAGS
LDFLAGS = $LDFLAGS

EOF

test "x$static" = xno && echo "STATIC_LIBS ="
test "x$shared" = xno && echo "SHARED_LIBS ="

Re: configure script for musl (?!)

[Solar Designer]

Rich,

This approach (including not using autoconf specifically) makes sense to
me.  One general problem with it, though, is that errors are not
distinguished from actually and purposefully missing features.
Sometimes it is preferable for the build to fail when an expected system
feature is not present rather than for it to succeed with unexpectedly
altered functionality.  This is especially true if the non-detected
feature had security relevance.  On Owl, we have the configure-presets
script for this reason - to pre-populate autoconf configure's cache.

On Tue, May 01, 2012 at 07:39:26PM -0400, Rich Felker wrote:
> # Get a temp filename we can use
> test -n "$TMPDIR" && tmpc="$TMPDIR/conf$$.c" || tmpc="/tmp/conf$$.c"
> set -C
> > "$tmpc" || {
> echo "$0: cannot create temporary file $tmpc"
> exit 1
> }
> set +C
> trap 'rm "$tmpc"' EXIT

Is "set -C" sufficiently portable?  (I don't know.  I guess that it is
if you're using it here.)

Assuming that "set -C" works, another user is able to prevent the above
from working by pre-creating the file.  I understand that you can't just
use mktemp here, and implementing its full functionality right in your
shell script is not pretty.

Another issue is that if the script is running with a relaxed umask like
002, then another user (in the same group, if relevant) may not only
notice that musl is being compiled, but also inject their own C code
into the temporary file and maybe take over the musl-building user
account (depending on whether you run the compiled code or not, as well
as on the compiler's features and bugs available to be invoked from the
source).  Perhaps you need to set a safe umask like 077 in this script
before it creates the temporary file (or chmod the file right after it's
been created, before its first actual use - and exit if the chmod fails).

Rather than use $TMPDIR or /tmp, I think it'd be safer to place the file
in the same directory with the configure script or in the current
directory.  You're going to create config.mak in one of these
directories anyway - so just create the temporary file in that same
directory not to incur any extra risk (that you do by writing into a
second directory).

Alexander

Re: configure script for musl (?!)

[Rich Felker]

On Wed, May 02, 2012 at 06:00:04PM +0400, Solar Designer wrote:
> Rich,
> 
> This approach (including not using autoconf specifically) makes sense to
> me.  One general problem with it, though, is that errors are not
> distinguished from actually and purposefully missing features.
> Sometimes it is preferable for the build to fail when an expected system
> feature is not present rather than for it to succeed with unexpectedly
> altered functionality.  This is especially true if the non-detected
> feature had security relevance.  On Owl, we have the configure-presets
> script for this reason - to pre-populate autoconf configure's cache.

I don't think we have any situations like that. The only options that
are "essential" but might fail are GCC-specific options to turn the
compiler into a conforming C99 freestanding implementation. The only
reason I test them at all is to support hypothetical
non-GCC-compatible compilers, and I assume (1) they're already
sufficiently conforming, and (2) if the user is using a non-mainstream
compiler, they know best what they're doing.

As for the floating point ones, only gcc >= 4.5.x can really give
correct behavior on x86, but -ffloat-store is sufficient to be mostly
correct, and the difference is usually just very small rounding
errors. For the really important functions that need to be correctly
rounded, glibc doesn't even get them right anyway, and we have asm
that DOES get them right even if the compiler is broken when it comes
to compiling the portable (in theory; if compilers weren't broken) C
version.

> On Tue, May 01, 2012 at 07:39:26PM -0400, Rich Felker wrote:
> > # Get a temp filename we can use
> > test -n "$TMPDIR" && tmpc="$TMPDIR/conf$$.c" || tmpc="/tmp/conf$$.c"
> > set -C
> > > "$tmpc" || {
> > echo "$0: cannot create temporary file $tmpc"
> > exit 1
> > }
> > set +C
> > trap 'rm "$tmpc"' EXIT
> 
> Is "set -C" sufficiently portable?  (I don't know.  I guess that it is
> if you're using it here.)

It's POSIX. My policy for this and future configure scripts I write is
that you need a POSIX compatible shell to run configure. This is the
same as MPlayer and FFmpeg/libav policy and I remember they took some
heat for it back in the day from users of broken unices like Solaris,
but telling those users to add /usr/xpg4/bin to their path and invoke
configure with that version of the shell generally made them happy. I
suspect it's a non-issue in 2012, and especially for musl where
building on anything non-Linux-based (and thus anything without
[bd]?ash as the shell) is going to be a special cross-compiling case.

> Assuming that "set -C" works, another user is able to prevent the above
> from working by pre-creating the file.  I understand that you can't just
> use mktemp here, and implementing its full functionality right in your
> shell script is not pretty.

Yes, mktemp does not work because it does not let you specify a
suffix, and most compilers fail to recognize input files as C unless
they have a .c suffix. It has a dry run option just to generate the
filename (and let the caller handle creating it like we do now) but
unfortunately busybox mktemp does not know that option so it doesn't
seem at all safe/portable.

I've actually been working on the issue and updated it to use $$,
$PPID, and a retry counter, so random failures will be extremely rare.
If anyone thinks it still matters, I'll add $(date|cksum) too.

> Another issue is that if the script is running with a relaxed umask like
> 002, then another user (in the same group, if relevant) may not only
> notice that musl is being compiled, but also inject their own C code
> into the temporary file and maybe take over the musl-building user
> account (depending on whether you run the compiled code or not, as well
> as on the compiler's features and bugs available to be invoked from the
> source).  Perhaps you need to set a safe umask like 077 in this script
> before it creates the temporary file (or chmod the file right after it's
> been created, before its first actual use - and exit if the chmod fails).

Indeed, this seems like a good idea.

> Rather than use $TMPDIR or /tmp, I think it'd be safer to place the file
> in the same directory with the configure script or in the current
> directory.  You're going to create config.mak in one of these
> directories anyway - so just create the temporary file in that same
> directory not to incur any extra risk (that you do by writing into a
> second directory).

This seems reasonable.

Rich

Re: configure script for musl (?!)

[Solar Designer]

On Wed, May 02, 2012 at 10:31:25AM -0400, Rich Felker wrote:
> On Wed, May 02, 2012 at 06:00:04PM +0400, Solar Designer wrote:
> > Is "set -C" sufficiently portable?  (I don't know.  I guess that it is
> > if you're using it here.)
> 
> It's POSIX. My policy for this and future configure scripts I write is
> that you need a POSIX compatible shell to run configure. This is the
> same as MPlayer and FFmpeg/libav policy and I remember they took some
> heat for it back in the day from users of broken unices like Solaris,
> but telling those users to add /usr/xpg4/bin to their path and invoke
> configure with that version of the shell generally made them happy. I
> suspect it's a non-issue in 2012, and especially for musl where
> building on anything non-Linux-based (and thus anything without
> [bd]?ash as the shell) is going to be a special cross-compiling case.

I think the biggest problem here is in the failure mode.  In the draft
script you posted, if "set -C" fails the script will proceed to
potentially clobber a file via a (sym)link provided by another user, or
it may use a FIFO or a regular file pre-created by another user (and set
to e.g. mode 666), in which case the input to the compiler would be
under that other user's control (even if we're running with a safe umask).

> > Assuming that "set -C" works, another user is able to prevent the above
> > from working by pre-creating the file.  I understand that you can't just
> > use mktemp here, and implementing its full functionality right in your
> > shell script is not pretty.
> 
> Yes, mktemp does not work because it does not let you specify a
> suffix, and most compilers fail to recognize input files as C unless
> they have a .c suffix. It has a dry run option just to generate the
> filename (and let the caller handle creating it like we do now) but
> unfortunately busybox mktemp does not know that option so it doesn't
> seem at all safe/portable.
> 
> I've actually been working on the issue and updated it to use $$,
> $PPID, and a retry counter, so random failures will be extremely rare.
> If anyone thinks it still matters, I'll add $(date|cksum) too.

Oh, I think it's better to simply use the approach I suggested last:

> > Rather than use $TMPDIR or /tmp, I think it'd be safer to place the file
> > in the same directory with the configure script or in the current
> > directory.  You're going to create config.mak in one of these
> > directories anyway - so just create the temporary file in that same
> > directory not to incur any extra risk (that you do by writing into a
> > second directory).
> 
> This seems reasonable.

Alexander

Re: configure script for musl (?!)

[Rich Felker]

On Wed, May 02, 2012 at 07:05:43PM +0400, Solar Designer wrote:
> I think the biggest problem here is in the failure mode.  In the draft
> script you posted, if "set -C" fails the script will proceed to
> potentially clobber a file via a (sym)link provided by another user, or
> it may use a FIFO or a regular file pre-created by another user (and set
> to e.g. mode 666), in which case the input to the compiler would be
> under that other user's control (even if we're running with a safe umask).

Since set is a shell builtin, I was assuming it won't fail, but I can
add "|| fail msg_here" to it.

> > I've actually been working on the issue and updated it to use $$,
> > $PPID, and a retry counter, so random failures will be extremely rare.
> > If anyone thinks it still matters, I'll add $(date|cksum) too.
> 
> Oh, I think it's better to simply use the approach I suggested last:
> 
> > > Rather than use $TMPDIR or /tmp, I think it'd be safer to place the file
> > > in the same directory with the configure script or in the current

I agree. In that case, secure creation is mostly a non-issue.

Rich

Re: configure script for musl (?!)

[Solar Designer]

On Wed, May 02, 2012 at 11:39:03AM -0400, Rich Felker wrote:
> On Wed, May 02, 2012 at 07:05:43PM +0400, Solar Designer wrote:
> > I think the biggest problem here is in the failure mode.  In the draft
> > script you posted, if "set -C" fails the script will proceed to
> > potentially clobber a file via a (sym)link provided by another user, or
> > it may use a FIFO or a regular file pre-created by another user (and set
> > to e.g. mode 666), in which case the input to the compiler would be
> > under that other user's control (even if we're running with a safe umask).
> 
> Since set is a shell builtin, I was assuming it won't fail, but I can
> add "|| fail msg_here" to it.

FWIW, I was referring to possible script invocations with a non-POSIX
shell.  If these merely fail to configure musl, that's acceptable.
If they expose the invoking user to unexpected security risks, that's
another story.

> > > > Rather than use $TMPDIR or /tmp, I think it'd be safer to place the file
> > > > in the same directory with the configure script or in the current
> 
> I agree. In that case, secure creation is mostly a non-issue.

Right.

Alexander

Re: configure script for musl (?!)

[Rich Felker]

On Wed, May 02, 2012 at 07:41:10PM +0400, Solar Designer wrote:
> > Since set is a shell builtin, I was assuming it won't fail, but I can
> > add "|| fail msg_here" to it.
> 
> FWIW, I was referring to possible script invocations with a non-POSIX
> shell.  If these merely fail to configure musl, that's acceptable.
> If they expose the invoking user to unexpected security risks, that's
> another story.

Yes, but I don't know any way to detect that a shell is non-conformant
short of checking for failure after we perform some particular option
(like set -C) that's possibly broken in non-POSIX shells.

Rich