updated cross-compiler build system

updated cross-compiler build system

[Rich Felker]

Hi everyone,

I've just pushed the new overhaul of musl-cross-make, my cross
compiler build system:

https://github.com/richfelker/musl-cross-make

This should make it faster (single-stage), safer (https and hash
checking), simpler (built-in fetching of gmp & friends), and cleaner
to build and install cross compilers that can be used to target J-core
Linux. The sh2eb-linux-muslfdpic target profile in the sample
config.mak(.dist) file is all you need to select and use. As usual it
also should work for all other (non J-core) musl targets too.

I've tested it pretty extensively on two systems I use for builds, but
as this is a big change/overhaul there might be things that break. Let
me know how using it goes.

Rich

Re: updated cross-compiler build system

[Szabolcs Nagy]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

* Rich Felker <dalias@libc.org> [2016-05-01 01:45:02 -0400]:
> I've just pushed the new overhaul of musl-cross-make, my cross
> compiler build system:
> 
> https://github.com/richfelker/musl-cross-make
> 
> This should make it faster (single-stage), safer (https and hash
> checking), simpler (built-in fetching of gmp & friends), and cleaner
> to build and install cross compilers that can be used to target J-core
> Linux. The sh2eb-linux-muslfdpic target profile in the sample
> config.mak(.dist) file is all you need to select and use. As usual it
> also should work for all other (non J-core) musl targets too.
> 
> I've tested it pretty extensively on two systems I use for builds, but
> as this is a big change/overhaul there might be things that break. Let
> me know how using it goes.

nice

i made some mods:

changed $$(LC_ROOT) to $${LC_ROOT} because it seems to
be passed to bash in config.status (as $$(, \$( and  $(
and the last one executes LC_ROOT as a command, this
might be an autoconf bug: it does not escape the args
correctly)

for me make fails in bfd/doc despite MAKINFO=false, but
the export AM_MAKEFLAGS=INFO_DEPS= hack fixed it.

src_mpfr dep was wrong.

i like saving the build log, but 2>&1 might not be what
everybody wants.

and linux headers are often needed to build things so
there could be an install target for it:

install-linuxheaders: install-toolchain
	case $(TARGET) in \
	i*86* | x86_64* | x32*) A=x86 ;; \
	arm*) A=arm ;; \
	aarch64*) A=arm64 ;; \
	mips*) A=mips ;; \
	or1k*) A=openrisc ;; \
	sh* | j[24]*) A=sh ;; \
	powerpc* | ppc*) A=powerpc ;; \
	microblaze*) A=microblaze ;; \
	esac; \
	$(MAKE) -C ../src_linux headers_install ARCH=$$A \
		CROSS_COMPILE=$(DESTDIR)$(OUTPUT)/$(TARGET)- \
		INSTALL_HDR_PATH=$(DESTDIR)$(OUTPUT)/$(TARGET)

[-- Attachment #2: a.diff --]
[-- Type: text/x-diff, Size: 1756 bytes --]

diff --git a/Makefile b/Makefile
index e85ccba..adac3c1 100644
--- a/Makefile
+++ b/Makefile
@@ -130,7 +130,7 @@ $(BUILD_DIR)/config.mak: | $(BUILD_DIR)
 	"-include ../config.mak"
 
 all: | $(SRC_DIRS) $(BUILD_DIR) $(BUILD_DIR)/Makefile $(BUILD_DIR)/config.mak
-	cd $(BUILD_DIR) && $(MAKE) $@
+	cd $(BUILD_DIR) && $(MAKE) $@ 2>&1 |tee -a build.log
 
 install: | $(SRC_DIRS) $(BUILD_DIR) $(BUILD_DIR)/Makefile $(BUILD_DIR)/config.mak
 	cd $(BUILD_DIR) && $(MAKE) OUTPUT=$(OUTPUT) $@
diff --git a/config.mak.dist b/config.mak.dist
index a152743..bde1641 100644
--- a/config.mak.dist
+++ b/config.mak.dist
@@ -35,6 +35,7 @@
 
 # COMMON_CONFIG += --disable-nls
 # COMMON_CONFIG += MAKEINFO=/bin/false
+# export AM_MAKEFLAGS = INFO_DEPS=
 # GCC_CONFIG += --enable-languages=c,c++
 # GCC_CONFIG += --disable-libquadmath --disable-decimal-float
 # GCC_CONFIG += --disable-multilib
diff --git a/litecross/Makefile b/litecross/Makefile
index 8657c56..16fc395 100644
--- a/litecross/Makefile
+++ b/litecross/Makefile
@@ -19,7 +19,7 @@ FULL_TOOLCHAIN_CONFIG = $(TOOLCHAIN_CONFIG) \
 	--disable-werror \
 	--target=$(TARGET) --prefix= \
 	--with-sysroot=/$(TARGET) \
-	--with-build-sysroot='$$(LC_ROOT)/obj_sysroot' \
+	--with-build-sysroot='$${LC_ROOT}/obj_sysroot' \
 	--enable-tls \
 	--disable-libmudflap --disable-libsanitizer
 
@@ -55,13 +55,13 @@ src_musl: | $(MUSL_SRCDIR)
 	ln -sf $(MUSL_SRCDIR) $@
 
 src_gmp: | $(GMP_SRCDIR)
-	ln -sf "$(GMP_SRCDIR)" $@
+	ln -sf $(GMP_SRCDIR) $@
 
 src_mpc: | $(MPC_SRCDIR)
-	ln -sf "$(MPC_SRCDIR)" $@
+	ln -sf $(MPC_SRCDIR) $@
 
-src_mpfr: | $(GMP_SRCDIR)
-	ln -sf "$(MPFR_SRCDIR)" $@
+src_mpfr: | $(MPFR_SRCDIR)
+	ln -sf $(MPFR_SRCDIR) $@
 
 src_toolchain: src_binutils src_gcc src_gmp src_mpc src_mpfr
 	rm -rf $@ $@.tmp

Re: [J-core] [musl] updated cross-compiler build system

[Rich Felker]

On Sun, May 01, 2016 at 05:38:04PM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2016-05-01 01:45:02 -0400]:
> > I've just pushed the new overhaul of musl-cross-make, my cross
> > compiler build system:
> > 
> > https://github.com/richfelker/musl-cross-make
> > 
> > This should make it faster (single-stage), safer (https and hash
> > checking), simpler (built-in fetching of gmp & friends), and cleaner
> > to build and install cross compilers that can be used to target J-core
> > Linux. The sh2eb-linux-muslfdpic target profile in the sample
> > config.mak(.dist) file is all you need to select and use. As usual it
> > also should work for all other (non J-core) musl targets too.
> > 
> > I've tested it pretty extensively on two systems I use for builds, but
> > as this is a big change/overhaul there might be things that break. Let
> > me know how using it goes.
> 
> nice
> 
> i made some mods:
> 
> changed $$(LC_ROOT) to $${LC_ROOT} because it seems to
> be passed to bash in config.status (as $$(, \$( and  $(
> and the last one executes LC_ROOT as a command, this
> might be an autoconf bug: it does not escape the args
> correctly)

I noticed that too but didn't get around to looking into the details
since nothing broke. The whole point though is to _avoid_ it getting
expanded in places like config.status -- I was trying to keep the
absolute path to the top-level dir from getting saved anywhere in the
build tree, so that it can safely be moved. Do you know if your change
preserves this property?

> for me make fails in bfd/doc despite MAKINFO=false, but
> the export AM_MAKEFLAGS=INFO_DEPS= hack fixed it.

Uhg, yes, I just installed texinfo on my VPS and forgot to check the
case where it's missing.

> src_mpfr dep was wrong.

Thanks.

> i like saving the build log, but 2>&1 might not be what
> everybody wants.

Indeed; see below for more thoughts on it:

> and linux headers are often needed to build things so
> there could be an install target for it:

I was thinking of this too, but it would be nice to be able to get
just the headers rather than the whole Linux source tree. I think one
or more musl-based distros might already have a stripped-down package
we could fetch, if the hosting is suitable.

> install-linuxheaders: install-toolchain
> 	case $(TARGET) in \
> 	i*86* | x86_64* | x32*) A=x86 ;; \
> 	arm*) A=arm ;; \
> 	aarch64*) A=arm64 ;; \
> 	mips*) A=mips ;; \
> 	or1k*) A=openrisc ;; \
> 	sh* | j[24]*) A=sh ;; \
> 	powerpc* | ppc*) A=powerpc ;; \
> 	microblaze*) A=microblaze ;; \
> 	esac; \
> 	$(MAKE) -C ../src_linux headers_install ARCH=$$A \
> 		CROSS_COMPILE=$(DESTDIR)$(OUTPUT)/$(TARGET)- \
> 		INSTALL_HDR_PATH=$(DESTDIR)$(OUTPUT)/$(TARGET)

AFAIK you don't need CROSS_COMPILE= to install headers, and in
principle it shouldn't be there because the install targets are not
intended to depend on each other. But you do need to install to a
staging dir and then copy to the final dest; the broken
headers_install target in Linux rm's the scsi headers (eew) provided
by libc and fails to provide replacements.

> diff --git a/Makefile b/Makefile
> index e85ccba..adac3c1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -130,7 +130,7 @@ $(BUILD_DIR)/config.mak: | $(BUILD_DIR)
>  	"-include ../config.mak"
>  
>  all: | $(SRC_DIRS) $(BUILD_DIR) $(BUILD_DIR)/Makefile $(BUILD_DIR)/config.mak
> -	cd $(BUILD_DIR) && $(MAKE) $@
> +	cd $(BUILD_DIR) && $(MAKE) $@ 2>&1 |tee -a build.log

Logging should probably either be left to the user or sent to a file
specific to the build directory rather than saved in the top-level
directory. The supported/intended use for the future is that you can
omit TARGET= in config.mak and instead have conditionals for he
settings you want for different targets, then do things like:

make TARGET=i486-linux-musl all

and have that build end up in its own isolated dir. It shouldn't be
hard to add "make TARGETS='a b c ...'" support too if desired using
another level of recursion, or even make target tuple names act as
make targets when $(TARGET) is unset.

> @@ -55,13 +55,13 @@ src_musl: | $(MUSL_SRCDIR)
>  	ln -sf $(MUSL_SRCDIR) $@
>  
>  src_gmp: | $(GMP_SRCDIR)
> -	ln -sf "$(GMP_SRCDIR)" $@
> +	ln -sf $(GMP_SRCDIR) $@
>  
>  src_mpc: | $(MPC_SRCDIR)
> -	ln -sf "$(MPC_SRCDIR)" $@
> +	ln -sf $(MPC_SRCDIR) $@
>  
> -src_mpfr: | $(GMP_SRCDIR)
> -	ln -sf "$(MPFR_SRCDIR)" $@
> +src_mpfr: | $(MPFR_SRCDIR)
> +	ln -sf $(MPFR_SRCDIR) $@

The dep was wrong, but the quoting was intentional, albeit untested.
The idea was to make broken symlinks that would then ENOENT out if the
caller does not want to provide these libs (i.e. wants to use the
system ones). But apparently symlink(2) fails with ENOENT in this
case, so I need a new solution...
 
Rich

Re: Re: [J-core] [musl] updated cross-compiler build system

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2016-05-01 13:15:55 -0400]:
> On Sun, May 01, 2016 at 05:38:04PM +0200, Szabolcs Nagy wrote:
> > changed $$(LC_ROOT) to $${LC_ROOT} because it seems to
> > be passed to bash in config.status (as $$(, \$( and  $(
> > and the last one executes LC_ROOT as a command, this
> > might be an autoconf bug: it does not escape the args
> > correctly)
> 
> I noticed that too but didn't get around to looking into the details
> since nothing broke. The whole point though is to _avoid_ it getting
> expanded in places like config.status -- I was trying to keep the
> absolute path to the top-level dir from getting saved anywhere in the
> build tree, so that it can safely be moved. Do you know if your change
> preserves this property?
> 

the only difference i see is the --with-build-sysroot in
gcc -v output has abs path in my version, but otherwise
strings gcc |grep / has the same output.

> > for me make fails in bfd/doc despite MAKINFO=false, but
> > the export AM_MAKEFLAGS=INFO_DEPS= hack fixed it.
> 
> Uhg, yes, I just installed texinfo on my VPS and forgot to check the
> case where it's missing.
> 

this hack is not ok in general as Makefile.am might
have its own AM_MAKEFLAGS.

> > 	$(MAKE) -C ../src_linux headers_install ARCH=$$A \
> > 		CROSS_COMPILE=$(DESTDIR)$(OUTPUT)/$(TARGET)- \
> > 		INSTALL_HDR_PATH=$(DESTDIR)$(OUTPUT)/$(TARGET)
> 
> AFAIK you don't need CROSS_COMPILE= to install headers, and in
> principle it shouldn't be there because the install targets are not
> intended to depend on each other. But you do need to install to a
> staging dir and then copy to the final dest; the broken
> headers_install target in Linux rm's the scsi headers (eew) provided
> by libc and fails to provide replacements.
> 

i see

> > @@ -55,13 +55,13 @@ src_musl: | $(MUSL_SRCDIR)
> >  	ln -sf $(MUSL_SRCDIR) $@
> >  
> >  src_gmp: | $(GMP_SRCDIR)
> > -	ln -sf "$(GMP_SRCDIR)" $@
> > +	ln -sf $(GMP_SRCDIR) $@
> >  
> >  src_mpc: | $(MPC_SRCDIR)
> > -	ln -sf "$(MPC_SRCDIR)" $@
> > +	ln -sf $(MPC_SRCDIR) $@
> >  
> > -src_mpfr: | $(GMP_SRCDIR)
> > -	ln -sf "$(MPFR_SRCDIR)" $@
> > +src_mpfr: | $(MPFR_SRCDIR)
> > +	ln -sf $(MPFR_SRCDIR) $@
> 
> The dep was wrong, but the quoting was intentional, albeit untested.
> The idea was to make broken symlinks that would then ENOENT out if the
> caller does not want to provide these libs (i.e. wants to use the
> system ones). But apparently symlink(2) fails with ENOENT in this
> case, so I need a new solution...

i see

Re: [J-core] [musl] updated cross-compiler build system

[Rich Felker]

On Sun, May 01, 2016 at 06:50:55PM -0500, Rob Landley wrote:
> On 05/01/2016 12:15 PM, Rich Felker wrote:
> > I noticed that too but didn't get around to looking into the details
> > since nothing broke. The whole point though is to _avoid_ it getting
> > expanded in places like config.status -- I was trying to keep the
> > absolute path to the top-level dir from getting saved anywhere in the
> > build tree, so that it can safely be moved. Do you know if your change
> > preserves this property?
> 
> Did you check that it doesn't wind up hard-wired into the binaries?

Either way it's not used at runtime, but the value is shown by gcc -v
since --with-build-sysroot was on the configure command line which
gets saved. In my version, only the unexpanded make var appears there.
It's not a big deal, but I'd rather not leak the builder's directory
structure in binaries. Obviously fully suppressing this requires
either use of -g0 or the gcc options to fake paths for debug info.

> >> for me make fails in bfd/doc despite MAKINFO=false, but
> >> the export AM_MAKEFLAGS=INFO_DEPS= hack fixed it.
> > 
> > Uhg, yes, I just installed texinfo on my VPS and forgot to check the
> > case where it's missing.
> 
> I've been patching that out of the binutils I build for over 10 years
> and they STILL haven't fixed it upstream.
> 
> https://github.com/landley/aboriginal/blob/master/sources/patches/binutils-screwinfo.patch

If you're using this approach I think you should just remove the line
rather than "exit 0"; otherwise no output file is created and thus the
make rule that triggered it will keep getting run. An empty output
file is probably better.

Maybe a better approach, though, would be putting a fake makeinfo
shell script in the build system and passing in MAKEINFO=our_fake.

> > AFAIK you don't need CROSS_COMPILE= to install headers, and in
> > principle it shouldn't be there because the install targets are not
> > intended to depend on each other. But you do need to install to a
> > staging dir and then copy to the final dest; the broken
> > headers_install target in Linux rm's the scsi headers (eew) provided
> > by libc and fails to provide replacements.
> 
> You _are_ allowed to have a for-linus-nonsh bratch with patches to go
> upstream outside the arch/sh directory. Just submit it as a separate
> pull request.
> 
> (Me, I mail patches to Andrew Morton, who is more or less the designated
> nongit input point to the kernel integration process.)

I assumed somebody wants that broken behavior. It's easy enough to
work around with a staged install, anyway, but I might try to get it
fixed at some point.

> >> @@ -55,13 +55,13 @@ src_musl: | $(MUSL_SRCDIR)
> >>  	ln -sf $(MUSL_SRCDIR) $@
> >>  
> >>  src_gmp: | $(GMP_SRCDIR)
> >> -	ln -sf "$(GMP_SRCDIR)" $@
> >> +	ln -sf $(GMP_SRCDIR) $@
> >>  
> >>  src_mpc: | $(MPC_SRCDIR)
> >> -	ln -sf "$(MPC_SRCDIR)" $@
> >> +	ln -sf $(MPC_SRCDIR) $@
> >>  
> >> -src_mpfr: | $(GMP_SRCDIR)
> >> -	ln -sf "$(MPFR_SRCDIR)" $@
> >> +src_mpfr: | $(MPFR_SRCDIR)
> >> +	ln -sf $(MPFR_SRCDIR) $@
> > 
> > The dep was wrong, but the quoting was intentional, albeit untested.
> > The idea was to make broken symlinks that would then ENOENT out if the
> > caller does not want to provide these libs (i.e. wants to use the
> > system ones). But apparently symlink(2) fails with ENOENT in this
> > case, so I need a new solution...
> 
> I don't understand? Your _destination_ must exist. Your source can more
> or less be an arbitrary string.

Except the empty string. Linux's symlink(2) fails with ENOENT when the
source is a zero-length string. I'm skeptical whether this behavior is
conforming.

> $ ln -s "martha the whistling tapeworm/////" fred
> $ ls -l fred
> lrwxrwxrwx 1 landley landley 31 May  1 18:49 fred -> martha the
> whistling tapeworm/////

Try ln -s "" fred

Rich

Re: [J-core] [musl] updated cross-compiler build system

[Rob Landley]

On 05/01/2016 09:58 PM, Rich Felker wrote:
>>> The dep was wrong, but the quoting was intentional, albeit untested.

In bash you can ${blah:-defaultvalue}, but make is crazy.

>>> The idea was to make broken symlinks that would then ENOENT out if the
>>> caller does not want to provide these libs (i.e. wants to use the
>>> system ones). But apparently symlink(2) fails with ENOENT in this
>>> case, so I need a new solution...
>>
>> I don't understand? Your _destination_ must exist. Your source can more
>> or less be an arbitrary string.
> 
> Except the empty string. Linux's symlink(2) fails with ENOENT when the
> source is a zero-length string. I'm skeptical whether this behavior is
> conforming.

Looks like it was introduced in Linux 0.99.7 back in 1993. In my
one-big-repository going back to 0.0.1 it's commit a2a2a154ed6 and the
comment is:

+ * POSIX.1 2.4: an empty pathname is invalid (ENOENT).

Posix says:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html

[ENOENT]
A component of the path prefix of path2 does not name an existing file
or path2 is an empty string.

And this would be also ENOENTing for path1 _and_ path2, but I'm pretty
sure that if Linux has had a historical behavior for 23 years and nobody
cared to correct it, Posix is what's wrong.

NetBSD wasn't sure what to do a few years ago:

https://mail-index.netbsd.org/tech-kern/2013/11/01/msg015865.html

MacOS fsck considers zero length symlinks to be an error in HFS:

https://www.reddit.com/r/applehelp/comments/25h4by/2009_macbook_pro_failing_to_start/

Rob

question: use of musl-libc

[anonymous]

#1  a repost, kinda.  i have no idea if you all can see my post - i 
wrote a post yesterday did not appear on list

#2  i'm interested in changing out gcc and glibc in the future (so many 
are for many reasons - like build failures of unix wares, slow build 
glibc, complexity to fix its every mutating abi, non unix abi, etc)

question: should musl-libc be used to replace libc6 and or libc5-compat? 
  or no, it is "a true unix abi" but really for new embedded projects, 
current projects may need significant altering "to work" - even though 
everything says "unix ABI" on it

https://sourceforge.net/p/x-lfs-2010/

question: i understand what musl-libc is. very unsure how hard it would 
be to get "most linux apps" including firefox-5,20,30 to compile. 
unsure if i tried it: unsure what "prefered C compiler is" and unsure 
what intended application is.  wondering if any of you have experience 
compling (many) linux apps from source against musl-libc and what was 
easy and what (wasn't).

(? BSD went from nice fast standard C to gcc now to clang but still g++ 
?.  apple is going clang bug uses ever mutating dialect "obj c" and 
other mutating interpreters.  the whole unix maket has been attacked 
with releases of mutations that are compatible with themselves and not 
even themselves)

(i'm almost thinking euro/asia is doing anything possible to prevent any 
standards from "maturing" into more mature unix/X11 releases - because 
they are always mutating languages util they no longer work with 
anything but the relesase they are selling (android) - and even then 
cause failures - except in the products the "pgp controllers" are 
releasing for sale.  never works as they say when others try it.)