Re: [musl] Re: [musl-cross-make] [PATCH v2] litecross: Fix system header dir when building native toolchains

[Rich Felker <dalias@libc.org>]

On Fri, May 08, 2020 at 05:27:11PM -0700, Michael Forney wrote:
> On 2020-03-22, Michael Forney <mforney@mforney.org> wrote:
> > Hi,
> >
> > Resending this patch because this is still an issue, and I realized
> > that the usr -> . symlink could also be removed in the build sysroot.
> > Anyone who has built a native toolchain with musl-cross-make has
> > probably run into this and worked around it by creating a usr -> .
> > symlink in the install directory manually.
> >
> > To demonstrate that this only currently works by accident with
> > cross-compilers, I'll start with current musl-cross-make master,
> > and override the sysroot to something else:
> >
> > $ make TARGET=aarch64-linux-musl SYSROOT=/something-else install
> > <snip>
> > $ echo '#include <stdlib.h>' | ./output/bin/aarch64-linux-musl-gcc -v -x c
> > -
> > <snip>
> > ignoring nonexistent directory
> > "/tmp/musl-cross-make/output/bin/../something-else/usr/local/include"
> > ignoring duplicate directory
> > "/tmp/musl-cross-make/output/bin/../lib/gcc/../../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/include"
> > ignoring nonexistent directory
> > "/tmp/musl-cross-make/output/bin/../something-else/usr/include"
> > ignoring duplicate directory
> > "/tmp/musl-cross-make/output/bin/../lib/gcc/../../lib/gcc/aarch64-linux-musl/9.2.0/include"
> > #include "..." search starts here:
> > #include <...> search starts here:
> > /tmp/musl-cross-make/output/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/include
> > /tmp/musl-cross-make/output/bin/../lib/gcc/aarch64-linux-musl/9.2.0/include
> > End of search list.
> > GNU C17 (GCC) version 9.2.0 (aarch64-linux-musl)
> >         compiled by GNU C version 9.2.0, GMP version 6.1.2, MPFR version
> > 4.0.2, MPC version 1.1.0, isl version none
> > GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
> > Compiler executable checksum: 32e14193b549d09fa3807c50f194d659
> > <stdin>:1:10: fatal error: stdlib.h: No such file or directory
> > compilation terminated.
> > $
> >
> > As you can see, gcc *is* searching for headers in the /something-else
> > sysroot, but under the incorrect system header directory. It still
> > searches for headers in /aarch64-linux-musl/include since that is
> > the gcc tool directory.
> >
> > To demonstrate the problem with native toolchains, let's build a
> > native toolchain and try to use it:
> >
> > $ make TARGET=x86_64-linux-musl NATIVE=1 install
> > <snip>
> > $ echo '#include <stdlib.h>' | ./output-x86_64-linux-musl/bin/gcc -v -x c -
> > <snip>
> > ignoring nonexistent directory
> > "/tmp/musl-cross-make/output-x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/include"
> > ignoring nonexistent directory
> > "/tmp/musl-cross-make/output-x86_64-linux-musl/bin/../usr/local/include"
> > ignoring nonexistent directory
> > "/tmp/musl-cross-make/output-x86_64-linux-musl/bin/../lib/gcc/../../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/include"
> > ignoring nonexistent directory
> > "/tmp/musl-cross-make/output-x86_64-linux-musl/bin/../usr/include"
> > ignoring duplicate directory
> > "/tmp/musl-cross-make/output-x86_64-linux-musl/bin/../lib/gcc/../../lib/gcc/x86_64-linux-musl/9.2.0/include"
> > #include "..." search starts here:
> > #include <...> search starts here:
> > /tmp/musl-cross-make/output-x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/9.2.0/include
> > End of search list.
> > GNU C17 (GCC) version 9.2.0 (x86_64-linux-musl)
> >         compiled by GNU C version 9.2.0, GMP version 6.1.2, MPFR version
> > 4.0.2, MPC version 1.1.0, isl version none
> > GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
> > Compiler executable checksum: 60320f85342923d8a86b40331aa7a54b
> > <stdin>:1:10: fatal error: stdlib.h: No such file or directory
> > compilation terminated.
> > $
> >
> > With the proposed patch, both of these cases are fixed. Hopefully
> > this makes the problem clear.
> 
> Hi Rich,
> 
> When I mentioned this patch on IRC, you expressed some concern that
> you didn't think --with-native-system-header-dir was relevant for
> cross compilers. However, I think I have made a pretty good case here
> showing that it is, and that the default value of /usr/include is not
> correct.
> 
> If you still have concerns, could you please let me know so I can try
> to address them?

Sorry I haven't had the time/attention to devote to this. I think
there's still a major mismatch in expectations -- the system you
install this native compiler onto is not going to have its include
files in /include but the standard /usr/include, among a lot of other
things. Cross-compiling a native compiler is in many ways a really
poor match for what musl-cross-make is intended to do. The "stop to
build musl in the middle" logic doesn't really even make much sense
when the target libs are being compiled by an existing cross compiler
toolchain that already has its own musl headers and libc.so.

I think you're right that we're doing something very wrong here, but I
don't understand exactly what it is, and before making changes I want
to get to the poing where we're on the same page about what it
*should* be doing. Maybe we mostly are, but I don't understand enough
yet to know whether we are.

I know there's also an issue someone (perhaps yourself; I don't
remember at the moment) brought to my attention that there's a mixup
in the logic in litecross/Makefile for "is a cross compile" vs "is
being cross compiled" that messes up completely in the canadian cross
case. This further suggests to me that there's a big mess that needs
to be unraveled..

I'll try to get back to this sometime soon, but at the moment my focus
is already split several ways between mallocng, initial entropy source
and csprng issues for musl (was considered a prereq for merging
mallocng but I think it's turning out to be mostly separate which may
mean I can punt on this for a while), bringing this all together for
next musl release, and paid work independent from musl.

Rich