Re: Building a solid musl automated-testing framework

[Rich Felker <dalias@libc.org>]

On Mon, Aug 04, 2014 at 02:18:55PM +0200, Waldemar Brodkorb wrote:
> Hi,
> Rich Felker wrote,
> 
> > I'd like to figure out how to setup the openadk test framework, or
> > adapt things from it, for automated testing all musl ports. The repo
> > is here:
> > 
> > http://www.openadk.org/cgi-bin/gitweb.cgi?p=adk-test-framework.git
> > 
> > There's a lot of stuff hard-coded for the openadk toolchains, whereas
> > I'd like to be able to use it with the musl-cross toolchains which are
> > more canonical. The scripts also seem to be incompatible with busybox
> > (using GNU features in something for making the initramfs, probably
> > cpio?). And by default it tries to test musl 1.0.1 and doesn't have an
> > obvious way to test from git.
> 
> I have copied the adk-test-framework logic to a new project.
> http://www.openadk.org/cgi-bin/gitweb.cgi?p=embedded-test.git;a=summary
> 
> It is more general and supports buildroot instead of openadk to make
> tests of toolchain building and runtime testing for glibc, uclibc,
> uclibc-ng and musl.

The big issue with using it directly is that we're not trying to test
whole toolchains, just musl, and testing toolchains instead of just
musl turns a 5 minute build (that's easy to run and re-run whenever
you want to test) into an hours-long process.

> The benefit is, you do not need any ready to go kernel and userland,
> because everything got compiled from scratch. So any fixes to 
> kernel config or userland can be done.
> 
> If there is an interest, I can add support for using an existing
> toolchain (like musl-cross) to build a kernel and userland to do
> the runtime testing.

I'd still much prefer to have the kernel built outside of the test
system, since it's not a component to be tested and in no way uses or
depends on the components to be tested. The situation is similar for
busybox, which is just used as a shell for executing the tests, not as
a component to be tested itself, but here there's also a consideration
of not introducing regressions in the shell that's controlling the
tests. I'd like to be using a "known good" shell and have only the
tests themselves linking against the new musl being tested.

Rich