Building a solid musl automated-testing framework

Building a solid musl automated-testing framework

[Rich Felker]

I'd like to figure out how to setup the openadk test framework, or
adapt things from it, for automated testing all musl ports. The repo
is here:

http://www.openadk.org/cgi-bin/gitweb.cgi?p=adk-test-framework.git

There's a lot of stuff hard-coded for the openadk toolchains, whereas
I'd like to be able to use it with the musl-cross toolchains which are
more canonical. The scripts also seem to be incompatible with busybox
(using GNU features in something for making the initramfs, probably
cpio?). And by default it tries to test musl 1.0.1 and doesn't have an
obvious way to test from git.

The idea I have in mind for using it as a basis for automated testing
would be:

1. Have a Makefile and a subdirectory full of cross compiler trees.
   The set of archs could be based purely on the set of cross
   compilers, computed by the Makefile based on directory contents.

2. Have the Makefile clone/pull one musl tree from upstream git and
   then clone/pull an additional copy of the tree per arch/subarch
   (with the set of archs determined as in point 1).

3. Build each musl (via recursive make, i.e. make calling configure
   and make in each musl clone) and install them into the cross
   compiler include/lib dirs for their corresponding cross compilers.

4. Do like points 2 and 3, but for libc-test repo.

5. Use the kernel, initrd, and qemu stuff from the adk-test-framework
   to build initrd images and run them.

The idea of using a Makefile (aside from declarative being The Right
Way to do anything like this :) is to make it easy to make a single
fix and re-run the test without rebuilding everything. Of course
rebuilding (via a "make clean" or similar) should aof coursebe an
option and what's usually used.

Does any of this make sense? Or is it simpler to get this kind of
automated testing just making a few tweaks to the existing scripts?

It would be ideal if the whole thing would work with dumping the test
repo on a system not necessarily setup for running it (e.g. some kind
of build farm or 'cloud computing' resource) and just produce the
results that could be read offline.

Rich

Re: Building a solid musl automated-testing framework

[u-igbb]

On Thu, Jul 31, 2014 at 05:47:16PM -0400, Rich Felker wrote:
> I'd like to figure out how to setup the openadk test framework, or

No useful opinion on the exact topic but

> The idea of using a Makefile (aside from declarative being The Right
> Way to do anything like this :) is to make it easy to make a single

+1 for declarative

Rune

Re: Building a solid musl automated-testing framework

[Bobby Bingham]

[-- Attachment #1: Type: text/plain, Size: 2503 bytes --]

On Thu, Jul 31, 2014 at 05:47:16PM -0400, Rich Felker wrote:
> I'd like to figure out how to setup the openadk test framework, or
> adapt things from it, for automated testing all musl ports. The repo

I like this idea.  I'll probably start looking into putting this
together soon if nobody gets to it first.

> is here:
>
> http://www.openadk.org/cgi-bin/gitweb.cgi?p=adk-test-framework.git
>
> There's a lot of stuff hard-coded for the openadk toolchains, whereas
> I'd like to be able to use it with the musl-cross toolchains which are
> more canonical. The scripts also seem to be incompatible with busybox
> (using GNU features in something for making the initramfs, probably
> cpio?). And by default it tries to test musl 1.0.1 and doesn't have an
> obvious way to test from git.
>
> The idea I have in mind for using it as a basis for automated testing
> would be:
>
> 1. Have a Makefile and a subdirectory full of cross compiler trees.
>    The set of archs could be based purely on the set of cross
>    compilers, computed by the Makefile based on directory contents.
>
> 2. Have the Makefile clone/pull one musl tree from upstream git and
>    then clone/pull an additional copy of the tree per arch/subarch
>    (with the set of archs determined as in point 1).
>
> 3. Build each musl (via recursive make, i.e. make calling configure
>    and make in each musl clone) and install them into the cross
>    compiler include/lib dirs for their corresponding cross compilers.
>
> 4. Do like points 2 and 3, but for libc-test repo.
>
> 5. Use the kernel, initrd, and qemu stuff from the adk-test-framework
>    to build initrd images and run them.
>

Last I looked, libc-test didn't support being built on one system and
run on another.  Has this changed?

> The idea of using a Makefile (aside from declarative being The Right
> Way to do anything like this :) is to make it easy to make a single
> fix and re-run the test without rebuilding everything. Of course

+1

> rebuilding (via a "make clean" or similar) should aof coursebe an
> option and what's usually used.
>
> Does any of this make sense? Or is it simpler to get this kind of
> automated testing just making a few tweaks to the existing scripts?
>
> It would be ideal if the whole thing would work with dumping the test
> repo on a system not necessarily setup for running it (e.g. some kind
> of build farm or 'cloud computing' resource) and just produce the
> results that could be read offline.

+1

>
> Rich

--
Bobby Bingham

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Building a solid musl automated-testing framework

[Szabolcs Nagy]

* Bobby Bingham <koorogi@koorogi.info> [2014-08-04 00:10:12 -0500]:
> On Thu, Jul 31, 2014 at 05:47:16PM -0400, Rich Felker wrote:
> >
> > 1. Have a Makefile and a subdirectory full of cross compiler trees.
> >    The set of archs could be based purely on the set of cross
> >    compilers, computed by the Makefile based on directory contents.
> >
> > 2. Have the Makefile clone/pull one musl tree from upstream git and
> >    then clone/pull an additional copy of the tree per arch/subarch
> >    (with the set of archs determined as in point 1).
> >
> > 3. Build each musl (via recursive make, i.e. make calling configure
> >    and make in each musl clone) and install them into the cross
> >    compiler include/lib dirs for their corresponding cross compilers.
> >
> > 4. Do like points 2 and 3, but for libc-test repo.
> >
> > 5. Use the kernel, initrd, and qemu stuff from the adk-test-framework
> >    to build initrd images and run them.
> >
> 
> Last I looked, libc-test didn't support being built on one system and
> run on another.  Has this changed?
> 

libc-test builds and runs the tests using make

building works with a cross-compiler, but running them does not

so one can build on one system and then manually run the tests
on another, there is no automated method for this now

Re: Building a solid musl automated-testing framework

[Waldemar Brodkorb]

Hi,
Rich Felker wrote,

> I'd like to figure out how to setup the openadk test framework, or
> adapt things from it, for automated testing all musl ports. The repo
> is here:
> 
> http://www.openadk.org/cgi-bin/gitweb.cgi?p=adk-test-framework.git
> 
> There's a lot of stuff hard-coded for the openadk toolchains, whereas
> I'd like to be able to use it with the musl-cross toolchains which are
> more canonical. The scripts also seem to be incompatible with busybox
> (using GNU features in something for making the initramfs, probably
> cpio?). And by default it tries to test musl 1.0.1 and doesn't have an
> obvious way to test from git.

I have copied the adk-test-framework logic to a new project.
http://www.openadk.org/cgi-bin/gitweb.cgi?p=embedded-test.git;a=summary

It is more general and supports buildroot instead of openadk to make
tests of toolchain building and runtime testing for glibc, uclibc,
uclibc-ng and musl.

The benefit is, you do not need any ready to go kernel and userland,
because everything got compiled from scratch. So any fixes to 
kernel config or userland can be done.

If there is an interest, I can add support for using an existing
toolchain (like musl-cross) to build a kernel and userland to do
the runtime testing.

I already removed, --quiet parameter from cpio, to provide better
usabilty on busybox based systems.

The old adk-toolchains will be removed soon. They are only for 
Linux x86_64 and dynamically linked. I may be provide some ready to
go toolchains for Linux/x86 statically linked with musl.

With embedded-test.sh you can build musl git or 1.1.4.

Without -s the version of the build system (here 1.1.4 is used)
./embedded-test.sh -l musl -v openadk 
With -s pointing to a musl-git tree, this version is used:
./embedded-test.sh -l musl -v openadk -s musl-git/

A basic bootup test can be used:
./embedded-test.sh -l musl -v openadk -b
Or the libc-test suite:
./embedded-test.sh -l musl -v openadk -t

As default all build system specific architectures are build.
You can just build arm with "-a arm".

If you have any suggestions or improvements I can add them.
 
I would like to keep using shell, without make for the basic
script.

best regards
 Waldemar

Re: Building a solid musl automated-testing framework

[Rich Felker]

On Mon, Aug 04, 2014 at 02:18:55PM +0200, Waldemar Brodkorb wrote:
> Hi,
> Rich Felker wrote,
> 
> > I'd like to figure out how to setup the openadk test framework, or
> > adapt things from it, for automated testing all musl ports. The repo
> > is here:
> > 
> > http://www.openadk.org/cgi-bin/gitweb.cgi?p=adk-test-framework.git
> > 
> > There's a lot of stuff hard-coded for the openadk toolchains, whereas
> > I'd like to be able to use it with the musl-cross toolchains which are
> > more canonical. The scripts also seem to be incompatible with busybox
> > (using GNU features in something for making the initramfs, probably
> > cpio?). And by default it tries to test musl 1.0.1 and doesn't have an
> > obvious way to test from git.
> 
> I have copied the adk-test-framework logic to a new project.
> http://www.openadk.org/cgi-bin/gitweb.cgi?p=embedded-test.git;a=summary
> 
> It is more general and supports buildroot instead of openadk to make
> tests of toolchain building and runtime testing for glibc, uclibc,
> uclibc-ng and musl.

The big issue with using it directly is that we're not trying to test
whole toolchains, just musl, and testing toolchains instead of just
musl turns a 5 minute build (that's easy to run and re-run whenever
you want to test) into an hours-long process.

> The benefit is, you do not need any ready to go kernel and userland,
> because everything got compiled from scratch. So any fixes to 
> kernel config or userland can be done.
> 
> If there is an interest, I can add support for using an existing
> toolchain (like musl-cross) to build a kernel and userland to do
> the runtime testing.

I'd still much prefer to have the kernel built outside of the test
system, since it's not a component to be tested and in no way uses or
depends on the components to be tested. The situation is similar for
busybox, which is just used as a shell for executing the tests, not as
a component to be tested itself, but here there's also a consideration
of not introducing regressions in the shell that's controlling the
tests. I'd like to be using a "known good" shell and have only the
tests themselves linking against the new musl being tested.

Rich

Re: Building a solid musl automated-testing framework

[Waldemar Brodkorb]

Hi,
Rich Felker wrote,

> On Mon, Aug 04, 2014 at 02:18:55PM +0200, Waldemar Brodkorb wrote:
> > Hi,
> > Rich Felker wrote,
> > 
> > > I'd like to figure out how to setup the openadk test framework, or
> > > adapt things from it, for automated testing all musl ports. The repo
> > > is here:
> > > 
> > > http://www.openadk.org/cgi-bin/gitweb.cgi?p=adk-test-framework.git
> > > 
> > > There's a lot of stuff hard-coded for the openadk toolchains, whereas
> > > I'd like to be able to use it with the musl-cross toolchains which are
> > > more canonical. The scripts also seem to be incompatible with busybox
> > > (using GNU features in something for making the initramfs, probably
> > > cpio?). And by default it tries to test musl 1.0.1 and doesn't have an
> > > obvious way to test from git.
> > 
> > I have copied the adk-test-framework logic to a new project.
> > http://www.openadk.org/cgi-bin/gitweb.cgi?p=embedded-test.git;a=summary
> > 
> > It is more general and supports buildroot instead of openadk to make
> > tests of toolchain building and runtime testing for glibc, uclibc,
> > uclibc-ng and musl.
> 
> The big issue with using it directly is that we're not trying to test
> whole toolchains, just musl, and testing toolchains instead of just
> musl turns a 5 minute build (that's easy to run and re-run whenever
> you want to test) into an hours-long process.

Only the first run will create a toolchain. On the second run
without -c, only the C library will be recompiled and installed.
 
> > The benefit is, you do not need any ready to go kernel and userland,
> > because everything got compiled from scratch. So any fixes to 
> > kernel config or userland can be done.
> > 
> > If there is an interest, I can add support for using an existing
> > toolchain (like musl-cross) to build a kernel and userland to do
> > the runtime testing.
> 
> I'd still much prefer to have the kernel built outside of the test
> system, since it's not a component to be tested and in no way uses or
> depends on the components to be tested. 

I do not agree here. For example the latest kernel includes
File-private POSIX locks.
https://lwn.net/Articles/586904/
If you want to add the needed bits and bytes to the C library it
would be nice to test it. Therefore you need the latest kernel.

> The situation is similar for
> busybox, which is just used as a shell for executing the tests, not as
> a component to be tested itself, but here there's also a consideration
> of not introducing regressions in the shell that's controlling the
> tests. I'd like to be using a "known good" shell and have only the
> tests themselves linking against the new musl being tested.

Recently I discovered a bug in uClibc in the machine dependent pipe
implementation for sparc by using mksh and its test suite.
So running application specific test suites for mksh/busybox might
discover C library bugs. So I want to support it.

Do you want to use a glibc basesystem to run the musl tests with
musl C library as in adk-test-framework or do you want to simply run
the test on a musl based system?
 
best regards 
 Waldemar