Release test framework

Release test framework

[Rich Felker]

One thing that's still missing that I had on the Roadmap for 0.9.15 is
establishing a formal testing procedure for releasing. Basically what
I have in mind is:

For each arch:
	Assume the existence of a musl-cross compiler for it.
	Build musl and install to a prefix under the rest root.
	Build libc-test configured to use the new headers/libs.
	Create cpio archive containing:
		Newly built musl libc.so.
		Newly built libc-test tree.
		Provided base system template containing:
			Busybox.
			Simple /etc tree.
			Minimal init script to run tests.
	Boot qemu using a provided kernel and the new initramfs.
	Save output of tests outside the qemu environment.
	Diff against expected results for comparison.

Does this seem like a reasonable and useful test procedure? Is anyone
willing to volunteer to write the scripts for it?

Rich

Re: Release test framework

[邓尧]

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]

If busybox is used, the test framework itself would depend  on musl-libc,
which means test test framework would depend on the test subject. In
theory, it's a bad bad idea.

0.02$


On Wed, Dec 4, 2013 at 6:11 AM, Rich Felker <dalias@aerifal.cx> wrote:

> One thing that's still missing that I had on the Roadmap for 0.9.15 is
> establishing a formal testing procedure for releasing. Basically what
> I have in mind is:
>
> For each arch:
>         Assume the existence of a musl-cross compiler for it.
>         Build musl and install to a prefix under the rest root.
>         Build libc-test configured to use the new headers/libs.
>         Create cpio archive containing:
>                 Newly built musl libc.so.
>                 Newly built libc-test tree.
>                 Provided base system template containing:
>                         Busybox.
>                         Simple /etc tree.
>                         Minimal init script to run tests.
>         Boot qemu using a provided kernel and the new initramfs.
>         Save output of tests outside the qemu environment.
>         Diff against expected results for comparison.
>
> Does this seem like a reasonable and useful test procedure? Is anyone
> willing to volunteer to write the scripts for it?
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 1741 bytes --]

Re: Release test framework

[Rich Felker]

On Wed, Dec 04, 2013 at 01:51:42PM +0800, 邓尧 wrote:
> If busybox is used, the test framework itself would depend  on musl-libc,
> which means test test framework would depend on the test subject. In
> theory, it's a bad bad idea.
> 
> 0.02$

The busybox used need not even be linked with musl; it could be a
uclibc-linked binary from aboriginal, for example. The point was not
to use busybox as any major part of the test framework, just to have a
shell to run the script that would run all the tests from the
libc-test package inside qemu.

Rich

Re: Release test framework

[邓尧]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

On Wed, Dec 4, 2013 at 2:01 PM, Rich Felker <dalias@aerifal.cx> wrote:

> On Wed, Dec 04, 2013 at 01:51:42PM +0800, 邓尧 wrote:
> > If busybox is used, the test framework itself would depend  on musl-libc,
> > which means test test framework would depend on the test subject. In
> > theory, it's a bad bad idea.
> >
> > 0.02$
>
> The busybox used need not even be linked with musl; it could be a
> uclibc-linked binary from aboriginal, for example. The point was not
> to use busybox as any major part of the test framework, just to have a
> shell to run the script that would run all the tests from the
> libc-test package inside qemu.
>

Extracting test result isn't easy, and creating these initramfs and kernel
images are time consuming. qemu-user may be a better choice, I'm not sure
about the stability of qemu-user though. The following shows qemu-user
could easily run a simple userspace program of a different architecture:

hoolala@ubuntu:/tmp$ cat hello.c
#include <stdio.h>

int main() {
    printf("hello\n");
    return 1;
}
hoolala@ubuntu:/tmp$ arm-linux-gnueabi-gcc hello.c -static
hoolala@ubuntu:/tmp$ qemu-arm ./a.out
hello
hoolala@ubuntu:/tmp$ echo $?
1

[-- Attachment #2: Type: text/html, Size: 1687 bytes --]

Re: Release test framework

[Rich Felker]

On Wed, Dec 04, 2013 at 02:22:58PM +0800, 邓尧 wrote:
> On Wed, Dec 4, 2013 at 2:01 PM, Rich Felker <dalias@aerifal.cx> wrote:
> 
> > On Wed, Dec 04, 2013 at 01:51:42PM +0800, 邓尧 wrote:
> > > If busybox is used, the test framework itself would depend  on musl-libc,
> > > which means test test framework would depend on the test subject. In
> > > theory, it's a bad bad idea.
> > >
> > > 0.02$
> >
> > The busybox used need not even be linked with musl; it could be a
> > uclibc-linked binary from aboriginal, for example. The point was not
> > to use busybox as any major part of the test framework, just to have a
> > shell to run the script that would run all the tests from the
> > libc-test package inside qemu.
> >
> 
> Extracting test result isn't easy,

Netcat (busybox version) it to a port on the host system, or just send
it to the emulated serial tty and log the whole output of the qemu
session.

> and creating these initramfs and kernel
> images are time consuming.

Kernel images don't have to be created by the process (and shouldn't
be); they can be stolen from anywhere (e.g. Aboriginal).

> qemu-user may be a better choice, I'm not sure
> about the stability of qemu-user though. The following shows qemu-user
> could easily run a simple userspace program of a different architecture:

qemu-user fails the old libc-testsuite badly due to various signal and
thread issues that are almost impossible to get right with the design
of qemu-user. I'm not sure offhand how badly it fails the new
libc-test package tests; it's a bit of work to even configure the
framework process to run them under qemu-user. But if there are any
tests for thread cancellation, they should fail spectacularly under
qemu-user...

There are also some bad failures with SOCK_CLOEXEC and SOCK_NONBLOCK,
iirc, and the degree of failure varies with the qemu version.

> hoolala@ubuntu:/tmp$ cat hello.c
> #include <stdio.h>
> 
> int main() {
>     printf("hello\n");
>     return 1;
> }
> hoolala@ubuntu:/tmp$ arm-linux-gnueabi-gcc hello.c -static
> hoolala@ubuntu:/tmp$ qemu-arm ./a.out
> hello
> hoolala@ubuntu:/tmp$ echo $?
> 1

Yes, lots of simple programs work. But qemu-user has a long way to go
before it could be a viable test platform.

Rich

Re: Release test framework

[Szabolcs Nagy]

* ?????? <torshie@gmail.com> [2013-12-04 14:22:58 +0800]:
> Extracting test result isn't easy, and creating these initramfs and kernel
> images are time consuming. qemu-user may be a better choice, I'm not sure
> about the stability of qemu-user though. The following shows qemu-user

the kernel syscall interface cannot be emulated in userspace
correctly (clone, signals) and qemu-user has other bugs on
various archs so it's not a reasonable design

it might make sense to separate out the "safe" tests
(no threads or fork, no filesystem modifications, etc)
so one can run those tests more quickly/easily

but in case of release testing you want to run all the tests
reliably anyway so there it does not make much sense

Re: Release test framework

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2013-12-03 17:11:53 -0500]:
> For each arch:
> 	Assume the existence of a musl-cross compiler for it.
> 	Build musl and install to a prefix under the rest root.
> 	Build libc-test configured to use the new headers/libs.
> 	Create cpio archive containing:
> 		Newly built musl libc.so.
> 		Newly built libc-test tree.
> 		Provided base system template containing:
> 			Busybox.
> 			Simple /etc tree.
> 			Minimal init script to run tests.
> 	Boot qemu using a provided kernel and the new initramfs.
> 	Save output of tests outside the qemu environment.
> 	Diff against expected results for comparison.
> 
> Does this seem like a reasonable and useful test procedure? Is anyone
> willing to volunteer to write the scripts for it?

libc-test is not well organized for running it out of tree,
it uses gnu make to run the tests and generate the reports now
so the target needs make as well

otherwise it looks ok

btw if tests are run as root then a bit more interfaces can be
tested (setuid, permission failure handling, etc) which are
not yet in the scope of libc-test

Re: Release test framework

[Waldemar Brodkorb]

Hi Rich,
Rich Felker wrote,

> One thing that's still missing that I had on the Roadmap for 0.9.15 is
> establishing a formal testing procedure for releasing. Basically what
> I have in mind is:
> 
> For each arch:
> 	Assume the existence of a musl-cross compiler for it.
> 	Build musl and install to a prefix under the rest root.
> 	Build libc-test configured to use the new headers/libs.
> 	Create cpio archive containing:
> 		Newly built musl libc.so.
> 		Newly built libc-test tree.
> 		Provided base system template containing:
> 			Busybox.
> 			Simple /etc tree.
> 			Minimal init script to run tests.
> 	Boot qemu using a provided kernel and the new initramfs.
> 	Save output of tests outside the qemu environment.
> 	Diff against expected results for comparison.
> 
> Does this seem like a reasonable and useful test procedure? Is anyone
> willing to volunteer to write the scripts for it?

Looks like an interesting task. I programmed a prototype:
git clone http://git.openadk.org/musl-test-framework.git
cd musl-test-framework; ./musl-test.sh

You need toolchains preinstalled. You could for example use OpenADK
glibc toolchains from here:
http://www.openadk.org/adk-toolchain.tar.xz
Or modify musl-test.sh to point to all toolchains via TCPATH/TCVENDOR.

It is tested with Qemu 1.7.0 (and required for microblaze support).

The support for microblaze is working, but I did not manage to get
the system automatically to reboot or powerdown. Any hints are
welcomed.

What do you think about it? 

best regards
 Waldemar