* Release test framework @ 2013-12-03 22:11 Rich Felker 2013-12-04 5:51 ` 邓尧 ` (2 more replies) 0 siblings, 3 replies; 8+ messages in thread From: Rich Felker @ 2013-12-03 22:11 UTC (permalink / raw) To: musl One thing that's still missing that I had on the Roadmap for 0.9.15 is establishing a formal testing procedure for releasing. Basically what I have in mind is: For each arch: Assume the existence of a musl-cross compiler for it. Build musl and install to a prefix under the rest root. Build libc-test configured to use the new headers/libs. Create cpio archive containing: Newly built musl libc.so. Newly built libc-test tree. Provided base system template containing: Busybox. Simple /etc tree. Minimal init script to run tests. Boot qemu using a provided kernel and the new initramfs. Save output of tests outside the qemu environment. Diff against expected results for comparison. Does this seem like a reasonable and useful test procedure? Is anyone willing to volunteer to write the scripts for it? Rich ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Release test framework 2013-12-03 22:11 Release test framework Rich Felker @ 2013-12-04 5:51 ` 邓尧 2013-12-04 6:01 ` Rich Felker 2013-12-04 7:48 ` Szabolcs Nagy 2013-12-09 10:56 ` Waldemar Brodkorb 2 siblings, 1 reply; 8+ messages in thread From: 邓尧 @ 2013-12-04 5:51 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 1286 bytes --] If busybox is used, the test framework itself would depend on musl-libc, which means test test framework would depend on the test subject. In theory, it's a bad bad idea. 0.02$ On Wed, Dec 4, 2013 at 6:11 AM, Rich Felker <dalias@aerifal.cx> wrote: > One thing that's still missing that I had on the Roadmap for 0.9.15 is > establishing a formal testing procedure for releasing. Basically what > I have in mind is: > > For each arch: > Assume the existence of a musl-cross compiler for it. > Build musl and install to a prefix under the rest root. > Build libc-test configured to use the new headers/libs. > Create cpio archive containing: > Newly built musl libc.so. > Newly built libc-test tree. > Provided base system template containing: > Busybox. > Simple /etc tree. > Minimal init script to run tests. > Boot qemu using a provided kernel and the new initramfs. > Save output of tests outside the qemu environment. > Diff against expected results for comparison. > > Does this seem like a reasonable and useful test procedure? Is anyone > willing to volunteer to write the scripts for it? > > Rich > [-- Attachment #2: Type: text/html, Size: 1741 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Release test framework 2013-12-04 5:51 ` 邓尧 @ 2013-12-04 6:01 ` Rich Felker 2013-12-04 6:22 ` 邓尧 0 siblings, 1 reply; 8+ messages in thread From: Rich Felker @ 2013-12-04 6:01 UTC (permalink / raw) To: musl On Wed, Dec 04, 2013 at 01:51:42PM +0800, 邓尧 wrote: > If busybox is used, the test framework itself would depend on musl-libc, > which means test test framework would depend on the test subject. In > theory, it's a bad bad idea. > > 0.02$ The busybox used need not even be linked with musl; it could be a uclibc-linked binary from aboriginal, for example. The point was not to use busybox as any major part of the test framework, just to have a shell to run the script that would run all the tests from the libc-test package inside qemu. Rich ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Release test framework 2013-12-04 6:01 ` Rich Felker @ 2013-12-04 6:22 ` 邓尧 2013-12-04 7:06 ` Rich Felker 2013-12-04 7:16 ` Szabolcs Nagy 0 siblings, 2 replies; 8+ messages in thread From: 邓尧 @ 2013-12-04 6:22 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 1208 bytes --] On Wed, Dec 4, 2013 at 2:01 PM, Rich Felker <dalias@aerifal.cx> wrote: > On Wed, Dec 04, 2013 at 01:51:42PM +0800, 邓尧 wrote: > > If busybox is used, the test framework itself would depend on musl-libc, > > which means test test framework would depend on the test subject. In > > theory, it's a bad bad idea. > > > > 0.02$ > > The busybox used need not even be linked with musl; it could be a > uclibc-linked binary from aboriginal, for example. The point was not > to use busybox as any major part of the test framework, just to have a > shell to run the script that would run all the tests from the > libc-test package inside qemu. > Extracting test result isn't easy, and creating these initramfs and kernel images are time consuming. qemu-user may be a better choice, I'm not sure about the stability of qemu-user though. The following shows qemu-user could easily run a simple userspace program of a different architecture: hoolala@ubuntu:/tmp$ cat hello.c #include <stdio.h> int main() { printf("hello\n"); return 1; } hoolala@ubuntu:/tmp$ arm-linux-gnueabi-gcc hello.c -static hoolala@ubuntu:/tmp$ qemu-arm ./a.out hello hoolala@ubuntu:/tmp$ echo $? 1 [-- Attachment #2: Type: text/html, Size: 1687 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Release test framework 2013-12-04 6:22 ` 邓尧 @ 2013-12-04 7:06 ` Rich Felker 2013-12-04 7:16 ` Szabolcs Nagy 1 sibling, 0 replies; 8+ messages in thread From: Rich Felker @ 2013-12-04 7:06 UTC (permalink / raw) To: musl On Wed, Dec 04, 2013 at 02:22:58PM +0800, 邓尧 wrote: > On Wed, Dec 4, 2013 at 2:01 PM, Rich Felker <dalias@aerifal.cx> wrote: > > > On Wed, Dec 04, 2013 at 01:51:42PM +0800, 邓尧 wrote: > > > If busybox is used, the test framework itself would depend on musl-libc, > > > which means test test framework would depend on the test subject. In > > > theory, it's a bad bad idea. > > > > > > 0.02$ > > > > The busybox used need not even be linked with musl; it could be a > > uclibc-linked binary from aboriginal, for example. The point was not > > to use busybox as any major part of the test framework, just to have a > > shell to run the script that would run all the tests from the > > libc-test package inside qemu. > > > > Extracting test result isn't easy, Netcat (busybox version) it to a port on the host system, or just send it to the emulated serial tty and log the whole output of the qemu session. > and creating these initramfs and kernel > images are time consuming. Kernel images don't have to be created by the process (and shouldn't be); they can be stolen from anywhere (e.g. Aboriginal). > qemu-user may be a better choice, I'm not sure > about the stability of qemu-user though. The following shows qemu-user > could easily run a simple userspace program of a different architecture: qemu-user fails the old libc-testsuite badly due to various signal and thread issues that are almost impossible to get right with the design of qemu-user. I'm not sure offhand how badly it fails the new libc-test package tests; it's a bit of work to even configure the framework process to run them under qemu-user. But if there are any tests for thread cancellation, they should fail spectacularly under qemu-user... There are also some bad failures with SOCK_CLOEXEC and SOCK_NONBLOCK, iirc, and the degree of failure varies with the qemu version. > hoolala@ubuntu:/tmp$ cat hello.c > #include <stdio.h> > > int main() { > printf("hello\n"); > return 1; > } > hoolala@ubuntu:/tmp$ arm-linux-gnueabi-gcc hello.c -static > hoolala@ubuntu:/tmp$ qemu-arm ./a.out > hello > hoolala@ubuntu:/tmp$ echo $? > 1 Yes, lots of simple programs work. But qemu-user has a long way to go before it could be a viable test platform. Rich ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Release test framework 2013-12-04 6:22 ` 邓尧 2013-12-04 7:06 ` Rich Felker @ 2013-12-04 7:16 ` Szabolcs Nagy 1 sibling, 0 replies; 8+ messages in thread From: Szabolcs Nagy @ 2013-12-04 7:16 UTC (permalink / raw) To: musl * ?????? <torshie@gmail.com> [2013-12-04 14:22:58 +0800]: > Extracting test result isn't easy, and creating these initramfs and kernel > images are time consuming. qemu-user may be a better choice, I'm not sure > about the stability of qemu-user though. The following shows qemu-user the kernel syscall interface cannot be emulated in userspace correctly (clone, signals) and qemu-user has other bugs on various archs so it's not a reasonable design it might make sense to separate out the "safe" tests (no threads or fork, no filesystem modifications, etc) so one can run those tests more quickly/easily but in case of release testing you want to run all the tests reliably anyway so there it does not make much sense ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Release test framework 2013-12-03 22:11 Release test framework Rich Felker 2013-12-04 5:51 ` 邓尧 @ 2013-12-04 7:48 ` Szabolcs Nagy 2013-12-09 10:56 ` Waldemar Brodkorb 2 siblings, 0 replies; 8+ messages in thread From: Szabolcs Nagy @ 2013-12-04 7:48 UTC (permalink / raw) To: musl * Rich Felker <dalias@aerifal.cx> [2013-12-03 17:11:53 -0500]: > For each arch: > Assume the existence of a musl-cross compiler for it. > Build musl and install to a prefix under the rest root. > Build libc-test configured to use the new headers/libs. > Create cpio archive containing: > Newly built musl libc.so. > Newly built libc-test tree. > Provided base system template containing: > Busybox. > Simple /etc tree. > Minimal init script to run tests. > Boot qemu using a provided kernel and the new initramfs. > Save output of tests outside the qemu environment. > Diff against expected results for comparison. > > Does this seem like a reasonable and useful test procedure? Is anyone > willing to volunteer to write the scripts for it? libc-test is not well organized for running it out of tree, it uses gnu make to run the tests and generate the reports now so the target needs make as well otherwise it looks ok btw if tests are run as root then a bit more interfaces can be tested (setuid, permission failure handling, etc) which are not yet in the scope of libc-test ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Release test framework 2013-12-03 22:11 Release test framework Rich Felker 2013-12-04 5:51 ` 邓尧 2013-12-04 7:48 ` Szabolcs Nagy @ 2013-12-09 10:56 ` Waldemar Brodkorb 2 siblings, 0 replies; 8+ messages in thread From: Waldemar Brodkorb @ 2013-12-09 10:56 UTC (permalink / raw) To: musl Hi Rich, Rich Felker wrote, > One thing that's still missing that I had on the Roadmap for 0.9.15 is > establishing a formal testing procedure for releasing. Basically what > I have in mind is: > > For each arch: > Assume the existence of a musl-cross compiler for it. > Build musl and install to a prefix under the rest root. > Build libc-test configured to use the new headers/libs. > Create cpio archive containing: > Newly built musl libc.so. > Newly built libc-test tree. > Provided base system template containing: > Busybox. > Simple /etc tree. > Minimal init script to run tests. > Boot qemu using a provided kernel and the new initramfs. > Save output of tests outside the qemu environment. > Diff against expected results for comparison. > > Does this seem like a reasonable and useful test procedure? Is anyone > willing to volunteer to write the scripts for it? Looks like an interesting task. I programmed a prototype: git clone http://git.openadk.org/musl-test-framework.git cd musl-test-framework; ./musl-test.sh You need toolchains preinstalled. You could for example use OpenADK glibc toolchains from here: http://www.openadk.org/adk-toolchain.tar.xz Or modify musl-test.sh to point to all toolchains via TCPATH/TCVENDOR. It is tested with Qemu 1.7.0 (and required for microblaze support). The support for microblaze is working, but I did not manage to get the system automatically to reboot or powerdown. Any hints are welcomed. What do you think about it? best regards Waldemar ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-12-09 10:56 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-12-03 22:11 Release test framework Rich Felker 2013-12-04 5:51 ` 邓尧 2013-12-04 6:01 ` Rich Felker 2013-12-04 6:22 ` 邓尧 2013-12-04 7:06 ` Rich Felker 2013-12-04 7:16 ` Szabolcs Nagy 2013-12-04 7:48 ` Szabolcs Nagy 2013-12-09 10:56 ` Waldemar Brodkorb
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/musl/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).