Unit/regression testing (was Re: [musl] Priorities for next release?)

[Rich Felker <dalias@aerifal.cx>]

On Fri, Aug 10, 2012 at 12:23:17PM -0700, Nathan McSween wrote:
> >Other requests?
> 
> Unit tests of some sort, it helps to document the code as well as
> stop some regressions.

There are already a number of different tests; see:

http://nsz.repo.hu/git/
http://git.etalabs.net/cgi-bin/cgit.cgi
https://github.com/lmarcetic/cluts

for some that were developed specifically with musl in mind. We've
also been using:

http://posixtest.sourceforge.net/

but it's rather outdated and incorrect in some areas.

I've intentionally kept tests out of the main musl source tree for two
reasons: For one thing, I want to avoid burdening non-developers who
just want to download musl to use it with unnecessary download/storage
bulk. But the other reason is that I want the majority of libc test
cases to be libc-agnostic, i.e. designed to run against any libc, not
just musl. In that case, including them as part of musl feels wrong. I
do however also want to develop some musl-specific regression tests
for things that can't be tested in a generic way: things like ensuring
that static linking doesn't pull in modules it shouldn't, that
unnecessary syscall overhead at startup doesn't creep in, or that
musl-specific QoI guarantees are preserved (such as certain interfaces
not taking superlinear time or being async-signal-safe even when not
required by POSIX).

As to how testing relates to this release thread, I think development
of further major tests is outside the realm of the immediate sort of
release priorities I was asking about. But it's something I've been
wanting to pursue again for a long time, and I'd be quite happy to
find contributors interested in working on it.

Rich