Re: Timeline for 1.1.20?

[Rich Felker <dalias@libc.org>]

On Sun, Jul 29, 2018 at 07:40:25AM -0400, Christopher Friedt wrote:
> > As for the immediate need, though, it's *not* fancy CI processes but
> > actual test coverage. I'm really wary of projects that have fancy
> > process frameworks but nothing to show for it.
> 
> Reports are useful. The fancy processes (automation, etc) just make
> life easier in the end and they're pretty easy to set up after one or
> two projects. I usually prefer to get it out of the way early. The
> low-hanging fruit. For unit testing, it's helpful to adopt a
> framework. gtest is good, although it assumes you have a working c++
> compiler.
> 
> I get that you want to keep libc-test and musl separate for now, but
> normally test code is part of the same repository as the source it's
> testing. Maybe consider that in the future.

It's highly intentional that it's not done this way.

1. Having tests separately versioned from the code they're testing
   allows validation in both directions. By using an older version of
   the code being tested with the latest version of the tests, you can
   validate that the tests actually catch bugs which were known to be
   present in the older version.

2. For the most part, the tests are against interfaces, not a specific
   implementation of those interfaces. They're equally valid to run
   against other libcs, and by doing so you can compare correctness
   properties.

3. Storage/download size burden. roughly 1/3 of the size of gcc is
   testsuite, it's hundreds of MB, and the vast majority of that only
   makes sense to be run by people developing gcc, not end users
   building it. Making everyone download it is not friendly. For musl
   the numbers are smaller but the same principle applies.

4. Putting both in the same package is not really cross-compile
   friendly, since in general you need the package source on the build
   system but the tests on the host/target system.

Probably also some others I'm forgetting...

> > In particular, coverage for changes since 1.1.19 would include:
> >
> > - getrandom/getentropy basic functionality check
> > - setvbuf non-stub inplementation: basic functionality, check for
> >   writes outside the buffer, etc.
> > - malloc interposition: check that partial replacement doesn't result
> >   in unsafe behavior.
> > - pthread_create: confirm that scheduling and other attributes still
> >   work as expected after refactoring work.
> > - getddrinfo AI_ADDRCONFIG (can't really be tested without network
> >   namespaces though)
> >
> > Particular bugfixes that call for functionality or regression tests:
> >
> > b123f23 fix getopt wrongly treating colons in optstring as valid option chars
> > 0cf5058 fix nl_langinfo_l(CODESET, loc) reporting wrong locale's value
> > 282b1cd fix fmaf wrong result
> > ae2a01d fix wrong result in casin and many related complex functions
> > 10e4bd3 fix incorrect results for catan with some inputs
> > 4bf0717 fix return value of nice function
> > 3f6dc30 fix out of bounds write for zero length buffer in gethostname
> > 9be4ed5 getopt_long_only: don't prefix-match long-options that match short ones
> > 55a661f fix iconv buffer overflow converting to legacy JIS-based encodings
> > 99f4237 fix iconv conversion to UTF-32 with implicit (big) endianness
> > 165a1e3 fix iconv mapping of big5-hkscs characters that map to two unicode chars
> > 029c622 fix output size handling for multi-unicode-char big5-hkscs characters
> > 5c8e692 inet_ntop: do not compress single zeros in IPv6
> > 8b8fb7f correctly handle non-matching symbols in dladdr
> > 9cad27a fix writes outside buffer by ungetc after setvbuf
> > b3fa0f2 fix regression in alignment of dirent structs produced by readdir
> >
> > Some fixes that would not be confirmed with just libc-test, but that
> > needs more of a framework for covering multiple build configurations:
> >
> > a7c53e0 fix out-of-tree build of crt files with stack protector enabled
> > e3c682a work around arm gcc's rejection of r7 asm constraints in thumb mode
> 
> You've just listed off a number of work tickets ;-) I'll set something
> up and then show you the results - my server is limited in horsepower
> though, and (in terms of actual hardware), I'm limited to x86_64,
> arm64, arm32 (various arch versions), and a mips32.

Great.

Rich