From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/3532 Path: news.gmane.org!not-for-mail From: Szabolcs Nagy Newsgroups: gmane.linux.lib.musl.general Subject: Re: Request for volunteers Date: Mon, 1 Jul 2013 22:58:57 +0200 Message-ID: <20130701205856.GS15323@port70.net> References: <20130630055202.GA16580@brightrain.aerifal.cx> <20130630104834.GP15323@port70.net> <20130701035117.GX29800@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1372712348 30677 80.91.229.3 (1 Jul 2013 20:59:08 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 1 Jul 2013 20:59:08 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-3536-gllmg-musl=m.gmane.org@lists.openwall.com Mon Jul 01 22:59:10 2013 Return-path: Envelope-to: gllmg-musl@plane.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1UtlBl-0003wo-PH for gllmg-musl@plane.gmane.org; Mon, 01 Jul 2013 22:59:09 +0200 Original-Received: (qmail 31790 invoked by uid 550); 1 Jul 2013 20:59:08 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 31781 invoked from network); 1 Jul 2013 20:59:08 -0000 Content-Disposition: inline In-Reply-To: <20130701035117.GX29800@brightrain.aerifal.cx> User-Agent: Mutt/1.5.21 (2010-09-15) Xref: news.gmane.org gmane.linux.lib.musl.general:3532 Archived-At: * Rich Felker [2013-06-30 23:51:17 -0400]: > This could involve tasks like: > > - Reading the git log and making a list of noteworthy bugs to add > regression tests for. > > - Determining (perhaps via automated coverage tools) major code we > lack any testing for. > > - Cross-checking glibc bug tracker and/or glibc tests for issues that > should be checked in musl too. i was thinking about this and a few category of tests: functional: black box testing of libc interfaces (eg input-output test vectors) tries to achieve good coverage regression: tests for bugs we found sometimes the bugs are musl specific or arch specific so i think these are worth keeping separately from the general functional tests (eg same repo but different dir) static: testing without executing code eg check the symbols and types in headers (i have tests that cover all posix headers only using cc) may be the binaries can be checked in some way as well static-src: code analyzis can be done on the source of musl (eg sparse, cppcheck, clang-analyzer were tried earlier) (this is different from the other tests and probably should be set up separately) metrics: benchmarks and quality of implementation metrics (eg performance, memory usage the usual checks but even ulp error measurements may be in this category) other tools: a coverage tool would be useful, i'm not sure if anyone set up one with musl yet (it's not just good for testing, but also for determining what interfaces are actually in use in the libc) clever fuzzer tools would be nice as well, but i don't know anything that can be used directly on a library (maybe with small effort they can be used to get better coverage) as a first step i guess we need to start with the functional and regression tests design goals of the test system: - tests should be easy to run even a single test in isolation (so test should be self contained if possible) - output is a report, failure cause should be clear - the system should not have external dependencies (other than libc, posix sh, gnu make: so tests are in .c files with simple buildsystem or .sh wrapper) - the failure of one test should not interfere with other tests (so tests should be in separate .c files each with main() and narrow scope, otherwise a build failure can affect a lot of tests) - the test system should run on all archs (so arch specific and implementation defined things should be treated carefully) - the test results should be robust (failures are always reported, deterministically if possible) - tests should leave the system in clean state (or easily cleanable state) some difficulties: - some "test framework" functionality would be nice, but can be problematic: eg using nice error reporting function on top of stdio may cause loss of info because of buffering in case of a crash - special compiler or linker flags (can be maintained in makefile or in the .c files as comments) - tests may require special environment, filesystem access, etc i'm not sure what's the best way to manage that (and some tests may need two different uid or other capabilities) - i looked at the bug history and many bugs are in hard to trigger cornercases (eg various races) or internally invoke ub in a way that may be hard to verify in a robust way - some tests may need significant support code to achieve good coverage (printf, math, string handling close to 2G,..) (in such cases we can go with simple self-contained tests without much coverage, but easy maintainance, or with something sophisticated) does that sound right? i think i can reorganize my libc tests to be more "scalabe" in these directions..