Missing tests

Missing tests

[Rich Felker]

Here are some things that don't seem to be tested that would be nice
to get added to the test suite (http://nsz.repo.hu/git/?p=libc-test)

- Interpretation of POSIX TZ strings
- Daylight time transition tests with zoneinfo and POSIX zones
- Lots of corner cases for strftime and strptime
- Regression test for recent pthread_once race (hard to catch)
- Sanity-checking ucontext_t layout via signal handlers
- Alternate signal stacks
- Basic sanity checks for most syscalls
- Checks for various subtle stdio behavior:
  - File position on exit
  - Interactions of ungetc and file position
  - Append mode file positions

It might make sense to make a new tree separate from functional for
tests that don't really check the functionality of the interface but
just whether it's present and "doing something". Some existing tests
might be candidates for moving there too..?

Rich


P.S. I started writing this email quite a while back when I had
connectivity problems and never got around to finishing/sending it, so
some of it is motivated by discussions that were going on at the time.
:-)

Re: Missing tests

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2014-05-14 09:05:30 -0400]:
> Here are some things that don't seem to be tested that would be nice
> to get added to the test suite (http://nsz.repo.hu/git/?p=libc-test)
> 
> - Interpretation of POSIX TZ strings
> - Daylight time transition tests with zoneinfo and POSIX zones
> - Lots of corner cases for strftime and strptime
> - Regression test for recent pthread_once race (hard to catch)
> - Sanity-checking ucontext_t layout via signal handlers
> - Alternate signal stacks
i have a (regression) test for sigaltstack

> - Basic sanity checks for most syscalls
this can be tricky

> It might make sense to make a new tree separate from functional for
> tests that don't really check the functionality of the interface but
> just whether it's present and "doing something". Some existing tests
> might be candidates for moving there too..?

the code organization is not optimal

we can add a directory for trivial tests

the difficult part is when the test needs special environment
(eg syscall sanity checks or getaddrinfo): users/groups,
root-priv, fs-access, /etc/ changes,...
we can collect them in separate dirs and run only
when in a completely isolated environment (eg qemu)

linux specific tests should be separate too

another todo item is to improve cross compilation
(running cross compiled tests does not really work
now because it is done from make)
(the dso tests are a bit broken too because they use
relative path so only work when started from a specific
dir, maybe such test should be started from an .sh script)

Re: Missing tests

[Rich Felker]

On Wed, May 14, 2014 at 04:30:46PM +0200, Szabolcs Nagy wrote:
> > It might make sense to make a new tree separate from functional for
> > tests that don't really check the functionality of the interface but
> > just whether it's present and "doing something". Some existing tests
> > might be candidates for moving there too..?
> 
> the code organization is not optimal
> 
> we can add a directory for trivial tests
> 
> the difficult part is when the test needs special environment
> (eg syscall sanity checks or getaddrinfo): users/groups,
> root-priv, fs-access, /etc/ changes,...
> we can collect them in separate dirs and run only
> when in a completely isolated environment (eg qemu)

For getaddrinfo, I want a way to override the config via an
environment variable or similar, but I'm not sure what the right way
to do it would be. If we come up with a working solution, then the
tests could provide a fake DNS, and even do things like generate
malformed responses, to test the resolver.

Some of the others might be possible to test with user
namespaces/containers without needing root, but that would need to be
an option since it doesn't work on non-Linux or old-Linux systems.

> linux specific tests should be separate too

And musl-specific ones, if we have any (ones which test for behaviors
that musl documents but which are not required by the standards, like
AS-safety of snprintf, dprintf, etc.).

> another todo item is to improve cross compilation
> (running cross compiled tests does not really work
> now because it is done from make)
> (the dso tests are a bit broken too because they use
> relative path so only work when started from a specific
> dir, maybe such test should be started from an .sh script)

Yes, it would be nice if running the tests could be separated from
building them, but without sacrificing the declarative makefile which
is ideal for native testing.

Rich