Open POSIX Test Suite analysis

Open POSIX Test Suite analysis

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

Hi all,

I've just run through the OPTS (posixtest.sourceforge.net) tests again
and analyzed the failures in the thread, semaphore, timer, and shared
memory tests. A few of them resulted in bug fixes which have been
committed over the past couple hours. The majority of the failures are
bogus tests, which I have documented. This information should be
useful if we want to automate these tests for use as regression tests,
or if we want to make an updated/fixed version of the testsuite (which
hasn't been updated since 2005).

Rich

[-- Attachment #2: opts.txt --]
[-- Type: text/plain, Size: 3168 bytes --]


pthread_attr_getstack
1-1: unspecified behavior, calls getstack before setstack

pthread_attr_setinheritsched:
2-2, 2-3, 2-4: requires root

pthread_cancel
5-1: undefined behavior, uses thread id after lifetime has ended
3-1: requires root for sched param

pthread_cond_destroy
speculative/4-1: undefined behavior, destroys busy condvar

pthread_create
threads_scenarii.c wrongly requires page size to divide min stack size
2-1: undefined behavior, uses thread id after lifetime has ended
1-5, 3-2: probably bogus tests, trying to measure stack usage

pthread_detach
1-1, 1-2: undefined behavior, attempts to join a non-joinable thread
4-1: undefined behavior, calls pthread_detach on a non-joinable thread
4-2: undefined behavior, uses thread id after lifetime has ended

pthread_getcpuclockid
speculative/3-1: undefined behavior, uses thread id after lifetime has ended

pthread_getschedparam
1-2, 1-3: requires root for sched param

pthread_join
speculative/6-1: undefined behavior, attempts to join a non-joinable thread

pthread_key_create
2-1: undefined behavior, uses uninitialized key object
speculative/5-1: buggy test, <= should be <

pthread_kill
6-1: undefined behavior, uses thread id after lifetime has ended

pthread_mutex_getprioceiling
1-1: unsupported feature

pthread_mutex_init
1-2, 3-2: undefined behavior, attempts to deadlock a default-type mutex
5-1: utterly nonsensical test, attempts to fail by exhausting memory
speculative/5-2: utterly nonsensical test, no relation to mutexes

pthread_mutex_lock
3-1: spurious failure? crash? can't reproduce

pthread_mutexattr_getprotocol
1-2: testing unsupported feature

pthread_mutexattr_gettype
speculative/3-1: undefined behavior, accessing uninitialized object

pthread_mutexattr_setprotocol
1-1: testing unsupported feature

pthread_rwlock_rdlock
2-1, 2-2: testing desirable but unsatisfiable property, also missing root

pthread_rwlock_unlock
3-1: testing desirable but unsatisfiable property, also missing root

pthread_rwlock_wrlock
3-1: testing for error when deadlock is explicitly allowed behavior

pthread_setcanceltype
1-1: undefined behavior, calls async-cancel-unsafe functions with AC

pthread_setschedparam
1-2, 4-1: need root for scheduling

pthread_setschedprio
1-1: need root for scheduling

sem_init
7-1: nonsensical test, expects failure

sem_post
8-1: needs unsupported options

sem_unlink
3-1: needs root
4-1: undefined behavior, uninitialized filename
5-1: off-by-one error in check for ENAMETOOLONG, but test is wrong anyway

all AIO tests
wrongly check for ver == 200112L rather than >=

timer_create
10-1, 11-1: nonsensical test, expects a cputime timer to fire while sleeping

strftime
1-1: musl failure, broken %U format
2-1: musl failure, broken %OU format

clock_getcpuclockid
2-1: invalid test; ignores EPERM and assumes the clockid was invalid

mmap
11-4, 11-5: linux failure, no workaround (partial page bug)
23-1: linux failure, but musl could work around it
28-1: bogus test, ENXIO is for devices not files
31-1: bogus test, EOVERFLOW is for files with 32-bit off_t

shm_open
39-2: invalid test, name contains /'s

shm_unlink
10-2: invalid test, name contains /'s

Re: Open POSIX Test Suite analysis

[Rob Landley]

On 06/26/2013 09:27:43 PM, Rich Felker wrote:
> Hi all,
> 
> I've just run through the OPTS (posixtest.sourceforge.net) tests again
> and analyzed the failures in the thread, semaphore, timer, and shared
> memory tests. A few of them resulted in bug fixes which have been
> committed over the past couple hours. The majority of the failures are
> bogus tests, which I have documented. This information should be
> useful if we want to automate these tests for use as regression tests,
> or if we want to make an updated/fixed version of the testsuite (which
> hasn't been updated since 2005).

Documented where?

Rob

Re: Open POSIX Test Suite analysis

[Rich Felker]

On Wed, Jul 03, 2013 at 02:40:02AM -0500, Rob Landley wrote:
> On 06/26/2013 09:27:43 PM, Rich Felker wrote:
> >Hi all,
> >
> >I've just run through the OPTS (posixtest.sourceforge.net) tests again
> >and analyzed the failures in the thread, semaphore, timer, and shared
> >memory tests. A few of them resulted in bug fixes which have been
> >committed over the past couple hours. The majority of the failures are
> >bogus tests, which I have documented. This information should be
> >useful if we want to automate these tests for use as regression tests,
> >or if we want to make an updated/fixed version of the testsuite (which
> >hasn't been updated since 2005).
> 
> Documented where?

In the attachment to the email you replied to.

Rich