TLS and TSS APIs have race conditions in musl

TLS and TSS APIs have race conditions in musl

[Bruno Haible]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

Hi,

The attached test programs exercise the <pthread.h> thread-local storage (TLS)
API and the <threads.h> thread-specific storage (TSS) API in libc.

On Alpine Linux 3.7 and 2.9, these test programs, when run, produce this output:

$ ./test-tls
Starting test_tls ... OK
Starting test_tls_dtorcheck1 ... OK
Starting test_tls_dtorcheck2 ... OK
Starting test_tls_racecheck ...Aborted
$ ./test-tss
Starting test_tss ... OK
Starting test_tss_dtorcheck1 ... OK
Starting test_tss_dtorcheck2 ... OK
Starting test_tss_racecheck ...Aborted

The expected output - like seen on glibc, macOS, FreeBSD, NetBSD, OpenBSD,
AIX, HP-UX, IRIX, Solaris, Haiku, and Cygwin - is

$ ./test-tls
Starting test_tls ... OK
Starting test_tls_dtorcheck1 ... OK
Starting test_tls_dtorcheck2 ... OK
Starting test_tls_racecheck ... OK

test_tls_racecheck provokes races by doing many pthread_key_create and
pthread_key_delete calls in parallel threads.

test_tss_racecheck provokes races by doing many tss_create and tss_delete
calls in parallel threads.

I would guess that it is a musl libc bug.


[-- Attachment #2: test-tls.c --]
[-- Type: text/x-csrc, Size: 15405 bytes --]

/* Test of thread-local storage in multithreaded situations.
   Copyright (C) 2005, 2008-2019 Free Software Foundation, Inc.

   This program is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3 of the License, or
   (at your option) any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program.  If not, see <https://www.gnu.org/licenses/>.  */

/* Written by Bruno Haible <bruno@clisp.org>, 2005.  */

/* Whether to help the scheduler through explicit yield().
   Uncomment this to see if the operating system has a fair scheduler.  */
#define EXPLICIT_YIELD 1

/* Whether to print debugging messages.  */
#define ENABLE_DEBUGGING 0

#include <pthread.h>
#include <sched.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <signal.h>
#include <unistd.h>

#if ENABLE_DEBUGGING
# define dbgprintf printf
#else
# define dbgprintf if (0) printf
#endif

#if EXPLICIT_YIELD
# define yield() sched_yield ()
#else
# define yield()
#endif

#include <stdio.h>
#include <stdlib.h>

#ifndef FALLTHROUGH
# if __GNUC__ < 7
#  define FALLTHROUGH ((void) 0)
# else
#  define FALLTHROUGH __attribute__ ((__fallthrough__))
# endif
#endif

/* Define ASSERT_STREAM before including this file if ASSERT must
   target a stream other than stderr.  */
#ifndef ASSERT_STREAM
# define ASSERT_STREAM stderr
#endif

/* ASSERT (condition);
   verifies that the specified condition is fulfilled.  If not, a message
   is printed to ASSERT_STREAM if defined (defaulting to stderr if
   undefined) and the program is terminated with an error code.

   This macro has the following properties:
     - The programmer specifies the expected condition, not the failure
       condition.  This simplifies thinking.
     - The condition is tested always, regardless of compilation flags.
       (Unlike the macro from <assert.h>.)
     - On Unix platforms, the tester can debug the test program with a
       debugger (provided core dumps are enabled: "ulimit -c unlimited").
     - For the sake of platforms where no debugger is available (such as
       some mingw systems), an error message is printed on the error
       stream that includes the source location of the ASSERT invocation.
 */
#define ASSERT(expr) \
  do                                                                         \
    {                                                                        \
      if (!(expr))                                                           \
        {                                                                    \
          fprintf (ASSERT_STREAM, "%s:%d: assertion '%s' failed\n",     \
                   __FILE__, __LINE__, #expr);                          \
          fflush (ASSERT_STREAM);                                            \
          abort ();                                                          \
        }                                                                    \
    }                                                                        \
  while (0)

static void
perhaps_yield (void)
{
  /* Call yield () only with a certain probability, otherwise with GNU Pth
     the sequence of thread activations is too predictable.  */
  if ((((unsigned int) rand () >> 3) % 4) == 0)
    yield ();
}


/* ----------------------- Test thread-local storage ----------------------- */

/* Number of simultaneous threads.  */
#define THREAD_COUNT 16

/* Number of operations performed in each thread.  */
#define REPEAT_COUNT 50000

#define KEYS_COUNT 4

static pthread_key_t mykeys[KEYS_COUNT];

static void *
worker_thread (void *arg)
{
  unsigned int id = (unsigned int) (uintptr_t) arg;
  int i, j, repeat;
  unsigned int values[KEYS_COUNT];

  dbgprintf ("Worker %p started\n", (void *) pthread_self ());

  /* Initialize the per-thread storage.  */
  for (i = 0; i < KEYS_COUNT; i++)
    {
      values[i] = (((unsigned int) rand () >> 3) % 1000000) * THREAD_COUNT + id;
      /* Hopefully no arithmetic overflow.  */
      if ((values[i] % THREAD_COUNT) != id)
        abort ();
    }
  perhaps_yield ();

  /* Verify that the initial value is NULL.  */
  dbgprintf ("Worker %p before initial verify\n", (void *) pthread_self ());
  for (i = 0; i < KEYS_COUNT; i++)
    if (pthread_getspecific (mykeys[i]) != NULL)
      abort ();
  dbgprintf ("Worker %p after  initial verify\n", (void *) pthread_self ());
  perhaps_yield ();

  /* Initialize the per-thread storage.  */
  dbgprintf ("Worker %p before first tls_set\n", (void *) pthread_self ());
  for (i = 0; i < KEYS_COUNT; i++)
    {
      unsigned int *ptr = (unsigned int *) malloc (sizeof (unsigned int));
      *ptr = values[i];
      ASSERT (pthread_setspecific (mykeys[i], ptr) == 0);
    }
  dbgprintf ("Worker %p after  first tls_set\n", (void *) pthread_self ());
  perhaps_yield ();

  /* Shuffle around the pointers.  */
  for (repeat = REPEAT_COUNT; repeat > 0; repeat--)
    {
      dbgprintf ("Worker %p doing value swapping\n", (void *) pthread_self ());
      i = ((unsigned int) rand () >> 3) % KEYS_COUNT;
      j = ((unsigned int) rand () >> 3) % KEYS_COUNT;
      if (i != j)
        {
          void *vi = pthread_getspecific (mykeys[i]);
          void *vj = pthread_getspecific (mykeys[j]);

          ASSERT (pthread_setspecific (mykeys[i], vj) == 0);
          ASSERT (pthread_setspecific (mykeys[j], vi) == 0);
        }
      perhaps_yield ();
    }

  /* Verify that all the values are from this thread.  */
  dbgprintf ("Worker %p before final verify\n", (void *) pthread_self ());
  for (i = 0; i < KEYS_COUNT; i++)
    if ((*(unsigned int *) pthread_getspecific (mykeys[i]) % THREAD_COUNT) != id)
      abort ();
  dbgprintf ("Worker %p after  final verify\n", (void *) pthread_self ());
  perhaps_yield ();

  dbgprintf ("Worker %p dying.\n", (void *) pthread_self ());
  return NULL;
}

static void
test_tls (void)
{
  int pass, i;

  for (pass = 0; pass < 2; pass++)
    {
      pthread_t threads[THREAD_COUNT];

      if (pass == 0)
        for (i = 0; i < KEYS_COUNT; i++)
          ASSERT (pthread_key_create (&mykeys[i], free) == 0);
      else
        for (i = KEYS_COUNT - 1; i >= 0; i--)
          ASSERT (pthread_key_create (&mykeys[i], free) == 0);

      /* Spawn the threads.  */
      for (i = 0; i < THREAD_COUNT; i++)
        ASSERT (pthread_create (&threads[i], NULL, worker_thread, (void *) (uintptr_t) i) == 0);

      /* Wait for the threads to terminate.  */
      for (i = 0; i < THREAD_COUNT; i++)
        ASSERT (pthread_join (threads[i], NULL) == 0);

      for (i = 0; i < KEYS_COUNT; i++)
        ASSERT (pthread_key_delete (mykeys[i]) == 0);
    }
}

#undef KEYS_COUNT
#undef REPEAT_COUNT
#undef THREAD_COUNT


/* --------------- Test thread-local storage with destructors --------------- */

/* Number of simultaneous threads.  */
#define THREAD_COUNT 10

/* Number of keys to allocate in each thread.  */
#define KEYS_COUNT 10

static pthread_mutex_t sumlock = PTHREAD_MUTEX_INITIALIZER;
static uintptr_t sum;

static void
inc_sum (uintptr_t value)
{
  ASSERT (pthread_mutex_lock (&sumlock) == 0);
  sum += value;
  ASSERT (pthread_mutex_unlock (&sumlock) == 0);
}

static void
destructor0 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 0)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor1 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 1)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor2 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 2)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor3 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 3)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor4 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 4)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor5 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 5)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor6 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 6)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor7 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 7)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor8 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 8)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor9 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 9)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void (*destructor_table[10]) (void *) =
  {
    destructor0,
    destructor1,
    destructor2,
    destructor3,
    destructor4,
    destructor5,
    destructor6,
    destructor7,
    destructor8,
    destructor9
  };

static pthread_key_t dtorcheck_keys[THREAD_COUNT][KEYS_COUNT];

/* Worker thread that uses destructors that verify that the destructor belongs
   to the right thread.  */
static void *
dtorcheck1_thread (void *arg)
{
  unsigned int id = (unsigned int) (uintptr_t) arg;
  pthread_key_t *keys = dtorcheck_keys[id]; /* an array of KEYS_COUNT keys */
  int i;

  for (i = 0; i < KEYS_COUNT; i++)
    ASSERT (pthread_key_create (&keys[i], destructor_table[i]) == 0);

  for (i = 0; i < KEYS_COUNT; i++)
    ASSERT (pthread_setspecific (keys[i], (void *) (uintptr_t) (10 * id + i + 1)) == 0);

  return NULL;
}

static void
test_tls_dtorcheck1 (void)
{
  pthread_t threads[THREAD_COUNT];
  unsigned int id;
  int i;
  uintptr_t expected_sum;

  sum = 0;

  /* Spawn the threads.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (pthread_create (&threads[id], NULL, dtorcheck1_thread, (void *) (uintptr_t) id) == 0);

  /* Wait for the threads to terminate.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (pthread_join (threads[id], NULL) == 0);

  /* Clean up the keys.  */
  for (id = 0; id < THREAD_COUNT; id++)
    for (i = 0; i < KEYS_COUNT; i++)
      ASSERT (pthread_key_delete (dtorcheck_keys[id][i]) == 0);

  /* Check that the destructor was invoked for each key.  */
  expected_sum = 10 * KEYS_COUNT * (THREAD_COUNT * (THREAD_COUNT - 1) / 2)
                 + THREAD_COUNT * (KEYS_COUNT * (KEYS_COUNT - 1) / 2)
                 + THREAD_COUNT * KEYS_COUNT;
  if (sum != expected_sum)
    abort ();
}

/* Worker thread that uses destructors that verify that the destructor belongs
   to the right key allocated within the thread.  */
static void *
dtorcheck2_thread (void *arg)
{
  unsigned int id = (unsigned int) (uintptr_t) arg;
  pthread_key_t *keys = dtorcheck_keys[id]; /* an array of KEYS_COUNT keys */
  int i;

  for (i = 0; i < KEYS_COUNT; i++)
    ASSERT (pthread_key_create (&keys[i], destructor_table[id]) == 0);

  for (i = 0; i < KEYS_COUNT; i++)
    ASSERT (pthread_setspecific (keys[i], (void *) (uintptr_t) (10 * i + id + 1)) == 0);

  return NULL;
}

static void
test_tls_dtorcheck2 (void)
{
  pthread_t threads[THREAD_COUNT];
  unsigned int id;
  int i;
  uintptr_t expected_sum;

  sum = 0;

  /* Spawn the threads.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (pthread_create (&threads[id], NULL, dtorcheck2_thread, (void *) (uintptr_t) id) == 0);

  /* Wait for the threads to terminate.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (pthread_join (threads[id], NULL) == 0);

  /* Clean up the keys.  */
  for (id = 0; id < THREAD_COUNT; id++)
    for (i = 0; i < KEYS_COUNT; i++)
      ASSERT (pthread_key_delete (dtorcheck_keys[id][i]) == 0);

  /* Check that the destructor was invoked for each key.  */
  expected_sum = 10 * THREAD_COUNT * (KEYS_COUNT * (KEYS_COUNT - 1) / 2)
                 + KEYS_COUNT * (THREAD_COUNT * (THREAD_COUNT - 1) / 2)
                 + THREAD_COUNT * KEYS_COUNT;
  if (sum != expected_sum)
    abort ();
}

#undef KEYS_COUNT
#undef THREAD_COUNT


/* --- Test thread-local storage with with races between init and destroy --- */

/* Number of simultaneous threads.  */
#define THREAD_COUNT 10

/* Number of keys to allocate in each thread.  */
#define KEYS_COUNT 10

/* Number of times to destroy and reallocate a key in each thread.  */
#define REPEAT_COUNT 100000

static pthread_key_t racecheck_keys[THREAD_COUNT][KEYS_COUNT];

/* Worker thread that does many destructions and reallocations of keys, and also
   uses destructors that verify that the destructor belongs to the right key.  */
static void *
racecheck_thread (void *arg)
{
  unsigned int id = (unsigned int) (uintptr_t) arg;
  pthread_key_t *keys = racecheck_keys[id]; /* an array of KEYS_COUNT keys */
  int repeat;
  int i;

  dbgprintf ("Worker %p started\n", (void *) pthread_self ());

  for (i = 0; i < KEYS_COUNT; i++)
    {
      ASSERT (pthread_key_create (&keys[i], destructor_table[i]) == 0);
      ASSERT (pthread_setspecific (keys[i], (void *) (uintptr_t) (10 * id + i + 1)) == 0);
    }

  for (repeat = REPEAT_COUNT; repeat > 0; repeat--)
    {
      i = ((unsigned int) rand () >> 3) % KEYS_COUNT;
      dbgprintf ("Worker %p reallocating key %d\n", (void *) pthread_self (), i);
      ASSERT (pthread_key_delete (keys[i]) == 0);
      ASSERT (pthread_key_create (&keys[i], destructor_table[i]) == 0);
      ASSERT (pthread_setspecific (keys[i], (void *) (uintptr_t) (10 * id + i + 1)) == 0);
    }

  dbgprintf ("Worker %p dying.\n", (void *) pthread_self ());
  return NULL;
}

static void
test_tls_racecheck (void)
{
  pthread_t threads[THREAD_COUNT];
  unsigned int id;
  int i;
  uintptr_t expected_sum;

  sum = 0;

  /* Spawn the threads.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (pthread_create (&threads[id], NULL, racecheck_thread, (void *) (uintptr_t) id) == 0);

  /* Wait for the threads to terminate.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (pthread_join (threads[id], NULL) == 0);

  /* Clean up the keys.  */
  for (id = 0; id < THREAD_COUNT; id++)
    for (i = 0; i < KEYS_COUNT; i++)
      ASSERT (pthread_key_delete (racecheck_keys[id][i]) == 0);

  /* Check that the destructor was invoked for each key.  */
  expected_sum = 10 * KEYS_COUNT * (THREAD_COUNT * (THREAD_COUNT - 1) / 2)
                 + THREAD_COUNT * (KEYS_COUNT * (KEYS_COUNT - 1) / 2)
                 + THREAD_COUNT * KEYS_COUNT;
  if (sum != expected_sum)
    abort ();
}

#undef REPEAT_COUNT
#undef KEYS_COUNT
#undef THREAD_COUNT


/* -------------------------------------------------------------------------- */

int
main ()
{
  /* Declare failure if test takes too long, by using default abort
     caused by SIGALRM.  */
  int alarm_value = 600;
  signal (SIGALRM, SIG_DFL);
  alarm (alarm_value);

  printf ("Starting test_tls ..."); fflush (stdout);
  test_tls ();
  printf (" OK\n"); fflush (stdout);

  printf ("Starting test_tls_dtorcheck1 ..."); fflush (stdout);
  test_tls_dtorcheck1 ();
  printf (" OK\n"); fflush (stdout);

  printf ("Starting test_tls_dtorcheck2 ..."); fflush (stdout);
  test_tls_dtorcheck2 ();
  printf (" OK\n"); fflush (stdout);

  printf ("Starting test_tls_racecheck ..."); fflush (stdout);
  test_tls_racecheck ();
  printf (" OK\n"); fflush (stdout);

  return 0;
}

[-- Attachment #3: test-tss.c --]
[-- Type: text/x-csrc, Size: 15891 bytes --]

/* Test of thread-local storage in multithreaded situations.
   Copyright (C) 2005, 2008-2019 Free Software Foundation, Inc.

   This program is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3 of the License, or
   (at your option) any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program.  If not, see <https://www.gnu.org/licenses/>.  */

/* Written by Bruno Haible <bruno@clisp.org>, 2005.  */

/* Whether to help the scheduler through explicit yield().
   Uncomment this to see if the operating system has a fair scheduler.  */
#define EXPLICIT_YIELD 1

/* Whether to print debugging messages.  */
#define ENABLE_DEBUGGING 0

#include <threads.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <signal.h>
#include <unistd.h>

#if ENABLE_DEBUGGING
# define dbgprintf printf
#else
# define dbgprintf if (0) printf
#endif

#if EXPLICIT_YIELD
# define yield() thrd_yield ()
#else
# define yield()
#endif

#include <stdio.h>
#include <stdlib.h>

#ifndef FALLTHROUGH
# if __GNUC__ < 7
#  define FALLTHROUGH ((void) 0)
# else
#  define FALLTHROUGH __attribute__ ((__fallthrough__))
# endif
#endif

/* Define ASSERT_STREAM before including this file if ASSERT must
   target a stream other than stderr.  */
#ifndef ASSERT_STREAM
# define ASSERT_STREAM stderr
#endif

/* ASSERT (condition);
   verifies that the specified condition is fulfilled.  If not, a message
   is printed to ASSERT_STREAM if defined (defaulting to stderr if
   undefined) and the program is terminated with an error code.

   This macro has the following properties:
     - The programmer specifies the expected condition, not the failure
       condition.  This simplifies thinking.
     - The condition is tested always, regardless of compilation flags.
       (Unlike the macro from <assert.h>.)
     - On Unix platforms, the tester can debug the test program with a
       debugger (provided core dumps are enabled: "ulimit -c unlimited").
     - For the sake of platforms where no debugger is available (such as
       some mingw systems), an error message is printed on the error
       stream that includes the source location of the ASSERT invocation.
 */
#define ASSERT(expr) \
  do                                                                         \
    {                                                                        \
      if (!(expr))                                                           \
        {                                                                    \
          fprintf (ASSERT_STREAM, "%s:%d: assertion '%s' failed\n",     \
                   __FILE__, __LINE__, #expr);                          \
          fflush (ASSERT_STREAM);                                            \
          abort ();                                                          \
        }                                                                    \
    }                                                                        \
  while (0)

/* Returns a reference to the current thread as a pointer, for debugging.  */
#if defined __MVS__
  /* On IBM z/OS, pthread_t is a struct with an 8-byte '__' field.
     The first three bytes of this field appear to uniquely identify a
     pthread_t, though not necessarily representing a pointer.  */
# define thrd_current_pointer() (*((void **) thrd_current ().__))
#elif defined __sun
  /* On Solaris, thrd_t is merely an 'unsigned int'.  */
# define thrd_current_pointer() ((void *) (uintptr_t) thrd_current ())
#else
# define thrd_current_pointer() ((void *) thrd_current ())
#endif

static void
perhaps_yield (void)
{
  /* Call yield () only with a certain probability, otherwise the
     sequence of thread activations may be too predictable.  */
  if ((((unsigned int) rand () >> 3) % 4) == 0)
    yield ();
}


/* ----------------------- Test thread-local storage ----------------------- */

/* Number of simultaneous threads.  */
#define THREAD_COUNT 16

/* Number of operations performed in each thread.  */
#define REPEAT_COUNT 50000

#define KEYS_COUNT 4

static tss_t mykeys[KEYS_COUNT];

static int
worker_thread (void *arg)
{
  unsigned int id = (unsigned int) (uintptr_t) arg;
  int i, j, repeat;
  unsigned int values[KEYS_COUNT];

  dbgprintf ("Worker %p started\n", thrd_current_pointer ());

  /* Initialize the per-thread storage.  */
  for (i = 0; i < KEYS_COUNT; i++)
    {
      values[i] = (((unsigned int) rand () >> 3) % 1000000) * THREAD_COUNT + id;
      /* Hopefully no arithmetic overflow.  */
      if ((values[i] % THREAD_COUNT) != id)
        abort ();
    }
  perhaps_yield ();

  /* Verify that the initial value is NULL.  */
  dbgprintf ("Worker %p before initial verify\n", thrd_current_pointer ());
  for (i = 0; i < KEYS_COUNT; i++)
    if (tss_get (mykeys[i]) != NULL)
      abort ();
  dbgprintf ("Worker %p after  initial verify\n", thrd_current_pointer ());
  perhaps_yield ();

  /* Initialize the per-thread storage.  */
  dbgprintf ("Worker %p before first tss_set\n", thrd_current_pointer ());
  for (i = 0; i < KEYS_COUNT; i++)
    {
      unsigned int *ptr = (unsigned int *) malloc (sizeof (unsigned int));
      *ptr = values[i];
      ASSERT (tss_set (mykeys[i], ptr) == thrd_success);
    }
  dbgprintf ("Worker %p after  first tss_set\n", thrd_current_pointer ());
  perhaps_yield ();

  /* Shuffle around the pointers.  */
  for (repeat = REPEAT_COUNT; repeat > 0; repeat--)
    {
      dbgprintf ("Worker %p doing value swapping\n", thrd_current_pointer ());
      i = ((unsigned int) rand () >> 3) % KEYS_COUNT;
      j = ((unsigned int) rand () >> 3) % KEYS_COUNT;
      if (i != j)
        {
          void *vi = tss_get (mykeys[i]);
          void *vj = tss_get (mykeys[j]);

          ASSERT (tss_set (mykeys[i], vj) == thrd_success);
          ASSERT (tss_set (mykeys[j], vi) == thrd_success);
        }
      perhaps_yield ();
    }

  /* Verify that all the values are from this thread.  */
  dbgprintf ("Worker %p before final verify\n", thrd_current_pointer ());
  for (i = 0; i < KEYS_COUNT; i++)
    if ((*(unsigned int *) tss_get (mykeys[i]) % THREAD_COUNT) != id)
      abort ();
  dbgprintf ("Worker %p after  final verify\n", thrd_current_pointer ());
  perhaps_yield ();

  dbgprintf ("Worker %p dying.\n", thrd_current_pointer ());
  return 0;
}

static void
test_tss (void)
{
  int pass, i;

  for (pass = 0; pass < 2; pass++)
    {
      thrd_t threads[THREAD_COUNT];

      if (pass == 0)
        for (i = 0; i < KEYS_COUNT; i++)
          ASSERT (tss_create (&mykeys[i], free) == thrd_success);
      else
        for (i = KEYS_COUNT - 1; i >= 0; i--)
          ASSERT (tss_create (&mykeys[i], free) == thrd_success);

      /* Spawn the threads.  */
      for (i = 0; i < THREAD_COUNT; i++)
        ASSERT (thrd_create (&threads[i], worker_thread, (void *) (uintptr_t) i)
                == thrd_success);

      /* Wait for the threads to terminate.  */
      for (i = 0; i < THREAD_COUNT; i++)
        ASSERT (thrd_join (threads[i], NULL) == thrd_success);

      for (i = 0; i < KEYS_COUNT; i++)
        tss_delete (mykeys[i]);
    }
}

#undef KEYS_COUNT
#undef REPEAT_COUNT
#undef THREAD_COUNT


/* --------------- Test thread-local storage with destructors --------------- */

/* Number of simultaneous threads.  */
#define THREAD_COUNT 10

/* Number of keys to allocate in each thread.  */
#define KEYS_COUNT 10

static mtx_t sumlock;
static uintptr_t sum;

static void
inc_sum (uintptr_t value)
{
  ASSERT (mtx_lock (&sumlock) == thrd_success);
  sum += value;
  ASSERT (mtx_unlock (&sumlock) == thrd_success);
}

static void
destructor0 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 0)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor1 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 1)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor2 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 2)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor3 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 3)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor4 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 4)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor5 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 5)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor6 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 6)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor7 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 7)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor8 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 8)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void
destructor9 (void *value)
{
  if ((((uintptr_t) value - 1) % 10) != 9)
    abort ();
  inc_sum ((uintptr_t) value);
}

static void (*destructor_table[10]) (void *) =
  {
    destructor0,
    destructor1,
    destructor2,
    destructor3,
    destructor4,
    destructor5,
    destructor6,
    destructor7,
    destructor8,
    destructor9
  };

static tss_t dtorcheck_keys[THREAD_COUNT][KEYS_COUNT];

/* Worker thread that uses destructors that verify that the destructor belongs
   to the right thread.  */
static int
dtorcheck1_thread (void *arg)
{
  unsigned int id = (unsigned int) (uintptr_t) arg;
  tss_t *keys = dtorcheck_keys[id]; /* an array of KEYS_COUNT keys */
  int i;

  for (i = 0; i < KEYS_COUNT; i++)
    ASSERT (tss_create (&keys[i], destructor_table[i]) == thrd_success);

  for (i = 0; i < KEYS_COUNT; i++)
    ASSERT (tss_set (keys[i], (void *) (uintptr_t) (10 * id + i + 1))
            == thrd_success);

  return 0;
}

static void
test_tss_dtorcheck1 (void)
{
  thrd_t threads[THREAD_COUNT];
  unsigned int id;
  int i;
  uintptr_t expected_sum;

  sum = 0;

  /* Spawn the threads.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (thrd_create (&threads[id], dtorcheck1_thread, (void *) (uintptr_t) id)
            == thrd_success);

  /* Wait for the threads to terminate.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (thrd_join (threads[id], NULL) == thrd_success);

  /* Clean up the keys.  */
  for (id = 0; id < THREAD_COUNT; id++)
    for (i = 0; i < KEYS_COUNT; i++)
      tss_delete (dtorcheck_keys[id][i]);

  /* Check that the destructor was invoked for each key.  */
  expected_sum = 10 * KEYS_COUNT * (THREAD_COUNT * (THREAD_COUNT - 1) / 2)
                 + THREAD_COUNT * (KEYS_COUNT * (KEYS_COUNT - 1) / 2)
                 + THREAD_COUNT * KEYS_COUNT;
  if (sum != expected_sum)
    abort ();
}

/* Worker thread that uses destructors that verify that the destructor belongs
   to the right key allocated within the thread.  */
static int
dtorcheck2_thread (void *arg)
{
  unsigned int id = (unsigned int) (uintptr_t) arg;
  tss_t *keys = dtorcheck_keys[id]; /* an array of KEYS_COUNT keys */
  int i;

  for (i = 0; i < KEYS_COUNT; i++)
    ASSERT (tss_create (&keys[i], destructor_table[id]) == thrd_success);

  for (i = 0; i < KEYS_COUNT; i++)
    ASSERT (tss_set (keys[i], (void *) (uintptr_t) (10 * i + id + 1))
            == thrd_success);

  return 0;
}

static void
test_tss_dtorcheck2 (void)
{
  thrd_t threads[THREAD_COUNT];
  unsigned int id;
  int i;
  uintptr_t expected_sum;

  sum = 0;

  /* Spawn the threads.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (thrd_create (&threads[id], dtorcheck2_thread, (void *) (uintptr_t) id)
            == thrd_success);

  /* Wait for the threads to terminate.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (thrd_join (threads[id], NULL) == thrd_success);

  /* Clean up the keys.  */
  for (id = 0; id < THREAD_COUNT; id++)
    for (i = 0; i < KEYS_COUNT; i++)
      tss_delete (dtorcheck_keys[id][i]);

  /* Check that the destructor was invoked for each key.  */
  expected_sum = 10 * THREAD_COUNT * (KEYS_COUNT * (KEYS_COUNT - 1) / 2)
                 + KEYS_COUNT * (THREAD_COUNT * (THREAD_COUNT - 1) / 2)
                 + THREAD_COUNT * KEYS_COUNT;
  if (sum != expected_sum)
    abort ();
}

#undef KEYS_COUNT
#undef THREAD_COUNT


/* --- Test thread-local storage with with races between init and destroy --- */

/* Number of simultaneous threads.  */
#define THREAD_COUNT 10

/* Number of keys to allocate in each thread.  */
#define KEYS_COUNT 10

/* Number of times to destroy and reallocate a key in each thread.  */
#define REPEAT_COUNT 100000

static tss_t racecheck_keys[THREAD_COUNT][KEYS_COUNT];

/* Worker thread that does many destructions and reallocations of keys, and also
   uses destructors that verify that the destructor belongs to the right key.  */
static int
racecheck_thread (void *arg)
{
  unsigned int id = (unsigned int) (uintptr_t) arg;
  tss_t *keys = racecheck_keys[id]; /* an array of KEYS_COUNT keys */
  int repeat;
  int i;

  dbgprintf ("Worker %p started\n", thrd_current_pointer ());

  for (i = 0; i < KEYS_COUNT; i++)
    {
      ASSERT (tss_create (&keys[i], destructor_table[i]) == thrd_success);
      ASSERT (tss_set (keys[i], (void *) (uintptr_t) (10 * id + i + 1))
              == thrd_success);
    }

  for (repeat = REPEAT_COUNT; repeat > 0; repeat--)
    {
      i = ((unsigned int) rand () >> 3) % KEYS_COUNT;
      dbgprintf ("Worker %p reallocating key %d\n", thrd_current_pointer (), i);
      tss_delete (keys[i]);
      ASSERT (tss_create (&keys[i], destructor_table[i]) == thrd_success);
      ASSERT (tss_set (keys[i], (void *) (uintptr_t) (10 * id + i + 1))
              == thrd_success);
    }

  dbgprintf ("Worker %p dying.\n", thrd_current_pointer ());
  return 0;
}

static void
test_tss_racecheck (void)
{
  thrd_t threads[THREAD_COUNT];
  unsigned int id;
  int i;
  uintptr_t expected_sum;

  sum = 0;

  /* Spawn the threads.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (thrd_create (&threads[id], racecheck_thread, (void *) (uintptr_t) id)
            == thrd_success);

  /* Wait for the threads to terminate.  */
  for (id = 0; id < THREAD_COUNT; id++)
    ASSERT (thrd_join (threads[id], NULL) == thrd_success);

  /* Clean up the keys.  */
  for (id = 0; id < THREAD_COUNT; id++)
    for (i = 0; i < KEYS_COUNT; i++)
      tss_delete (racecheck_keys[id][i]);

  /* Check that the destructor was invoked for each key.  */
  expected_sum = 10 * KEYS_COUNT * (THREAD_COUNT * (THREAD_COUNT - 1) / 2)
                 + THREAD_COUNT * (KEYS_COUNT * (KEYS_COUNT - 1) / 2)
                 + THREAD_COUNT * KEYS_COUNT;
  if (sum != expected_sum)
    abort ();
}

#undef REPEAT_COUNT
#undef KEYS_COUNT
#undef THREAD_COUNT


/* -------------------------------------------------------------------------- */

int
main ()
{
  /* Declare failure if test takes too long, by using default abort
     caused by SIGALRM.  */
  int alarm_value = 600;
  signal (SIGALRM, SIG_DFL);
  alarm (alarm_value);

  ASSERT (mtx_init (&sumlock, mtx_plain) == thrd_success);

  printf ("Starting test_tss ..."); fflush (stdout);
  test_tss ();
  printf (" OK\n"); fflush (stdout);

  printf ("Starting test_tss_dtorcheck1 ..."); fflush (stdout);
  test_tss_dtorcheck1 ();
  printf (" OK\n"); fflush (stdout);

  printf ("Starting test_tss_dtorcheck2 ..."); fflush (stdout);
  test_tss_dtorcheck2 ();
  printf (" OK\n"); fflush (stdout);

  printf ("Starting test_tss_racecheck ..."); fflush (stdout);
  test_tss_racecheck ();
  printf (" OK\n"); fflush (stdout);

  return 0;
}

Re: TLS and TSS APIs have race conditions in musl

[Rich Felker]

On Wed, Jun 26, 2019 at 01:42:25AM +0200, Bruno Haible wrote:
> Hi,
> 
> The attached test programs exercise the <pthread.h> thread-local storage (TLS)
> API and the <threads.h> thread-specific storage (TSS) API in libc.
> 
> On Alpine Linux 3.7 and 2.9, these test programs, when run, produce this output:

Have you tried with current musl (1.1.22)? There were known race
conditions in older versions and I'm pretty sure Alpine 3.7 is
affected.

Rich

Re: TLS and TSS APIs have race conditions in musl

[Bruno Haible]

Hi Rich,

> > On Alpine Linux 3.7 and 2.9, these test programs, when run, produce this output:

This was a typo: I meant Alpine Linux 3.7 and 3.9.

> Have you tried with current musl (1.1.22)?

No. It surely is easier for you to try it with the current musl:
  $ gcc -Wall test-tls.c -o test-tls && ./test-tls

Bruno

Re: TLS and TSS APIs have race conditions in musl

[Rich Felker]

On Wed, Jun 26, 2019 at 03:02:10AM +0200, Bruno Haible wrote:
> Hi Rich,
> 
> > > On Alpine Linux 3.7 and 2.9, these test programs, when run, produce this output:
> 
> This was a typo: I meant Alpine Linux 3.7 and 3.9.
> 
> > Have you tried with current musl (1.1.22)?
> 
> No. It surely is easier for you to try it with the current musl:
>   $ gcc -Wall test-tls.c -o test-tls && ./test-tls

Indeed, sorry about that. I just ran it with git master and got "OK"
for all the results. So I think you were seeing one of the known,
already-fixed races. Thanks for reporting, though.

Rich

Re: TLS and TSS APIs have race conditions in musl

[Song Fangrui]

On Wed, Jun 26, 2019 at 1:02 AM Bruno Haible <bruno@clisp.org> wrote:
>
> Hi Rich,
>
> > > On Alpine Linux 3.7 and 2.9, these test programs, when run, produce this output:
>
> This was a typo: I meant Alpine Linux 3.7 and 3.9.
>
> > Have you tried with current musl (1.1.22)?
>
> No. It surely is easier for you to try it with the current musl:
>   $ gcc -Wall test-tls.c -o test-tls && ./test-tls
>
> Bruno
>

I checkout commit 59f88d766263344ce3e124d969ba66720aff4590 and the
previous commit.

The data race you saw was fixed by it "fix data race choosing next key
slot in pthread_key_create"

Re: TLS and TSS APIs have race conditions in musl

[Bruno Haible]

Song Fangrui wrote:
> I checkout commit 59f88d766263344ce3e124d969ba66720aff4590 and the
> previous commit.
> 
> The data race you saw was fixed by it "fix data race choosing next key
> slot in pthread_key_create"

Thanks! Glad to know it's fixed.

Bruno