[musl] pthread_mutex_t shared between processes with different pid namespaces

[musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Personal]

Hello everyone,
I'm working on a library linked by some processes in order to exchange
information. Such library uses some pthread_mutex_t instances to safely
read/write the information to exchange: the mutexes are created with
the PTHREAD_PROCESS_SHARED and PTHREAD_MUTEX_ROBUST attributes and
shared through shared memory mmapped by the processes.

Now, for certain reasons, I have to run one of the processes in a
container and I found that, after a random interval of time, the
process in the container got stuck in a pthread_mutex_lock without any
reason.

After some investigation I figured out that if the container is started
without pid namespace isolation everithing works like a charm.

So the questions: is the pid namespace isolation a problem when working
with shared mutexes or should I investigate in other directions?
If the problem is pid namespace isolation, what could be done to make
it working apart from sharing the same pid namespace?

The actual development is based on musl 1.2.4 built with Yocto
Scarthgap for aarch64 and arm.

Thanks in advance for any help,
Daniele.

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Rich Felker]

On Tue, Jan 28, 2025 at 02:22:31PM +0100, Daniele Personal wrote:
> Hello everyone,
> I'm working on a library linked by some processes in order to exchange
> information. Such library uses some pthread_mutex_t instances to safely
> read/write the information to exchange: the mutexes are created with
> the PTHREAD_PROCESS_SHARED and PTHREAD_MUTEX_ROBUST attributes and
> shared through shared memory mmapped by the processes.
> 
> Now, for certain reasons, I have to run one of the processes in a
> container and I found that, after a random interval of time, the
> process in the container got stuck in a pthread_mutex_lock without any
> reason.
> 
> After some investigation I figured out that if the container is started
> without pid namespace isolation everithing works like a charm.
> 
> So the questions: is the pid namespace isolation a problem when working
> with shared mutexes or should I investigate in other directions?
> If the problem is pid namespace isolation, what could be done to make
> it working apart from sharing the same pid namespace?
> 
> The actual development is based on musl 1.2.4 built with Yocto
> Scarthgap for aarch64 and arm.

Yes, the pid namespace boundary is your problem. Process-shared
mutexes only work on the same logical system with a unique set of
thread identifiers. If you're trying to share them across different
pid namespaces, the same pid/tid may refer to different
processes/threads in different ones, and it's not usable as a mutex
ownership identity.

If you want robust-mutex-like functionality that bridges pid
namespaces, sysv semaphores are probably your only option.

Rich

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Personal]

On Tue, 2025-01-28 at 10:02 -0500, Rich Felker wrote:
> On Tue, Jan 28, 2025 at 02:22:31PM +0100, Daniele Personal wrote:
> > Hello everyone,
> > I'm working on a library linked by some processes in order to
> > exchange
> > information. Such library uses some pthread_mutex_t instances to
> > safely
> > read/write the information to exchange: the mutexes are created
> > with
> > the PTHREAD_PROCESS_SHARED and PTHREAD_MUTEX_ROBUST attributes and
> > shared through shared memory mmapped by the processes.
> > 
> > Now, for certain reasons, I have to run one of the processes in a
> > container and I found that, after a random interval of time, the
> > process in the container got stuck in a pthread_mutex_lock without
> > any
> > reason.
> > 
> > After some investigation I figured out that if the container is
> > started
> > without pid namespace isolation everithing works like a charm.
> > 
> > So the questions: is the pid namespace isolation a problem when
> > working
> > with shared mutexes or should I investigate in other directions?
> > If the problem is pid namespace isolation, what could be done to
> > make
> > it working apart from sharing the same pid namespace?
> > 
> > The actual development is based on musl 1.2.4 built with Yocto
> > Scarthgap for aarch64 and arm.
> 
> Yes, the pid namespace boundary is your problem. Process-shared
> mutexes only work on the same logical system with a unique set of
> thread identifiers. If you're trying to share them across different
> pid namespaces, the same pid/tid may refer to different
> processes/threads in different ones, and it's not usable as a mutex
> ownership identity.
> 
> If you want robust-mutex-like functionality that bridges pid
> namespaces, sysv semaphores are probably your only option.
> 
> Rich

Oh, thanks for clarifying it.

But at a first glance at man 2 semop I see:

Each semaphore in a System V semaphore set has the following
       associated values:

           unsigned short  semval;   /* semaphore value */
           unsigned short  semzcnt;  /* # waiting for zero */
           unsigned short  semncnt;  /* # waiting for increase */
           pid_t           sempid;   /* PID of process that last
                                        modified the semaphore value */

Could it be that also sysv semaphores don't like different pid
namespaces?

If so, the only chance is to use the same pid namespace, isn't it?

Daniele.

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Florian Weimer]

* Rich Felker:

> Yes, the pid namespace boundary is your problem. Process-shared
> mutexes only work on the same logical system with a unique set of
> thread identifiers. If you're trying to share them across different
> pid namespaces, the same pid/tid may refer to different
> processes/threads in different ones, and it's not usable as a mutex
> ownership identity.

Is this required for implementing the unlock-if-not-owner error code on
mutex unlock?

By the way, there is a proposal to teach the kernel to rewrite the
ownership list of task exit:

  [PATCH v2 0/4] futex: Drop ROBUST_LIST_LIMIT
  <https://lore.kernel.org/linux-kernel/20250127202608.223864-1-andrealmeid@igalia.com/>

I'm worried about the compatibility impact.

Thanks,
Florian

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Personal]

On Tue, 2025-01-28 at 19:24 +0100, Florian Weimer wrote:
> * Rich Felker:
> 
> > Yes, the pid namespace boundary is your problem. Process-shared
> > mutexes only work on the same logical system with a unique set of
> > thread identifiers. If you're trying to share them across different
> > pid namespaces, the same pid/tid may refer to different
> > processes/threads in different ones, and it's not usable as a mutex
> > ownership identity.

From what I see, the problem seems to happen only in case of contention
of the mutex.

int __pthread_mutex_timedlock(pthread_mutex_t *restrict m, const struct
timespec *restrict at)
{
	if ((m->_m_type&15) == PTHREAD_MUTEX_NORMAL
	    && !a_cas(&m->_m_lock, 0, EBUSY))
		return 0;

	int type = m->_m_type;
	int r, t, priv = (type & 128) ^ 128;

	r = __pthread_mutex_trylock(m);
	if (r != EBUSY) return r;

IIUC, if it is not locked, the __pthread_mutex_timedlock will acquire
it and return 0 (don't understand if with the first check or with the
__pthread_mutex_trylock) and everything works.

If instead it is locked the problem arises only inside the container.
If it was a pthread_mutex_lock it waits forever, if it was a timed lock
it exits after the timeout and you can retry.

Is this correct?

> 
> Is this required for implementing the unlock-if-not-owner error code
> on
> mutex unlock?

No, I don't see problems related to EOWNERDEAD.

> 
> By the way, there is a proposal to teach the kernel to rewrite the
> ownership list of task exit:
> 
>   [PATCH v2 0/4] futex: Drop ROBUST_LIST_LIMIT
>  
> <https://lore.kernel.org/linux-kernel/20250127202608.223864-1-andreal
> meid@igalia.com/>
> 
> I'm worried about the compatibility impact.
> 
> Thanks,
> Florian
> 

Thanks,
Daniele.

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Markus Wichmann]

Am Fri, Jan 31, 2025 at 10:31:46AM +0100 schrieb Daniele Personal:
> IIUC, if it is not locked, the __pthread_mutex_timedlock will acquire
> it and return 0 (don't understand if with the first check or with the
> __pthread_mutex_trylock) and everything works.
>
> If instead it is locked the problem arises only inside the container.
> If it was a pthread_mutex_lock it waits forever, if it was a timed lock
> it exits after the timeout and you can retry.
>
> Is this correct?
>

Essentially yes. If uncontended, kernel space never gets involved at all
and everything just works, but if contended, futex wait and futex wake
do not meet each other if issued from different PID namespaces. Thus
they end up waiting until the timeout expires. Unless there is no
timeout, then they wait until the user gets bored and kills the process.

Ciao,
Markus

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Florian Weimer]

* Daniele Personal:

>> Is this required for implementing the unlock-if-not-owner error code
>> on mutex unlock?
>
> No, I don't see problems related to EOWNERDEAD.

Sorry, what I meant is that the TID is needed for efficient reporting of
usage errors.  It's not imposed by the robust list protocol as such.
There could be a PID-namespace-compatible robust mutex type that does
not have this problem (but with less error checking).

Thanks,
Florian

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Personal]

On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> * Daniele Personal:
> 
> > > Is this required for implementing the unlock-if-not-owner error
> > > code
> > > on mutex unlock?
> > 
> > No, I don't see problems related to EOWNERDEAD.
> 
> Sorry, what I meant is that the TID is needed for efficient reporting
> of
> usage errors.  It's not imposed by the robust list protocol as such.
> There could be a PID-namespace-compatible robust mutex type that does
> not have this problem (but with less error checking).
> 
> Thanks,
> Florian
> 

Are you saying that there are pthread_mutexes which can be shared
across processes run on different pid namespaces? If yes I'm definitely
interested on this. Can you tell me something more?

What do you mean with "less error checking"? I need for sure to be able
to detect the EOWNERDEAD condition in order to restore consistency but
I'm not interested in recursive locks.

Thanks,
Daniele. 

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Personal]

On Fri, 2025-01-31 at 21:30 +0100, Markus Wichmann wrote:
> Am Fri, Jan 31, 2025 at 10:31:46AM +0100 schrieb Daniele Personal:
> > IIUC, if it is not locked, the __pthread_mutex_timedlock will
> > acquire
> > it and return 0 (don't understand if with the first check or with
> > the
> > __pthread_mutex_trylock) and everything works.
> > 
> > If instead it is locked the problem arises only inside the
> > container.
> > If it was a pthread_mutex_lock it waits forever, if it was a timed
> > lock
> > it exits after the timeout and you can retry.
> > 
> > Is this correct?
> > 
> 
> Essentially yes. If uncontended, kernel space never gets involved at
> all
> and everything just works, but if contended, futex wait and futex
> wake
> do not meet each other if issued from different PID namespaces. Thus
> they end up waiting until the timeout expires. Unless there is no
> timeout, then they wait until the user gets bored and kills the
> process.
> 
> Ciao,
> Markus

Thanks for the explanation. So there's no way to have pthread_mutexes
mapped on shared memory shared between host and container working when
the container is created with its own pid namespece?

Daniele.

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Florian Weimer]

* Daniele Personal:

> On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
>> * Daniele Personal:
>> 
>> > > Is this required for implementing the unlock-if-not-owner error
>> > > code
>> > > on mutex unlock?
>> > 
>> > No, I don't see problems related to EOWNERDEAD.
>> 
>> Sorry, what I meant is that the TID is needed for efficient reporting
>> of
>> usage errors.  It's not imposed by the robust list protocol as such.
>> There could be a PID-namespace-compatible robust mutex type that does
>> not have this problem (but with less error checking).
>> 
>> Thanks,
>> Florian
>> 
>
> Are you saying that there are pthread_mutexes which can be shared
> across processes run on different pid namespaces? If yes I'm definitely
> interested on this. Can you tell me something more?

You would have to add a new mutex type that is a mix of
PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the latter,
but without the ownership checks.

Thanks,
Florian

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Personal]

[-- Attachment #1: Type: text/plain, Size: 2600 bytes --]

On Mon, 2025-02-03 at 18:25 +0100, Florian Weimer wrote:
> * Daniele Personal:
> 
> > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> > > * Daniele Personal:
> > > 
> > > > > Is this required for implementing the unlock-if-not-owner
> > > > > error
> > > > > code
> > > > > on mutex unlock?
> > > > 
> > > > No, I don't see problems related to EOWNERDEAD.
> > > 
> > > Sorry, what I meant is that the TID is needed for efficient
> > > reporting
> > > of
> > > usage errors.  It's not imposed by the robust list protocol as
> > > such.
> > > There could be a PID-namespace-compatible robust mutex type that
> > > does
> > > not have this problem (but with less error checking).
> > > 
> > > Thanks,
> > > Florian
> > > 
> > 
> > Are you saying that there are pthread_mutexes which can be shared
> > across processes run on different pid namespaces? If yes I'm
> > definitely
> > interested on this. Can you tell me something more?
> 
> You would have to add a new mutex type that is a mix of
> PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the latter,
> but without the ownership checks.
> 
> Thanks,
> Florian
> 

I wrote a stupid test to exercise things. It creates (if needed) a
shared object which will hold the pthread_mutex_t instance, mmaps it
and if the shared object has been created, initializes the mutex
(changing mutex initialization requires to delete the shared object).

Then it locks the mutex for 5 seconds, unlocks it and locks it again
after another 2 seconds and exits with the mutex locked.

I compiled the code linking to musl 1.2.4 with Yocto or for my laptop
linking to glibc 2.34 with

gcc -D_GNU_SOURCE -Wall -pthread -g -O2 -c -o main.o main.c
gcc -pthread -g -O2 -pthread -o mutex-test main.o -lrt

I have a container which is started with its own pid, mount and uts
namespaces (it shares IPC and network with the host).

Running the test separately on host and container, both can recognize
the case where the previous instance left the mutex locked and can
recover from it.

Running them in parallel gives two distinct results if the mutex is
initialized with PTHREAD_PRIO_INHERIT protocol or PTHREAD_PRIO_NONE.

If PTHREAD_PRIO_INHERIT is set, in case of contention, the waiter gets
stuck.

If PTHREAD_PRIO_NONE is used, everything seems to work fine: the
application which starts later waits for the mutex to be released by
the other one and gets waked property.

I now need to understand if this behavior is expected and reliable or
not.

Thanks in advance,
Daniele.

[-- Attachment #2: main.c --]
[-- Type: text/x-csrc, Size: 11941 bytes --]

/*
 * main.c
 *
 *  Created on: Feb 4, 2025
 */

#include <time.h>
#include <errno.h>
#include <fcntl.h>
#include <sched.h>
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>

/**
 * Definition of shared object to be
 * mmapped() by the different processes
 */
typedef struct _SharedMutex
{
  /* A magic number just to check if object has been initialized */
  uint32_t magic;

  /* The attributes used to initialize the shared mutex */
  pthread_mutexattr_t attr;

  /* The shared mutex */
  pthread_mutex_t lock;
} SharedMutex;

#define MAGIC 0xdeadbeef

/** Default name to use when printing debug stuff */
static const char *app = "A";

#define DBG_PRINT(f,a...) \
  do {\
      struct timespec now; \
      clock_gettime (CLOCK_MONOTONIC, &now); \
      printf ("(%s) [%li.%09li] "f"\n", app, now.tv_sec, now.tv_nsec, ##a); \
  } while (0);

/** Protocol type: by default PTHREAD_PRIO_NONE */
static int prio = PTHREAD_PRIO_NONE;

/**
 * @brief Initialize a mutex instance setting #PTHREAD_MUTEX_ROBUST and
 *        #PTHREAD_PROCESS_SHARED attributes. This allows to have a mutex
 *        usable within different processes and recoverable in case the
 *        process which owns it crashes.
 *
 * @param lock A #pthread_mutex_t instance to initialize
 *
 * @return 0 on success, -1 otherwise
 */
static int mutex_init (pthread_mutex_t *lock, pthread_mutexattr_t *attr)
{
  int res;

  pthread_mutexattr_init (attr);

  DBG_PRINT ("setting PTHREAD_PROCESS_SHARED attribute");
  res = pthread_mutexattr_setpshared (attr, PTHREAD_PROCESS_SHARED);
  if (res != 0)
    {
      /* Failed to set SHARED */
      printf ("failed to set PTHREAD_PROCESS_SHARED: %i %s\n", res,
          strerror (res));
      return -1;
    }

  if (prio != PTHREAD_PRIO_NONE)
    {
      DBG_PRINT ("setting PTHREAD_PRIO_INHERIT attribute");
      res = pthread_mutexattr_setprotocol (attr, PTHREAD_PRIO_INHERIT);
      if (res != 0)
        {
          /* Failed to set protocol */
          printf ("failed to set PTHREAD_PRIO_INHERIT: %i %s\n", res,
              strerror (res));
          return -1;
        }
    }

  DBG_PRINT ("setting PTHREAD_MUTEX_ROBUST attribute");
  res = pthread_mutexattr_setrobust (attr, PTHREAD_MUTEX_ROBUST);
  if (res != 0)
    {
      printf ("failed to set PTHREAD_MUTEX_ROBUST: %i %s\n", res,
          strerror (res));
      return -1;
    }

  /*
   * Initialize mutex instance.
   * These method always return 0 so no need to check for success.
   */
  pthread_mutex_init (lock, attr);

  return 0;
}

/**
 * @brief Initialize a mutex instance setting #PTHREAD_MUTEX_ROBUST and
 *        #PTHREAD_PROCESS_SHARED attributes. This allows to have a mutex
 *        usable within different processes and recoverable in case the
 *        process which owns it crashes.
 *
 * @param lock A #pthread_mutex_t instance to initialize
 *
 * @return 0 on success, -1 otherwise
 */
static int mutexattr_check (pthread_mutexattr_t *attr)
{
  int res;
  int val;

  DBG_PRINT ("checking PTHREAD_PROCESS attribute");
  res = pthread_mutexattr_getpshared (attr, &val);
  if (res != 0)
    {
      /* Failed to set SHARED */
      printf ("failed to get PTHREAD_PROCESS attribute: %i %s\n",
          res, strerror (res));
      return -1;
    }
  if (val != PTHREAD_PROCESS_SHARED)
    {
      printf ("mutex was not initialized with PTHREAD_PROCESS_SHARED"
          " attribute\n");
      return -1;
    }

  DBG_PRINT ("checking protocol attribute");
  res = pthread_mutexattr_getprotocol (attr, &val);
  if (res != 0)
    {
      /* Failed to set protocol */
      printf ("failed to get protocol attribute: %i %s\n", res,
          strerror (res));
      return -1;
    }
  if (val != prio)
    {
      printf ("mutex was initialized with protocol %s but we wanted %s\n",
          (val == PTHREAD_PRIO_NONE) ? "PTHREAD_PRIO_NONE" : "PTHREAD_PRIO_INHERIT",
          (prio == PTHREAD_PRIO_NONE) ? "PTHREAD_PRIO_NONE" : "PTHREAD_PRIO_INHERIT");
      return -1;
    }

  DBG_PRINT ("checking PTHREAD_MUTEX_ROBUST attribute");
  res = pthread_mutexattr_getrobust (attr, &val);
  if (res != 0)
    {
      printf ("failed to get robust attribute: %i %s\n", res,
          strerror (res));
      return -1;
    }
  if (val != PTHREAD_MUTEX_ROBUST)
    {
      printf ("mutex was not initialized with PTHREAD_MUTEX_ROBUST"
          " attribute\n");
      return -1;
    }

  return 0;
}

/**
 * @brief Lock the given mutex. If the mutex is currently unlocked, it
 *        becomes locked and owned by the calling thread, and method
 *        returns immediately. If the mutex is already locked by another
 *        thread, method suspends the calling thread until the mutex is
 *        unlocked. If the thread which was owning the mutex terminates,
 *        method restores consistency of the mutex and locks it returning 0.
 *
 * @param lock A #pthread_mutex_t instance to lock
 *
 * @return 0 on success, a positive error code otherwise
 */
static int mutex_lock (pthread_mutex_t *lock)
{
  /* Try to lock mutex */
  int res = pthread_mutex_lock (lock);
  switch (res) {
    case 0:
      /* Common use case */
      break;
    case EOWNERDEAD:
      /*
       * Process which was holding the mutex terminated: now we're
       * holding it but before to go on we have to make it consistent.
       */
      DBG_PRINT ("restoring mutex consistency");
      res = pthread_mutex_consistent (lock);
      if (res != 0)
        {
          /* Failed to restore consistency */
          printf ("failed to restore consistency of mutex: %i %s\n", res,
              strerror (res));
        }

      break;
    default:
      printf ("failed to lock mutex: %i %s\n", res, strerror (res));
      break;
  }

  return res;
}

/**
 * @brief Try to open @shmfile shared object and if this is the first instance
 *        that opens it, initialize shared memory.
 *
 * @param name Name of the shared memory object to be created or opened.
 *             For portable use, a shared memory object should be identified
 *             by a name of the form /some-name; that is, a null-terminated
 *             string of up to NAME_MAX (i.e., 255) characters consisting of
 *             an initial slash, followed by one or more characters, none of
 *             which are slashes.
 *
 * @return pointer to #SharedMutex instance on success, NULL otherwise
 */
static SharedMutex *shm_map (const char *name)
{
#define SHM_OPS_INIT  1
#define SHM_OPS_CHECK 2

  int fd;
  int res;
  int ops = SHM_OPS_INIT;
  size_t size = sizeof (SharedMutex);

  /* Sanity checks */
  if (name == NULL)
    {
      printf ("unable to open shared object: missing shared object name\n");
      return NULL;
    }

  /* Open handle to shared object */
  fd = shm_open (name, O_RDWR | O_CREAT | O_EXCL, 0600);
  if (fd < 0 && errno == EEXIST)
    {
      /*
       * If memory object @name already exists, another process has
       * initialized the memory area.
       */
      ops = SHM_OPS_CHECK;
      fd = shm_open (name, O_RDWR, 0600);
    }

  if (fd < 0)
    {
      /* Failed to open shared object */
      printf ("unable to open shared object %s: %i %s\n", name, errno,
          strerror (errno));
      return NULL;
    }

  if (ops == SHM_OPS_INIT)
    {
      /* Set desired size of shared object */
      res = ftruncate (fd, (off_t) size);
      if (res < 0)
        {
          printf ("unable to set size of shared object %s: %i %s\n", name,
              errno, strerror (errno));
          close (fd);
          shm_unlink (name);
          return NULL;
        }
    }

  /* Map shared memory region */
  void *p = mmap (NULL, size, (PROT_READ | PROT_WRITE), MAP_SHARED, fd, 0);
  if (p == MAP_FAILED)
    {
      printf ("unable to map shared object %s contents: %i %s\n", name,
          errno, strerror (errno));

      close (fd);
      if (ops == SHM_OPS_INIT)
        {
          /* Also unlink shared object */
          shm_unlink (name);
        }

      return NULL;
    }

  /* We can safely close file descriptor */
  close (fd);

  /* Helper to access shared info */
  SharedMutex *info = p;

  switch (ops) {
    case SHM_OPS_INIT:
      /*
       * Clear contents of shared memory area before to start working on it.
       * This should not be needed because #ftruncate() should do it but
       * it doesn't harm.
       */
      memset (p, 0, size);

      /* Initialize shared lock */
      if (mutex_init (&info->lock, &info->attr) != 0)
        {
          /* Cleanup stuff */
          munmap (p, size);

          /* Remove shared object */
          shm_unlink (name);

          return NULL;
        }

      /* Write magic number */
      info->magic = MAGIC;
      break;
    case SHM_OPS_CHECK:
      /*
       * Shared object has already been created. Last thing that is set is the
       * magic value: loop until it becomes valid before to do other things.
       */
      while (info->magic != MAGIC)
        {
          if (info->magic != 0 && info->magic != MAGIC)
            {
              printf ("shared object %s initialized with wrong magic:"
                  " aborting\n", name);
              return NULL;
            }

          sched_yield ();
        }

      /* Check if attributes are consistent with our choices */
      res = mutexattr_check (&info->attr);
      if (res != 0)
        {
          printf ("mutex attributes incompatible: aborting\n");

          /* Cleanup stuff */
          munmap (p, size);

          return NULL;
        }

      break;
    default:
      /* This should never happen */
      printf ("invalid initialization option\n");
      return NULL;
  }

  /* Return mapped shared object */
  return info;
}

/**
 * @brief Show application usage
 */
static void usage (const char *name)
{
  printf ("Usage: %s [options]\n", name);
  printf (" -h | --help : show this help\n");
  printf (" -n | --name : a string to distinguish between running instances"
      " (default \"A\"\n");
  printf (" -p | --prio_inherit : set PTHREAD_PRIO_INHERIT (default"
      " PTHREAD_PRIO_NONE)\n");
}

/**
 * Main method
 */
int main (int argc, char *argv [])
{
  int res;
  SharedMutex *obj;

  for (int c = 1; c < argc; c++)
    {
      if (strcmp (argv [c], "-h") == 0 ||
          strcmp (argv [c], "--help") == 0)
        {
          usage (argv [0]);
          return -1;
        }

      if (strcmp (argv [c], "-n") == 0 ||
          strcmp (argv [c], "--name") == 0)
        {
          if ((++c) >= argc)
            {
              /* Missing name argument */
              usage (argv [0]);
              return -1;
            }

          /* Replace application name */
          app = argv [c];
          continue;
        }

      if (strcmp (argv [c], "-p") == 0 ||
          strcmp (argv [c], "--prio-inherit") == 0)
        {
          /* Set priority inheritance */
          prio = PTHREAD_PRIO_INHERIT;
          continue;
        }
    }

  DBG_PRINT ("Mapping shared object");
  obj = shm_map ("/test");
  if (obj == NULL)
    {
      /* Failed to create/open shared object */
      return -1;
    }

  DBG_PRINT ("locking mutex");
  res = mutex_lock (&obj->lock);
  if (res != 0)
    {
      /* Failed to lock mutex */
      return -1;
    }

  DBG_PRINT ("waiting 5 [s] before to unlock mutex");
  sleep (5);

  DBG_PRINT ("releasing mutex");
  res = pthread_mutex_unlock (&obj->lock);
  if (res != 0)
    {
      /* Failed to unlock */
      DBG_PRINT ("failed to unlock mutex: %i %s", res, strerror (res));
      return -1;
    }

  DBG_PRINT ("waiting 2 [s] before to retry locking mutex");
  sleep (2);

  DBG_PRINT ("locking mutex");
  res = mutex_lock (&obj->lock);
  if (res != 0)
    {
      /* Failed to lock mutex */
      return -1;
    }

  DBG_PRINT ("terminating with mutex locked");

  return 0;
}

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Rich Felker]

On Mon, Feb 03, 2025 at 06:25:41PM +0100, Florian Weimer wrote:
> * Daniele Personal:
> 
> > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> >> * Daniele Personal:
> >> 
> >> > > Is this required for implementing the unlock-if-not-owner error
> >> > > code
> >> > > on mutex unlock?
> >> > 
> >> > No, I don't see problems related to EOWNERDEAD.
> >> 
> >> Sorry, what I meant is that the TID is needed for efficient reporting
> >> of
> >> usage errors.  It's not imposed by the robust list protocol as such..
> >> There could be a PID-namespace-compatible robust mutex type that does
> >> not have this problem (but with less error checking).
> >> 
> >> Thanks,
> >> Florian
> >> 
> >
> > Are you saying that there are pthread_mutexes which can be shared
> > across processes run on different pid namespaces? If yes I'm definitely
> > interested on this. Can you tell me something more?
> 
> You would have to add a new mutex type that is a mix of
> PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the latter,
> but without the ownership checks.

This is inaccurate. Robust mutexes fundamentally depend on having the
owner's tid in the owner field, and on this value not matching the tid
of any other task that might hold the mutex. If these properties don't
hold, the mutex may fail to unlock when the owner dies, or incorrectly
unlock when another task mimicking the owner dies.

The Linux robust mutex protocol fundamentally does not work across pid
namespaces.

Rich

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Personal]

On Tue, 2025-02-04 at 13:53 -0500, Rich Felker wrote:
> On Mon, Feb 03, 2025 at 06:25:41PM +0100, Florian Weimer wrote:
> > * Daniele Personal:
> > 
> > > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> > > > * Daniele Personal:
> > > > 
> > > > > > Is this required for implementing the unlock-if-not-owner
> > > > > > error
> > > > > > code
> > > > > > on mutex unlock?
> > > > > 
> > > > > No, I don't see problems related to EOWNERDEAD.
> > > > 
> > > > Sorry, what I meant is that the TID is needed for efficient
> > > > reporting
> > > > of
> > > > usage errors.  It's not imposed by the robust list protocol as
> > > > such..
> > > > There could be a PID-namespace-compatible robust mutex type
> > > > that does
> > > > not have this problem (but with less error checking).
> > > > 
> > > > Thanks,
> > > > Florian
> > > > 
> > > 
> > > Are you saying that there are pthread_mutexes which can be shared
> > > across processes run on different pid namespaces? If yes I'm
> > > definitely
> > > interested on this. Can you tell me something more?
> > 
> > You would have to add a new mutex type that is a mix of
> > PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the
> > latter,
> > but without the ownership checks.
> 
> This is inaccurate. Robust mutexes fundamentally depend on having the
> owner's tid in the owner field, and on this value not matching the
> tid
> of any other task that might hold the mutex. If these properties
> don't
> hold, the mutex may fail to unlock when the owner dies, or
> incorrectly
> unlock when another task mimicking the owner dies.
> 
> The Linux robust mutex protocol fundamentally does not work across
> pid
> namespaces.
> 
> Rich

Looking at the code for musl 1.2.4, a pthread_mutex_t which has been
initialized as shared and robust but not PI capable leaves uncovered
only the case of pthread_mutex_unlock().

int __pthread_mutex_unlock(pthread_mutex_t *m)
{
	pthread_t self;
	int waiters = m->_m_waiters;
	int cont;
	int type = m->_m_type & 15;  <== type = 4 (robust)
	int priv = (m->_m_type & 128) ^ 128; <== priv = 0 (shared)
	int new = 0;
	int old;

	if (type != PTHREAD_MUTEX_NORMAL) {
		/* this is executed because type != 0 */
		self = __pthread_self();
		old = m->_m_lock;
		int own = old & 0x3fffffff;
		if (own != self->tid) <== TIDs could collide!!!
			return EPERM;
		if ((type&3) == PTHREAD_MUTEX_RECURSIVE && m-
>_m_count)
			return m->_m_count--, 0;
		^^^ not executed: type&3 = 0

		if ((type&4) && (old&0x40000000))
			new = 0x7fffffff;
		if (!priv) {
			/* this is executed */
			self->robust_list.pending = &m->_m_next;
			__vm_lock();
		}
		volatile void *prev = m->_m_prev;
		volatile void *next = m->_m_next;
		*(volatile void *volatile *)prev = next;
		if (next != &self->robust_list.head) *(volatile void
*volatile *)
			((char *)next - sizeof(void *)) = prev;
	}
	if (type&8) {
		/* this is NOT executed: not PI capable */
		if (old<0 || a_cas(&m->_m_lock, old, new)!=old) {
			if (new) a_store(&m->_m_waiters, -1);
			__syscall(SYS_futex, &m->_m_lock,
FUTEX_UNLOCK_PI|priv);
		}
		cont = 0;
		waiters = 0;
	} else {
		/* this is executed */
		cont = a_swap(&m->_m_lock, new);
	}
	if (type != PTHREAD_MUTEX_NORMAL && !priv) {
		/* this is executed */
		self->robust_list.pending = 0;
		__vm_unlock();
	}
	if (waiters || cont<0) 
		__wake(&m->_m_lock, 1, priv);
	return 0;
}

As mentioned by Rich, since TIDs are not unique across different
namespaces, a task might unlock a mutex hold by another one if they
have the same TID.

I don't see other possible errors, am I missing something?

Anyway, apart from the implementation which could improve or cover
corner cases, I have not found any paper which gives a clear statement
of how things should behave. Moreover it might be good to mention the
information that robust mutex protocol fundamentally does not work
across pid namespaces or with which limitations.

Daniele.

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Florian Weimer]

* Daniele Personal:

> On Tue, 2025-02-04 at 13:53 -0500, Rich Felker wrote:
>> On Mon, Feb 03, 2025 at 06:25:41PM +0100, Florian Weimer wrote:
>> > * Daniele Personal:
>> > 
>> > > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
>> > > > * Daniele Personal:
>> > > > 
>> > > > > > Is this required for implementing the unlock-if-not-owner
>> > > > > > error
>> > > > > > code
>> > > > > > on mutex unlock?
>> > > > > 
>> > > > > No, I don't see problems related to EOWNERDEAD.
>> > > > 
>> > > > Sorry, what I meant is that the TID is needed for efficient
>> > > > reporting
>> > > > of
>> > > > usage errors.  It's not imposed by the robust list protocol as
>> > > > such..
>> > > > There could be a PID-namespace-compatible robust mutex type
>> > > > that does
>> > > > not have this problem (but with less error checking).
>> > > > 
>> > > > Thanks,
>> > > > Florian
>> > > > 
>> > > 
>> > > Are you saying that there are pthread_mutexes which can be shared
>> > > across processes run on different pid namespaces? If yes I'm
>> > > definitely
>> > > interested on this. Can you tell me something more?
>> > 
>> > You would have to add a new mutex type that is a mix of
>> > PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the
>> > latter,
>> > but without the ownership checks.
>> 
>> This is inaccurate. Robust mutexes fundamentally depend on having the
>> owner's tid in the owner field, and on this value not matching the
>> tid of any other task that might hold the mutex. If these properties
>> don't hold, the mutex may fail to unlock when the owner dies, or
>> incorrectly unlock when another task mimicking the owner dies.
>> 
>> The Linux robust mutex protocol fundamentally does not work across
>> pid namespaces.

Thank you, Rich, for the correction.

> Looking at the code for musl 1.2.4, a pthread_mutex_t which has been
> initialized as shared and robust but not PI capable leaves uncovered
> only the case of pthread_mutex_unlock().

> As mentioned by Rich, since TIDs are not unique across different
> namespaces, a task might unlock a mutex hold by another one if they
> have the same TID.
>
> I don't see other possible errors, am I missing something?

The kernel code uses the owner TID to handle some special cases:

	/*
	 * Special case for regular (non PI) futexes. The unlock path in
	 * user space has two race scenarios:
	 *
	 * 1. The unlock path releases the user space futex value and
	 *    before it can execute the futex() syscall to wake up
	 *    waiters it is killed.
	 *
	 * 2. A woken up waiter is killed before it can acquire the
	 *    futex in user space.
	 *
	 * In the second case, the wake up notification could be generated
	 * by the unlock path in user space after setting the futex value
	 * to zero or by the kernel after setting the OWNER_DIED bit below.
	 *
	 * In both cases the TID validation below prevents a wakeup of
	 * potential waiters which can cause these waiters to block
	 * forever.
	 *
	 * In both cases the following conditions are met:
	 *
	 *	1) task->robust_list->list_op_pending != NULL
	 *	   @pending_op == true
	 *	2) The owner part of user space futex value == 0
	 *	3) Regular futex: @pi == false
	 *
	 * If these conditions are met, it is safe to attempt waking up a
	 * potential waiter without touching the user space futex value and
	 * trying to set the OWNER_DIED bit. If the futex value is zero,
	 * the rest of the user space mutex state is consistent, so a woken
	 * waiter will just take over the uncontended futex. Setting the
	 * OWNER_DIED bit would create inconsistent state and malfunction
	 * of the user space owner died handling. Otherwise, the OWNER_DIED
	 * bit is already set, and the woken waiter is expected to deal with
	 * this.
	 */
	owner = uval & FUTEX_TID_MASK;

	if (pending_op && !pi && !owner) {
		futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, 1,
			   FUTEX_BITSET_MATCH_ANY);
		return 0;
	}

As a result, it's definitely just a userspace-only change if you need to
use the robust mutex list across PID namespaces.

Thanks,
Florian

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Personal]

On Wed, 2025-02-05 at 11:32 +0100, Florian Weimer wrote:
> * Daniele Personal:
> 
> > On Tue, 2025-02-04 at 13:53 -0500, Rich Felker wrote:
> > > On Mon, Feb 03, 2025 at 06:25:41PM +0100, Florian Weimer wrote:
> > > > * Daniele Personal:
> > > > 
> > > > > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> > > > > > * Daniele Personal:
> > > > > > 
> > > > > > > > Is this required for implementing the unlock-if-not-
> > > > > > > > owner
> > > > > > > > error
> > > > > > > > code
> > > > > > > > on mutex unlock?
> > > > > > > 
> > > > > > > No, I don't see problems related to EOWNERDEAD.
> > > > > > 
> > > > > > Sorry, what I meant is that the TID is needed for efficient
> > > > > > reporting
> > > > > > of
> > > > > > usage errors.  It's not imposed by the robust list protocol
> > > > > > as
> > > > > > such..
> > > > > > There could be a PID-namespace-compatible robust mutex type
> > > > > > that does
> > > > > > not have this problem (but with less error checking).
> > > > > > 
> > > > > > Thanks,
> > > > > > Florian
> > > > > > 
> > > > > 
> > > > > Are you saying that there are pthread_mutexes which can be
> > > > > shared
> > > > > across processes run on different pid namespaces? If yes I'm
> > > > > definitely
> > > > > interested on this. Can you tell me something more?
> > > > 
> > > > You would have to add a new mutex type that is a mix of
> > > > PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the
> > > > latter,
> > > > but without the ownership checks.
> > > 
> > > This is inaccurate. Robust mutexes fundamentally depend on having
> > > the
> > > owner's tid in the owner field, and on this value not matching
> > > the
> > > tid of any other task that might hold the mutex. If these
> > > properties
> > > don't hold, the mutex may fail to unlock when the owner dies, or
> > > incorrectly unlock when another task mimicking the owner dies.
> > > 
> > > The Linux robust mutex protocol fundamentally does not work
> > > across
> > > pid namespaces.
> 
> Thank you, Rich, for the correction.
> 
> > Looking at the code for musl 1.2.4, a pthread_mutex_t which has
> > been
> > initialized as shared and robust but not PI capable leaves
> > uncovered
> > only the case of pthread_mutex_unlock().
> 
> > As mentioned by Rich, since TIDs are not unique across different
> > namespaces, a task might unlock a mutex hold by another one if they
> > have the same TID.
> > 
> > I don't see other possible errors, am I missing something?
> 
> The kernel code uses the owner TID to handle some special cases:
> 
> 	/*
> 	 * Special case for regular (non PI) futexes. The unlock
> path in
> 	 * user space has two race scenarios:
> 	 *
> 	 * 1. The unlock path releases the user space futex value
> and
> 	 *    before it can execute the futex() syscall to wake up
> 	 *    waiters it is killed.
> 	 *
> 	 * 2. A woken up waiter is killed before it can acquire the
> 	 *    futex in user space.
> 	 *
> 	 * In the second case, the wake up notification could be
> generated
> 	 * by the unlock path in user space after setting the futex
> value
> 	 * to zero or by the kernel after setting the OWNER_DIED bit
> below.
> 	 *
> 	 * In both cases the TID validation below prevents a wakeup
> of
> 	 * potential waiters which can cause these waiters to block
> 	 * forever.
> 	 *
> 	 * In both cases the following conditions are met:
> 	 *
> 	 *	1) task->robust_list->list_op_pending != NULL
> 	 *	   @pending_op == true
> 	 *	2) The owner part of user space futex value == 0
> 	 *	3) Regular futex: @pi == false
> 	 *
> 	 * If these conditions are met, it is safe to attempt waking
> up a
> 	 * potential waiter without touching the user space futex
> value and
> 	 * trying to set the OWNER_DIED bit. If the futex value is
> zero,
> 	 * the rest of the user space mutex state is consistent, so
> a woken
> 	 * waiter will just take over the uncontended futex. Setting
> the
> 	 * OWNER_DIED bit would create inconsistent state and
> malfunction
> 	 * of the user space owner died handling. Otherwise, the
> OWNER_DIED
> 	 * bit is already set, and the woken waiter is expected to
> deal with
> 	 * this.
> 	 */
> 	owner = uval & FUTEX_TID_MASK;
> 
> 	if (pending_op && !pi && !owner) {
> 		futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, 1,
> 			   FUTEX_BITSET_MATCH_ANY);
> 		return 0;
> 	}
> 
> As a result, it's definitely just a userspace-only change if you need
> to
> use the robust mutex list across PID namespaces.
> 

I tried to understand what you mean here but can't: can you please
explain me which userspace-only change is needed?

> Thanks,
> Florian
> 

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Rich Felker]

On Thu, Feb 06, 2025 at 08:45:14AM +0100, Daniele Personal wrote:
> On Wed, 2025-02-05 at 11:32 +0100, Florian Weimer wrote:
> > * Daniele Personal:
> > 
> > > On Tue, 2025-02-04 at 13:53 -0500, Rich Felker wrote:
> > > > On Mon, Feb 03, 2025 at 06:25:41PM +0100, Florian Weimer wrote:
> > > > > * Daniele Personal:
> > > > > 
> > > > > > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> > > > > > > * Daniele Personal:
> > > > > > > 
> > > > > > > > > Is this required for implementing the unlock-if-not-
> > > > > > > > > owner
> > > > > > > > > error
> > > > > > > > > code
> > > > > > > > > on mutex unlock?
> > > > > > > > 
> > > > > > > > No, I don't see problems related to EOWNERDEAD.
> > > > > > > 
> > > > > > > Sorry, what I meant is that the TID is needed for efficient
> > > > > > > reporting
> > > > > > > of
> > > > > > > usage errors.  It's not imposed by the robust list protocol
> > > > > > > as
> > > > > > > such..
> > > > > > > There could be a PID-namespace-compatible robust mutex type
> > > > > > > that does
> > > > > > > not have this problem (but with less error checking).
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Florian
> > > > > > > 
> > > > > > 
> > > > > > Are you saying that there are pthread_mutexes which can be
> > > > > > shared
> > > > > > across processes run on different pid namespaces? If yes I'm
> > > > > > definitely
> > > > > > interested on this. Can you tell me something more?
> > > > > 
> > > > > You would have to add a new mutex type that is a mix of
> > > > > PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the
> > > > > latter,
> > > > > but without the ownership checks.
> > > > 
> > > > This is inaccurate. Robust mutexes fundamentally depend on having
> > > > the
> > > > owner's tid in the owner field, and on this value not matching
> > > > the
> > > > tid of any other task that might hold the mutex. If these
> > > > properties
> > > > don't hold, the mutex may fail to unlock when the owner dies, or
> > > > incorrectly unlock when another task mimicking the owner dies.
> > > > 
> > > > The Linux robust mutex protocol fundamentally does not work
> > > > across
> > > > pid namespaces.
> > 
> > Thank you, Rich, for the correction.
> > 
> > > Looking at the code for musl 1.2.4, a pthread_mutex_t which has
> > > been
> > > initialized as shared and robust but not PI capable leaves
> > > uncovered
> > > only the case of pthread_mutex_unlock().
> > 
> > > As mentioned by Rich, since TIDs are not unique across different
> > > namespaces, a task might unlock a mutex hold by another one if they
> > > have the same TID.
> > > 
> > > I don't see other possible errors, am I missing something?
> > 
> > The kernel code uses the owner TID to handle some special cases:
> > 
> > 	/*
> > 	 * Special case for regular (non PI) futexes. The unlock
> > path in
> > 	 * user space has two race scenarios:
> > 	 *
> > 	 * 1. The unlock path releases the user space futex value
> > and
> > 	 *    before it can execute the futex() syscall to wake up
> > 	 *    waiters it is killed.
> > 	 *
> > 	 * 2. A woken up waiter is killed before it can acquire the
> > 	 *    futex in user space.
> > 	 *
> > 	 * In the second case, the wake up notification could be
> > generated
> > 	 * by the unlock path in user space after setting the futex
> > value
> > 	 * to zero or by the kernel after setting the OWNER_DIED bit
> > below.
> > 	 *
> > 	 * In both cases the TID validation below prevents a wakeup
> > of
> > 	 * potential waiters which can cause these waiters to block
> > 	 * forever.
> > 	 *
> > 	 * In both cases the following conditions are met:
> > 	 *
> > 	 *	1) task->robust_list->list_op_pending != NULL
> > 	 *	   @pending_op == true
> > 	 *	2) The owner part of user space futex value == 0
> > 	 *	3) Regular futex: @pi == false
> > 	 *
> > 	 * If these conditions are met, it is safe to attempt waking
> > up a
> > 	 * potential waiter without touching the user space futex
> > value and
> > 	 * trying to set the OWNER_DIED bit. If the futex value is
> > zero,
> > 	 * the rest of the user space mutex state is consistent, so
> > a woken
> > 	 * waiter will just take over the uncontended futex. Setting
> > the
> > 	 * OWNER_DIED bit would create inconsistent state and
> > malfunction
> > 	 * of the user space owner died handling. Otherwise, the
> > OWNER_DIED
> > 	 * bit is already set, and the woken waiter is expected to
> > deal with
> > 	 * this.
> > 	 */
> > 	owner = uval & FUTEX_TID_MASK;
> > 
> > 	if (pending_op && !pi && !owner) {
> > 		futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, 1,
> > 			   FUTEX_BITSET_MATCH_ANY);
> > 		return 0;
> > 	}
> > 
> > As a result, it's definitely just a userspace-only change if you need
> > to
> > use the robust mutex list across PID namespaces.
> > 
> 
> I tried to understand what you mean here but can't: can you please
> explain me which userspace-only change is needed?

No such change is possible. Robust futexes inherently rely on the
kernel being able to evaluate, on async process death, whether the
dying task was the owner of a mutex in the robust list. This depends
on the tid stored in memory being an accurate and unique identifier
for the task. If you violate this, you can hack things make the
userspace side work, but the whole robust functionality you want will
fail to work.

Rich

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Florian Weimer]

* Daniele Personal:

>> As a result, it's definitely [not] just a userspace-only change if
>> you need to use the robust mutex list across PID namespaces.
>> 
>
> I tried to understand what you mean here but can't: can you please
> explain me which userspace-only change is needed?

Sorry, there was a ”not” missing in the quoted conclusion.

Florian

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Dario]

[-- Attachment #1: Type: text/plain, Size: 6136 bytes --]

But wouldn't this mean that robust mutexes functionality is totally
incompatible with pid namespaces?
If the kernel relies on tid stored in memory by the process this always
lacks the information about the pid namespace the tid belongs to.

Daniele.


Il giorno ven 7 feb 2025 alle ore 17:19 Rich Felker <dalias@libc.org> ha
scritto:

> On Thu, Feb 06, 2025 at 08:45:14AM +0100, Daniele Personal wrote:
> > On Wed, 2025-02-05 at 11:32 +0100, Florian Weimer wrote:
> > > * Daniele Personal:
> > >
> > > > On Tue, 2025-02-04 at 13:53 -0500, Rich Felker wrote:
> > > > > On Mon, Feb 03, 2025 at 06:25:41PM +0100, Florian Weimer wrote:
> > > > > > * Daniele Personal:
> > > > > >
> > > > > > > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> > > > > > > > * Daniele Personal:
> > > > > > > >
> > > > > > > > > > Is this required for implementing the unlock-if-not-
> > > > > > > > > > owner
> > > > > > > > > > error
> > > > > > > > > > code
> > > > > > > > > > on mutex unlock?
> > > > > > > > >
> > > > > > > > > No, I don't see problems related to EOWNERDEAD.
> > > > > > > >
> > > > > > > > Sorry, what I meant is that the TID is needed for efficient
> > > > > > > > reporting
> > > > > > > > of
> > > > > > > > usage errors.  It's not imposed by the robust list protocol
> > > > > > > > as
> > > > > > > > such..
> > > > > > > > There could be a PID-namespace-compatible robust mutex type
> > > > > > > > that does
> > > > > > > > not have this problem (but with less error checking).
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Florian
> > > > > > > >
> > > > > > >
> > > > > > > Are you saying that there are pthread_mutexes which can be
> > > > > > > shared
> > > > > > > across processes run on different pid namespaces? If yes I'm
> > > > > > > definitely
> > > > > > > interested on this. Can you tell me something more?
> > > > > >
> > > > > > You would have to add a new mutex type that is a mix of
> > > > > > PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the
> > > > > > latter,
> > > > > > but without the ownership checks.
> > > > >
> > > > > This is inaccurate. Robust mutexes fundamentally depend on having
> > > > > the
> > > > > owner's tid in the owner field, and on this value not matching
> > > > > the
> > > > > tid of any other task that might hold the mutex. If these
> > > > > properties
> > > > > don't hold, the mutex may fail to unlock when the owner dies, or
> > > > > incorrectly unlock when another task mimicking the owner dies.
> > > > >
> > > > > The Linux robust mutex protocol fundamentally does not work
> > > > > across
> > > > > pid namespaces.
> > >
> > > Thank you, Rich, for the correction.
> > >
> > > > Looking at the code for musl 1.2.4, a pthread_mutex_t which has
> > > > been
> > > > initialized as shared and robust but not PI capable leaves
> > > > uncovered
> > > > only the case of pthread_mutex_unlock().
> > >
> > > > As mentioned by Rich, since TIDs are not unique across different
> > > > namespaces, a task might unlock a mutex hold by another one if they
> > > > have the same TID.
> > > >
> > > > I don't see other possible errors, am I missing something?
> > >
> > > The kernel code uses the owner TID to handle some special cases:
> > >
> > >     /*
> > >      * Special case for regular (non PI) futexes. The unlock
> > > path in
> > >      * user space has two race scenarios:
> > >      *
> > >      * 1. The unlock path releases the user space futex value
> > > and
> > >      *    before it can execute the futex() syscall to wake up
> > >      *    waiters it is killed.
> > >      *
> > >      * 2. A woken up waiter is killed before it can acquire the
> > >      *    futex in user space.
> > >      *
> > >      * In the second case, the wake up notification could be
> > > generated
> > >      * by the unlock path in user space after setting the futex
> > > value
> > >      * to zero or by the kernel after setting the OWNER_DIED bit
> > > below.
> > >      *
> > >      * In both cases the TID validation below prevents a wakeup
> > > of
> > >      * potential waiters which can cause these waiters to block
> > >      * forever.
> > >      *
> > >      * In both cases the following conditions are met:
> > >      *
> > >      *      1) task->robust_list->list_op_pending != NULL
> > >      *         @pending_op == true
> > >      *      2) The owner part of user space futex value == 0
> > >      *      3) Regular futex: @pi == false
> > >      *
> > >      * If these conditions are met, it is safe to attempt waking
> > > up a
> > >      * potential waiter without touching the user space futex
> > > value and
> > >      * trying to set the OWNER_DIED bit. If the futex value is
> > > zero,
> > >      * the rest of the user space mutex state is consistent, so
> > > a woken
> > >      * waiter will just take over the uncontended futex. Setting
> > > the
> > >      * OWNER_DIED bit would create inconsistent state and
> > > malfunction
> > >      * of the user space owner died handling. Otherwise, the
> > > OWNER_DIED
> > >      * bit is already set, and the woken waiter is expected to
> > > deal with
> > >      * this.
> > >      */
> > >     owner = uval & FUTEX_TID_MASK;
> > >
> > >     if (pending_op && !pi && !owner) {
> > >             futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, 1,
> > >                        FUTEX_BITSET_MATCH_ANY);
> > >             return 0;
> > >     }
> > >
> > > As a result, it's definitely just a userspace-only change if you need
> > > to
> > > use the robust mutex list across PID namespaces.
> > >
> >
> > I tried to understand what you mean here but can't: can you please
> > explain me which userspace-only change is needed?
>
> No such change is possible. Robust futexes inherently rely on the
> kernel being able to evaluate, on async process death, whether the
> dying task was the owner of a mutex in the robust list. This depends
> on the tid stored in memory being an accurate and unique identifier
> for the task. If you violate this, you can hack things make the
> userspace side work, but the whole robust functionality you want will
> fail to work.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 8645 bytes --]

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Rich Felker]

On Sat, Feb 08, 2025 at 10:20:45AM +0100, Daniele Dario wrote:
> But wouldn't this mean that robust mutexes functionality is totally
> incompatible with pid namespaces?

No, only with trying to synchronize *across* different pid namespaces.

> If the kernel relies on tid stored in memory by the process this always
> lacks the information about the pid namespace the tid belongs to.

It's necessarily within the same pid namespace as the process itself.

Functionally, you should consider different pid namespaces as
different systems that happen to be capable of sharing some resources.

Rich



> Il giorno ven 7 feb 2025 alle ore 17:19 Rich Felker <dalias@libc.org> ha
> scritto:
> 
> > On Thu, Feb 06, 2025 at 08:45:14AM +0100, Daniele Personal wrote:
> > > On Wed, 2025-02-05 at 11:32 +0100, Florian Weimer wrote:
> > > > * Daniele Personal:
> > > >
> > > > > On Tue, 2025-02-04 at 13:53 -0500, Rich Felker wrote:
> > > > > > On Mon, Feb 03, 2025 at 06:25:41PM +0100, Florian Weimer wrote:
> > > > > > > * Daniele Personal:
> > > > > > >
> > > > > > > > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> > > > > > > > > * Daniele Personal:
> > > > > > > > >
> > > > > > > > > > > Is this required for implementing the unlock-if-not-
> > > > > > > > > > > owner
> > > > > > > > > > > error
> > > > > > > > > > > code
> > > > > > > > > > > on mutex unlock?
> > > > > > > > > >
> > > > > > > > > > No, I don't see problems related to EOWNERDEAD.
> > > > > > > > >
> > > > > > > > > Sorry, what I meant is that the TID is needed for efficient
> > > > > > > > > reporting
> > > > > > > > > of
> > > > > > > > > usage errors.  It's not imposed by the robust list protocol
> > > > > > > > > as
> > > > > > > > > such..
> > > > > > > > > There could be a PID-namespace-compatible robust mutex type
> > > > > > > > > that does
> > > > > > > > > not have this problem (but with less error checking).
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Florian
> > > > > > > > >
> > > > > > > >
> > > > > > > > Are you saying that there are pthread_mutexes which can be
> > > > > > > > shared
> > > > > > > > across processes run on different pid namespaces? If yes I'm
> > > > > > > > definitely
> > > > > > > > interested on this. Can you tell me something more?
> > > > > > >
> > > > > > > You would have to add a new mutex type that is a mix of
> > > > > > > PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the
> > > > > > > latter,
> > > > > > > but without the ownership checks.
> > > > > >
> > > > > > This is inaccurate. Robust mutexes fundamentally depend on having
> > > > > > the
> > > > > > owner's tid in the owner field, and on this value not matching
> > > > > > the
> > > > > > tid of any other task that might hold the mutex. If these
> > > > > > properties
> > > > > > don't hold, the mutex may fail to unlock when the owner dies, or
> > > > > > incorrectly unlock when another task mimicking the owner dies.
> > > > > >
> > > > > > The Linux robust mutex protocol fundamentally does not work
> > > > > > across
> > > > > > pid namespaces.
> > > >
> > > > Thank you, Rich, for the correction.
> > > >
> > > > > Looking at the code for musl 1.2.4, a pthread_mutex_t which has
> > > > > been
> > > > > initialized as shared and robust but not PI capable leaves
> > > > > uncovered
> > > > > only the case of pthread_mutex_unlock().
> > > >
> > > > > As mentioned by Rich, since TIDs are not unique across different
> > > > > namespaces, a task might unlock a mutex hold by another one if they
> > > > > have the same TID.
> > > > >
> > > > > I don't see other possible errors, am I missing something?
> > > >
> > > > The kernel code uses the owner TID to handle some special cases:
> > > >
> > > >     /*
> > > >      * Special case for regular (non PI) futexes. The unlock
> > > > path in
> > > >      * user space has two race scenarios:
> > > >      *
> > > >      * 1. The unlock path releases the user space futex value
> > > > and
> > > >      *    before it can execute the futex() syscall to wake up
> > > >      *    waiters it is killed.
> > > >      *
> > > >      * 2. A woken up waiter is killed before it can acquire the
> > > >      *    futex in user space.
> > > >      *
> > > >      * In the second case, the wake up notification could be
> > > > generated
> > > >      * by the unlock path in user space after setting the futex
> > > > value
> > > >      * to zero or by the kernel after setting the OWNER_DIED bit
> > > > below.
> > > >      *
> > > >      * In both cases the TID validation below prevents a wakeup
> > > > of
> > > >      * potential waiters which can cause these waiters to block
> > > >      * forever.
> > > >      *
> > > >      * In both cases the following conditions are met:
> > > >      *
> > > >      *      1) task->robust_list->list_op_pending != NULL
> > > >      *         @pending_op == true
> > > >      *      2) The owner part of user space futex value == 0
> > > >      *      3) Regular futex: @pi == false
> > > >      *
> > > >      * If these conditions are met, it is safe to attempt waking
> > > > up a
> > > >      * potential waiter without touching the user space futex
> > > > value and
> > > >      * trying to set the OWNER_DIED bit. If the futex value is
> > > > zero,
> > > >      * the rest of the user space mutex state is consistent, so
> > > > a woken
> > > >      * waiter will just take over the uncontended futex. Setting
> > > > the
> > > >      * OWNER_DIED bit would create inconsistent state and
> > > > malfunction
> > > >      * of the user space owner died handling. Otherwise, the
> > > > OWNER_DIED
> > > >      * bit is already set, and the woken waiter is expected to
> > > > deal with
> > > >      * this.
> > > >      */
> > > >     owner = uval & FUTEX_TID_MASK;
> > > >
> > > >     if (pending_op && !pi && !owner) {
> > > >             futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, 1,
> > > >                        FUTEX_BITSET_MATCH_ANY);
> > > >             return 0;
> > > >     }
> > > >
> > > > As a result, it's definitely just a userspace-only change if you need
> > > > to
> > > > use the robust mutex list across PID namespaces.
> > > >
> > >
> > > I tried to understand what you mean here but can't: can you please
> > > explain me which userspace-only change is needed?
> >
> > No such change is possible. Robust futexes inherently rely on the
> > kernel being able to evaluate, on async process death, whether the
> > dying task was the owner of a mutex in the robust list. This depends
> > on the tid stored in memory being an accurate and unique identifier
> > for the task. If you violate this, you can hack things make the
> > userspace side work, but the whole robust functionality you want will
> > fail to work.
> >
> > Rich
> >

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Daniele Dario]

[-- Attachment #1: Type: text/plain, Size: 7596 bytes --]

Il sab 8 feb 2025, 13:39 Rich Felker <dalias@libc.org> ha scritto:

> On Sat, Feb 08, 2025 at 10:20:45AM +0100, Daniele Dario wrote:
> > But wouldn't this mean that robust mutexes functionality is totally
> > incompatible with pid namespaces?
>
> No, only with trying to synchronize *across* different pid namespaces.
>
> > If the kernel relies on tid stored in memory by the process this always
> > lacks the information about the pid namespace the tid belongs to.
>
> It's necessarily within the same pid namespace as the process itself.
>
> Functionally, you should consider different pid namespaces as
> different systems that happen to be capable of sharing some resources.
>
> Rich
>

Yes, I'm just saying that sharing pthread_mutex_t instances across
processes within the same pid namespace but on a system with more than a
pid namespace could lead to issues anyway if the stored tid value is used
by the kernel as who to contact without the knowledge of on which pid
namespace.

I not saying this is true, I'm trying to understand and if possible,
improve things.

Daniele


>
>
> > Il giorno ven 7 feb 2025 alle ore 17:19 Rich Felker <dalias@libc.org> ha
> > scritto:
> >
> > > On Thu, Feb 06, 2025 at 08:45:14AM +0100, Daniele Personal wrote:
> > > > On Wed, 2025-02-05 at 11:32 +0100, Florian Weimer wrote:
> > > > > * Daniele Personal:
> > > > >
> > > > > > On Tue, 2025-02-04 at 13:53 -0500, Rich Felker wrote:
> > > > > > > On Mon, Feb 03, 2025 at 06:25:41PM +0100, Florian Weimer wrote:
> > > > > > > > * Daniele Personal:
> > > > > > > >
> > > > > > > > > On Sat, 2025-02-01 at 17:03 +0100, Florian Weimer wrote:
> > > > > > > > > > * Daniele Personal:
> > > > > > > > > >
> > > > > > > > > > > > Is this required for implementing the unlock-if-not-
> > > > > > > > > > > > owner
> > > > > > > > > > > > error
> > > > > > > > > > > > code
> > > > > > > > > > > > on mutex unlock?
> > > > > > > > > > >
> > > > > > > > > > > No, I don't see problems related to EOWNERDEAD.
> > > > > > > > > >
> > > > > > > > > > Sorry, what I meant is that the TID is needed for
> efficient
> > > > > > > > > > reporting
> > > > > > > > > > of
> > > > > > > > > > usage errors.  It's not imposed by the robust list
> protocol
> > > > > > > > > > as
> > > > > > > > > > such..
> > > > > > > > > > There could be a PID-namespace-compatible robust mutex
> type
> > > > > > > > > > that does
> > > > > > > > > > not have this problem (but with less error checking).
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Florian
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Are you saying that there are pthread_mutexes which can be
> > > > > > > > > shared
> > > > > > > > > across processes run on different pid namespaces? If yes
> I'm
> > > > > > > > > definitely
> > > > > > > > > interested on this. Can you tell me something more?
> > > > > > > >
> > > > > > > > You would have to add a new mutex type that is a mix of
> > > > > > > > PTHREAD_MUTEX_NORMAL amd PTHREAD_MUTEX_ROBUST.  Closer to the
> > > > > > > > latter,
> > > > > > > > but without the ownership checks.
> > > > > > >
> > > > > > > This is inaccurate. Robust mutexes fundamentally depend on
> having
> > > > > > > the
> > > > > > > owner's tid in the owner field, and on this value not matching
> > > > > > > the
> > > > > > > tid of any other task that might hold the mutex. If these
> > > > > > > properties
> > > > > > > don't hold, the mutex may fail to unlock when the owner dies,
> or
> > > > > > > incorrectly unlock when another task mimicking the owner dies.
> > > > > > >
> > > > > > > The Linux robust mutex protocol fundamentally does not work
> > > > > > > across
> > > > > > > pid namespaces.
> > > > >
> > > > > Thank you, Rich, for the correction.
> > > > >
> > > > > > Looking at the code for musl 1.2.4, a pthread_mutex_t which has
> > > > > > been
> > > > > > initialized as shared and robust but not PI capable leaves
> > > > > > uncovered
> > > > > > only the case of pthread_mutex_unlock().
> > > > >
> > > > > > As mentioned by Rich, since TIDs are not unique across different
> > > > > > namespaces, a task might unlock a mutex hold by another one if
> they
> > > > > > have the same TID.
> > > > > >
> > > > > > I don't see other possible errors, am I missing something?
> > > > >
> > > > > The kernel code uses the owner TID to handle some special cases:
> > > > >
> > > > >     /*
> > > > >      * Special case for regular (non PI) futexes. The unlock
> > > > > path in
> > > > >      * user space has two race scenarios:
> > > > >      *
> > > > >      * 1. The unlock path releases the user space futex value
> > > > > and
> > > > >      *    before it can execute the futex() syscall to wake up
> > > > >      *    waiters it is killed.
> > > > >      *
> > > > >      * 2. A woken up waiter is killed before it can acquire the
> > > > >      *    futex in user space.
> > > > >      *
> > > > >      * In the second case, the wake up notification could be
> > > > > generated
> > > > >      * by the unlock path in user space after setting the futex
> > > > > value
> > > > >      * to zero or by the kernel after setting the OWNER_DIED bit
> > > > > below.
> > > > >      *
> > > > >      * In both cases the TID validation below prevents a wakeup
> > > > > of
> > > > >      * potential waiters which can cause these waiters to block
> > > > >      * forever.
> > > > >      *
> > > > >      * In both cases the following conditions are met:
> > > > >      *
> > > > >      *      1) task->robust_list->list_op_pending != NULL
> > > > >      *         @pending_op == true
> > > > >      *      2) The owner part of user space futex value == 0
> > > > >      *      3) Regular futex: @pi == false
> > > > >      *
> > > > >      * If these conditions are met, it is safe to attempt waking
> > > > > up a
> > > > >      * potential waiter without touching the user space futex
> > > > > value and
> > > > >      * trying to set the OWNER_DIED bit. If the futex value is
> > > > > zero,
> > > > >      * the rest of the user space mutex state is consistent, so
> > > > > a woken
> > > > >      * waiter will just take over the uncontended futex. Setting
> > > > > the
> > > > >      * OWNER_DIED bit would create inconsistent state and
> > > > > malfunction
> > > > >      * of the user space owner died handling. Otherwise, the
> > > > > OWNER_DIED
> > > > >      * bit is already set, and the woken waiter is expected to
> > > > > deal with
> > > > >      * this.
> > > > >      */
> > > > >     owner = uval & FUTEX_TID_MASK;
> > > > >
> > > > >     if (pending_op && !pi && !owner) {
> > > > >             futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, 1,
> > > > >                        FUTEX_BITSET_MATCH_ANY);
> > > > >             return 0;
> > > > >     }
> > > > >
> > > > > As a result, it's definitely just a userspace-only change if you
> need
> > > > > to
> > > > > use the robust mutex list across PID namespaces.
> > > > >
> > > >
> > > > I tried to understand what you mean here but can't: can you please
> > > > explain me which userspace-only change is needed?
> > >
> > > No such change is possible. Robust futexes inherently rely on the
> > > kernel being able to evaluate, on async process death, whether the
> > > dying task was the owner of a mutex in the robust list. This depends
> > > on the tid stored in memory being an accurate and unique identifier
> > > for the task. If you violate this, you can hack things make the
> > > userspace side work, but the whole robust functionality you want will
> > > fail to work.
> > >
> > > Rich
> > >
>

[-- Attachment #2: Type: text/html, Size: 11432 bytes --]

Re: [musl] pthread_mutex_t shared between processes with different pid namespaces

[Rich Felker]

On Sat, Feb 08, 2025 at 03:40:18PM +0100, Daniele Dario wrote:
> Il sab 8 feb 2025, 13:39 Rich Felker <dalias@libc.org> ha scritto:
> 
> > On Sat, Feb 08, 2025 at 10:20:45AM +0100, Daniele Dario wrote:
> > > But wouldn't this mean that robust mutexes functionality is totally
> > > incompatible with pid namespaces?
> >
> > No, only with trying to synchronize *across* different pid namespaces.
> >
> > > If the kernel relies on tid stored in memory by the process this always
> > > lacks the information about the pid namespace the tid belongs to.
> >
> > It's necessarily within the same pid namespace as the process itself.
> >
> > Functionally, you should consider different pid namespaces as
> > different systems that happen to be capable of sharing some resources.
> >
> > Rich
> >
> 
> Yes, I'm just saying that sharing pthread_mutex_t instances across
> processes within the same pid namespace but on a system with more than a
> pid namespace could lead to issues anyway if the stored tid value is used
> by the kernel as who to contact without the knowledge of on which pid
> namespace.
> 
> I not saying this is true, I'm trying to understand and if possible,
> improve things.

That's not a problem. The stored tid is used only in the context of a
process exiting, where the kernel code knows the relevant pid
namespace (the one the exiting process is in) and uses the tid
relative to that. If it didn't work this way, it would be a fatal bug
in the pid namespace implementation, which is supposed to allow
essentially transparent containerization (which includes processes in
the ns being able to use their tids as they could if they were outside
of any container/in global ns).

Rich