[musl] Program terminated in __pthread_exit by signal SIGSEGV when accessing robust_list

[musl] Program terminated in __pthread_exit by signal SIGSEGV when accessing robust_list

[Bart Nys]

[-- Attachment #1: Type: text/plain, Size: 2141 bytes --]

Hi,

Our program is terminated by signal SIGSEGV when a detached thread is exiting.

Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_exit (result=<optimized out>) at src/thread/pthread_create.c:103
103               int waiters = m->_m_waiters;

The robust_list head pointer is not valid:
(gdb) list
98          __vm_lock();
99          volatile void *volatile *rp;
100         while ((rp=self->robust_list.head) && rp != &self->robust_list.head) {
101               pthread_mutex_t *m = (void *)((char *)rp
102                     - offsetof(pthread_mutex_t, _m_next));
103               int waiters = m->_m_waiters;
104               int priv = (m->_m_type & 128) ^ 128;
105               self->robust_list.pending = rp;
106               self->robust_list.head = *rp;
107               int cont = a_swap(&m->_m_lock, 0x40000000);
(gdb) p *self
$5 = {self = 0x7f7f947af8, dtv = 0x7f7f947bf0, prev = 0x7f7ff71af8, next = 0x7f8096aed8 <builtin_tls>, sysinfo = 0, canary = 3124346406391670196, canary2 = 0, tid = 18839, errno_val = 107,
  detach_state = 2, cancel = 0, canceldisable = 1 '\001', cancelasync = 0 '\000', tsd_used = 0 '\000', dlerror_flag = 0 '\000', map_base = 0x7f7f925000 "", map_size = 143360,
  stack = 0x7f7f947af8, stack_size = 133880, guard_size = 8192, result = 0x0, cancelbuf = 0x0, tsd = 0x7f7f947c00, robust_list = {head = 0x3c1, off = 0, pending = 0x0}, timer_id = 0,
  locale = 0x7f809697c0 <__libc+56>, killlock = {-2147483647}, dlerror_buf = 0x0, stdio_locks = 0x0, canary_at_end = 0, dtv_copy = 0x7f7f947bf0}

We are using  musl 1.1.24.

Our application has been running in the field for more than 10 years with glibc and never encountered this crash.

Any help that can point me in the right direction is greatly appreciated.

Bart.





[-- Attachment #2: Type: text/html, Size: 8350 bytes --]

Re: [musl] Program terminated in __pthread_exit by signal SIGSEGV when accessing robust_list

[Rich Felker]

On Tue, Apr 22, 2025 at 01:09:40PM +0000, Bart Nys wrote:
> Hi,
> 
> Our program is terminated by signal SIGSEGV when a detached thread is exiting.
> 
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  __pthread_exit (result=<optimized out>) at src/thread/pthread_create.c:103
> 103               int waiters = m->_m_waiters;
> 
> The robust_list head pointer is not valid:
> (gdb) list
> 98          __vm_lock();
> 99          volatile void *volatile *rp;
> 100         while ((rp=self->robust_list.head) && rp != &self->robust_list.head) {
> 101               pthread_mutex_t *m = (void *)((char *)rp
> 102                     - offsetof(pthread_mutex_t, _m_next));
> 103               int waiters = m->_m_waiters;
> 104               int priv = (m->_m_type & 128) ^ 128;
> 105               self->robust_list.pending = rp;
> 106               self->robust_list.head = *rp;
> 107               int cont = a_swap(&m->_m_lock, 0x40000000);
> (gdb) p *self
> $5 = {self = 0x7f7f947af8, dtv = 0x7f7f947bf0, prev = 0x7f7ff71af8, next = 0x7f8096aed8 <builtin_tls>, sysinfo = 0, canary = 3124346406391670196, canary2 = 0, tid = 18839, errno_val = 107,
>   detach_state = 2, cancel = 0, canceldisable = 1 '\001', cancelasync = 0 '\000', tsd_used = 0 '\000', dlerror_flag = 0 '\000', map_base = 0x7f7f925000 "", map_size = 143360,
>   stack = 0x7f7f947af8, stack_size = 133880, guard_size = 8192, result = 0x0, cancelbuf = 0x0, tsd = 0x7f7f947c00, robust_list = {head = 0x3c1, off = 0, pending = 0x0}, timer_id = 0,
>   locale = 0x7f809697c0 <__libc+56>, killlock = {-2147483647}, dlerror_buf = 0x0, stdio_locks = 0x0, canary_at_end = 0, dtv_copy = 0x7f7f947bf0}
> 
> We are using  musl 1.1.24.
> 
> Our application has been running in the field for more than 10 years
> with glibc and never encountered this crash.
> 
> Any help that can point me in the right direction is greatly
> appreciated.

This almost surely indicates that you program has clobbered memory in
one way or another, possibly related to freeing or reusing memory that
previously contained a locked recursive or errorchecking mutex without
unlocking it. I would start by inspecting any code relevant to
lifetimes of such mutexes. You might get lucky and find something by
running under valgrind, too.

Rich

Re: [musl] Program terminated in __pthread_exit by signal SIGSEGV when accessing robust_list

[Bart Nys]

[-- Attachment #1: Type: text/plain, Size: 3288 bytes --]

Hi Rich,

Thanks for the reply.

I indeed could find the problem with valgrind.
Issue was caused by destroying a recursive mutex that was still locked.

Bart.
________________________________
Van: Rich Felker <dalias@libc.org>
Verzonden: dinsdag 22 april 2025 15:24
Aan: Bart Nys <bart.nys@vantiva.com>
CC: musl@lists.openwall.com <musl@lists.openwall.com>
Onderwerp: Re: [musl] Program terminated in __pthread_exit by signal SIGSEGV when accessing robust_list

** CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **


On Tue, Apr 22, 2025 at 01:09:40PM +0000, Bart Nys wrote:
> Hi,
>
> Our program is terminated by signal SIGSEGV when a detached thread is exiting.
>
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  __pthread_exit (result=<optimized out>) at src/thread/pthread_create.c:103
> 103               int waiters = m->_m_waiters;
>
> The robust_list head pointer is not valid:
> (gdb) list
> 98          __vm_lock();
> 99          volatile void *volatile *rp;
> 100         while ((rp=self->robust_list.head) && rp != &self->robust_list.head) {
> 101               pthread_mutex_t *m = (void *)((char *)rp
> 102                     - offsetof(pthread_mutex_t, _m_next));
> 103               int waiters = m->_m_waiters;
> 104               int priv = (m->_m_type & 128) ^ 128;
> 105               self->robust_list.pending = rp;
> 106               self->robust_list.head = *rp;
> 107               int cont = a_swap(&m->_m_lock, 0x40000000);
> (gdb) p *self
> $5 = {self = 0x7f7f947af8, dtv = 0x7f7f947bf0, prev = 0x7f7ff71af8, next = 0x7f8096aed8 <builtin_tls>, sysinfo = 0, canary = 3124346406391670196, canary2 = 0, tid = 18839, errno_val = 107,
>   detach_state = 2, cancel = 0, canceldisable = 1 '\001', cancelasync = 0 '\000', tsd_used = 0 '\000', dlerror_flag = 0 '\000', map_base = 0x7f7f925000 "", map_size = 143360,
>   stack = 0x7f7f947af8, stack_size = 133880, guard_size = 8192, result = 0x0, cancelbuf = 0x0, tsd = 0x7f7f947c00, robust_list = {head = 0x3c1, off = 0, pending = 0x0}, timer_id = 0,
>   locale = 0x7f809697c0 <__libc+56>, killlock = {-2147483647}, dlerror_buf = 0x0, stdio_locks = 0x0, canary_at_end = 0, dtv_copy = 0x7f7f947bf0}
>
> We are using  musl 1.1.24.
>
> Our application has been running in the field for more than 10 years
> with glibc and never encountered this crash.
>
> Any help that can point me in the right direction is greatly
> appreciated.

This almost surely indicates that you program has clobbered memory in
one way or another, possibly related to freeing or reusing memory that
previously contained a locked recursive or errorchecking mutex without
unlocking it. I would start by inspecting any code relevant to
lifetimes of such mutexes. You might get lucky and find something by
running under valgrind, too.

Rich

[-- Attachment #2: Type: text/html, Size: 5710 bytes --]

Re: [musl] Program terminated in __pthread_exit by signal SIGSEGV when accessing robust_list

[Bart Nys]

[-- Attachment #1: Type: text/plain, Size: 3833 bytes --]

And freeing the memory containing that mutex, as you said.
________________________________
Van: Bart Nys <bart.nys@vantiva.com>
Verzonden: dinsdag 29 april 2025 11:35
Aan: Rich Felker <dalias@libc.org>
CC: musl@lists.openwall.com <musl@lists.openwall.com>
Onderwerp: Re: [musl] Program terminated in __pthread_exit by signal SIGSEGV when accessing robust_list

** CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **

Hi Rich,

Thanks for the reply.

I indeed could find the problem with valgrind.
Issue was caused by destroying a recursive mutex that was still locked.

Bart.
________________________________
Van: Rich Felker <dalias@libc.org>
Verzonden: dinsdag 22 april 2025 15:24
Aan: Bart Nys <bart.nys@vantiva.com>
CC: musl@lists.openwall.com <musl@lists.openwall.com>
Onderwerp: Re: [musl] Program terminated in __pthread_exit by signal SIGSEGV when accessing robust_list

** CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **


On Tue, Apr 22, 2025 at 01:09:40PM +0000, Bart Nys wrote:
> Hi,
>
> Our program is terminated by signal SIGSEGV when a detached thread is exiting.
>
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  __pthread_exit (result=<optimized out>) at src/thread/pthread_create.c:103
> 103               int waiters = m->_m_waiters;
>
> The robust_list head pointer is not valid:
> (gdb) list
> 98          __vm_lock();
> 99          volatile void *volatile *rp;
> 100         while ((rp=self->robust_list.head) && rp != &self->robust_list.head) {
> 101               pthread_mutex_t *m = (void *)((char *)rp
> 102                     - offsetof(pthread_mutex_t, _m_next));
> 103               int waiters = m->_m_waiters;
> 104               int priv = (m->_m_type & 128) ^ 128;
> 105               self->robust_list.pending = rp;
> 106               self->robust_list.head = *rp;
> 107               int cont = a_swap(&m->_m_lock, 0x40000000);
> (gdb) p *self
> $5 = {self = 0x7f7f947af8, dtv = 0x7f7f947bf0, prev = 0x7f7ff71af8, next = 0x7f8096aed8 <builtin_tls>, sysinfo = 0, canary = 3124346406391670196, canary2 = 0, tid = 18839, errno_val = 107,
>   detach_state = 2, cancel = 0, canceldisable = 1 '\001', cancelasync = 0 '\000', tsd_used = 0 '\000', dlerror_flag = 0 '\000', map_base = 0x7f7f925000 "", map_size = 143360,
>   stack = 0x7f7f947af8, stack_size = 133880, guard_size = 8192, result = 0x0, cancelbuf = 0x0, tsd = 0x7f7f947c00, robust_list = {head = 0x3c1, off = 0, pending = 0x0}, timer_id = 0,
>   locale = 0x7f809697c0 <__libc+56>, killlock = {-2147483647}, dlerror_buf = 0x0, stdio_locks = 0x0, canary_at_end = 0, dtv_copy = 0x7f7f947bf0}
>
> We are using  musl 1.1.24.
>
> Our application has been running in the field for more than 10 years
> with glibc and never encountered this crash.
>
> Any help that can point me in the right direction is greatly
> appreciated.

This almost surely indicates that you program has clobbered memory in
one way or another, possibly related to freeing or reusing memory that
previously contained a locked recursive or errorchecking mutex without
unlocking it. I would start by inspecting any code relevant to
lifetimes of such mutexes. You might get lucky and find something by
running under valgrind, too.

Rich

[-- Attachment #2: Type: text/html, Size: 7156 bytes --]