Re: [musl] Suggestion for thread safety

[Joakim Sindholt <opensource@zhasha.com>]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

On Mon, 28 Feb 2022 14:43:36 +0000, Lee Shallis <gb2985@gmail.com> wrote:
> Seems the wait just wasn't long enough, at about 4 yields onwards the
> results become consistent success, I've attached the file I did the
> experiments in, I even tried it under -O3 and no exits were
> encountered, so yes my method works, just needs a bit more wait time
> for extreme cases

Between the lines
> if ( !(shared->tid) )
and
> shared->tid = tid;
the kernel might suspend the running thread and allow the other to run,
or you might simply get unlucky and have the two threads do the checks
close enough to simultaneously that the memory hasn't been synchronized
yet. Either way you end up with both threads seeing that shared->tid is
zero and both of them writing their tids to it, and thus both enter the
critical section at the same time. And so the lock fails at the very
first hurdle: mutual exclusion. No amount of sleeping will make the bug
go away, only slightly more difficult to trigger.
The point of the clock_nanosleep call was to force a reschedule while
holding the lock. This also increases the runtime inside the lock which
in this case increases the likelihood that the thread trying to take the
lock will be waiting for it and end up racing with the thread that
currently has it when it unlocks and tries to relock it.

Now that you've inserted lots of sched_yield()s your lock is not only
still broken (in more ways than the one we've been trying to get you to
understand) but also extremely slow.

As a hint for your future education: the first (and far from only) thing
you'll need is compare-and-swap, aka. CAS.
You can read up on this class of bugs if you'd like. It's called "Time
Of Check to Time Of Use" or "TOCTOU" for short.

I didn't even need to poke at the code this time as the code you sent
breaks just the same on my machine.

I hope you'll learn from this.

[-- Attachment #2: not-on-my-machine.txt --]
[-- Type: text/plain, Size: 2849 bytes --]

zhasha@wirbelwind /home/zhasha ; cat lock.c
#define _GNU_SOURCE
#include <stdbool.h>
#include <unistd.h>
#include <errno.h>
#include <time.h>
#include <sched.h>
#include <pthread.h>
#include <stdlib.h>
#include <stdio.h>

typedef unsigned int uint;
typedef struct _LOCK { uint num; pid_t tid; } LOCK;

void LockSiData( LOCK *shared )
{
        pid_t tid = gettid();
        uint const yield = 4;
        uint i;
        /*
        struct timespec ts = {0};
        ts.tv_nsec = 1;
        */

        while ( shared->tid != tid )
        {
                if ( !(shared->tid) )
                        shared->tid = tid;
                //clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, 0);
                for ( i = 0; i < yield; ++i )
                        sched_yield();
        }

        shared->num++;
}

void FreeSiData( LOCK *shared )
{
        pid_t tid = gettid();
        if ( shared->tid != tid )
                return;
        shared->num--;
        if ( shared->num )
                return;
        shared->tid = (pid_t)0;
        //sched_yield();
}

volatile uint quit = 0;
volatile uint data = 0;
LOCK lock = {0};

void* thread( void *ud )
{
        uint i;
        struct timespec ts = {0};
        ts.tv_nsec = 1;
        (void)ud;

        while ( quit < CLOCKS_PER_SEC )
        {
                LockSiData( &lock );

                i = data++;
                if (i != 0)
                {
                        flockfile( stdout );
                        printf("var = %u, expected 0\n", i);
                        funlockfile( stdout );
                        FreeSiData( &lock );
                        exit(1);
                }

                /* Why is needed for the test? */
                clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, 0);

                i = data--;
                if (i != 1)
                {
                        flockfile( stdout );
                        printf("var = %u, expected 1\n", i);
                        funlockfile( stdout );
                        FreeSiData( &lock );
                        exit(1);
                }

                ++quit;

                FreeSiData( &lock );
        }
        return ud;
}

int main()
{
        pthread_t pt;
        int i;

        for (i = 0; i < 2; i++)
        {
                if ((errno = pthread_create(&pt, 0, thread, 0)) != 0 )
                {
                        flockfile( stdout );
                        printf("pthread_create failed: %m\n");
                        funlockfile( stdout );
                        return 1;
                }
        }
        pthread_exit(0);
}
zhasha@wirbelwind /home/zhasha ; gcc -lpthread -O3 lock.c
zhasha@wirbelwind /home/zhasha ; ./a.out
var = 1, expected 0
zhasha@wirbelwind /home/zhasha ; ./a.out
var = 1, expected 0
var = 2, expected 1
zhasha@wirbelwind /home/zhasha ;