Synchronising threads in a loop - c

Question

I want each thread to synchronize at the end of every loop. I have a condition variable at the end, which sends the thread to sleep if the other threads have not reached the pseudo-barrier at the end of the thread. I keep getting a deadlock. Can you help me spot my mistake?
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <pthread.h>
pthread_cond_t continue_cond;
pthread_mutex_t continue_mut;
pthread_mutex_t waiting_threads_mut;
int num_waiting_threads = 0;
pthread_mutex_t working_threads_mut;
int num_working_threads = 0;
int AllThreadsHere() {
pthread_mutex_lock(&waiting_threads_mut);
pthread_mutex_lock(&working_threads_mut);
//printf("%d: %d\n", num_waiting_threads, num_working_threads);
int res = (num_waiting_threads == num_working_threads) ? 1 : 0;
pthread_mutex_unlock(&working_threads_mut);
pthread_mutex_unlock(&waiting_threads_mut);
return res;
}
// used to
void WorkerProcess(int* thread_id) {
int to_process_indices = 1000;
while (to_process_indices > 0) {
// do computation here
to_process_indices -= *(thread_id + 1);
//
pthread_cond_broadcast(&continue_cond);
// increment number of waiting threads
pthread_mutex_lock(&waiting_threads_mut);
++num_waiting_threads;
pthread_mutex_unlock(&waiting_threads_mut);
// this mutex is necessary so as to make the process of sleeping, and decrementing the number of waiting threads atomic.
// note that if a thread wakes up from sleeping, then this mutex is locked again, meaning, the process of decrementing the number of working threads cannot occur without it unlocking.
// this is very important, as an incoming thread may otherwise just finish its chunk
pthread_mutex_lock(&continue_mut);
while (AllThreadsHere() == 0) {
//printf("Thread %d sleeping\n", args->thread_id);
// waits for signal from incoming threads.
pthread_cond_wait(&continue_cond, &continue_mut);
//printf("Thread %d woken\n", args->thread_id);
}
pthread_mutex_unlock(&continue_mut);
// need to decrease the number of waiting threads.
pthread_mutex_lock(&waiting_threads_mut);
--num_waiting_threads;
pthread_mutex_unlock(&waiting_threads_mut);
}
pthread_mutex_lock(&continue_mut);
pthread_mutex_lock(&working_threads_mut);
--num_working_threads;
pthread_cond_broadcast(&continue_cond);
pthread_mutex_unlock(&working_threads_mut);
pthread_mutex_unlock(&continue_mut);
}
and here is my int main which simply initialises the mutexes, pthreads, and joins the launched threads at the end.
int main() {
const unsigned int NUM_THREADS = 3;
const double PRECISION = 0.1;
// make the space for worker threads.
pthread_t* worker_threads = malloc(NUM_THREADS * sizeof(pthread_t));
int* worker_ids = malloc(sizeof(int) * NUM_THREADS);
pthread_cond_init(&continue_cond, NULL);
pthread_mutex_init(&waiting_threads_mut, NULL);
pthread_mutex_init(&working_threads_mut, NULL);
pthread_mutex_init(&continue_mut, NULL);
for (unsigned int k = 0; k < NUM_THREADS; ++k) {
worker_ids[k] = k;
pthread_create(worker_threads + k, NULL, WorkerProcess, (void*)(worker_ids + k));
}
for (unsigned int k = 0; k < NUM_THREADS; ++k) {
pthread_join(worker_threads[k], NULL);
}
}

Answer 1

Throwing code together and moving it around until it seems to work is not software development, it is gambling.
Instead, consider the behaviour you want from your barrier.
In my opinion, the barrier needs a condition variable that threads can wait on, until the number of threads reaches the set number of threads, at which point all waiting threads are woken up and released to proceed.
If the iteration is fast enough, then it is possible that the first-released thread arrives again at the barrier before all threads waiting in the barrier have released. So, we need a second condition variable for such incoming threads, plus at least one flag to indicate when the barrier is still being released.
Let's create a structure to describe such a barrier. Because linux lets us initialize mutexes and condition variables statically, we'll also define a static initializer.
#include <stdlib.h>
#include <pthread.h>
#include <stdio.h>
enum {
BARRIER_RELEASED = (1 << 0),
};
typedef struct {
pthread_mutex_t lock;
// Threads that arrive at the barrier before it has released
// all threads blocked in it, will wait on 'incoming'.
pthread_cond_t incoming;
// Threads blocked on the barrier wait on 'waiting'.
pthread_cond_t waiting;
// Number of threads in the barrier, or release trigger limit.
int limit;
// Number of threads blocked on the barrier (waiting on 'waiting').
int count;
// Barrier state flags, one bit per flag. See BARRIER_ flag enums.
int state;
} barrier;
#define BARRIER_INITIALIZER(limit_) \
{ .lock = PTHREAD_MUTEX_INITIALIZER, \
.incoming = PTHREAD_COND_INITIALIZER, \
.waiting = PTHREAD_COND_INITIALIZER, \
.limit = limit_, \
.count = 0, \
.state = 0 }
Since we do not necessarily know the number of threads participating in the barrier at compile time, let's also define an init function:
void barrier_init(barrier *b, int limit)
{
if (!b) {
fprintf(stderr, "barrier_init(): No barrier (NULL) specified.\n");
exit(EXIT_FAILURE);
}
if (limit < 0) {
fprintf(stderr, "barrier_init(): Negative limit (%d) specified.\n", limit);
exit(EXIT_FAILURE);
}
pthread_mutex_init(&(b->lock), NULL); // Cannot fail in Linux
pthread_cond_init(&(b->incoming), NULL); // Cannot fail in Linux
pthread_cond_init(&(b->waiting), NULL); // Cannot fail in Linux
b->limit = limit;
b->count = 0;
b->state = 0;
}
In both the initializer and the init function, the limit is the (expected) number of threads participating in the barrier.
(If we have each thread register itself in the barrier, then the first thread might iterate several times by itself while the barrier thread count is 1, before any other threads have a chance of registering themselves also.)
Whenever a thread no longer wants to participate in a barrier, it needs to "leave", so that the other threads waiting in the barrier won't wait there forever for the missing thread.
// When a thread no longer participates in a barrier, it needs to leave,
// so that the rest of the threads can keep gathering at the barrier without hanging.
void barrier_leave(barrier *b)
{
if (!b) {
fprintf(stderr, "barrier_leave(): No barrier (NULL) specified.\n");
exit(EXIT_FAILURE);
}
pthread_mutex_lock(&(b->lock));
b->limit--;
// Decreasing the limit may release the threads waiting on the barrier!
if (!(b->state & BARRIER_RELEASED) && b->count >= b->limit) {
b->state |= BARRIER_RELEASED;
pthread_cond_broadcast(&(b->waiting));
}
pthread_mutex_unlock(&(b->lock));
}
For completeness, we can define a function so that if one does not know the limit beforehand, one can init the barrier to an impossibly large number of threads, say SIZE_MAX/2, and then re-set the limit to the actual number of threads. This way, the created threads will start normally but wait at the barrier:
void barrier_set_limit(barrier *b, int limit)
{
if (!b) {
fprintf(stderr, "barrier_set_limit(): No barrier (NULL) specified.\n");
exit(EXIT_FAILURE);
}
if (limit < 0) {
fprintf(stderr, "barrier_set_limit(): Invalid, negative limit (%d) set.\n", limit);
exit(EXIT_FAILURE);
}
pthread_mutex_lock(&(b->lock));
b->limit = limit;
// Decreasing the limit may release the threads waiting on the barrier!
if (!(b->state & BARRIER_RELEASED) && b->count >= b->limit) {
b->state |= BARRIER_RELEASED;
pthread_cond_broadcast(&(b->waiting));
}
pthread_mutex_unlock(&(b->lock));
}
The final function left is the waiting at the barrier.
// Wait/gather at a barrier.
void barrier_wait(barrier *b)
{
if (!b) {
fprintf(stderr, "barrier_wait(): No barrier (NULL) specified.\n");
exit(EXIT_FAILURE);
}
pthread_mutex_lock(&(b->lock));
// Wait, if the barrier is being released.
if (b->state & BARRIER_RELEASED)
pthread_cond_wait(&(b->incoming), &(b->lock));
b->count++;
if (b->count >= b->limit) {
// We filled the barrier: release.
b->state |= BARRIER_RELEASED;
pthread_cond_broadcast(&(b->waiting));
} else {
// Barrier wasn't full, so we wait.
pthread_cond_wait(&(b->waiting), &(b->lock));
}
// If we are the last thread out of the barrier, we need to update
// the barrier state, and let the incoming threads advance.
b->count--;
if (b->count <= 0) {
b->state &= ~(BARRIER_RELEASED);
pthread_cond_broadcast(&(b->incoming));
}
pthread_mutex_unlock(&(b->lock));
}
Quick testing indicates the above indeed works, as long as the barrier is first initialized to the number of threads, and each thread participating in the barrier calls barrier_leave() before it exits.
In other words, you have a global, say static barrier turnstile;, and before creating say n threads, you call barrier_init(&turnstile, n);. Within the thread worker function, inside your loop, you call barrier_wait(&turnstile); to synchronize those threads. If one of those threads wants or needs to exit, it should call barrier_leave(&turnstile); first.
As you can see, the code looks nothing like yours. Everything related to a barrier is contained within the barrier structure, and all functions needed to manipulate barriers start with a barrier_ prefix, and should be straightforward to understand.
The entire process here started at thinking about "What do I need?", then implementing each need step by step. While I was writing this answer, I did need to "go back one step", because I didn't initially remember that an incoming condition queue is also needed, in case the first released thread(s) reach the same barrier again before all waiting threads in it have been released. But, because I went about it constructively, one step at a time (instead of throwing a lot of code together and then trying to see if it compiles and works), it wasn't a huge change –– even though for simplicity I rewrote the functions from scratch, only keeping the error checks.