Multi-threading, consumers and producers - c

I have a problem with multithreading, since I'm new to this topic. Code below is code I've been given from my University. It was in few versions, and I understood most of them. But I don't really understand the nready.nready variable and all this thread condition. Can anyone describe how those two work here? And why can't I just synchronise work of threads via mutex?
#include "unpipc.h"
#define MAXNITEMS 1000000
#define MAXNTHREADS 100
/* globals shared by threads */
int nitems; /* read-only by producer and consumer */
int buff[MAXNITEMS];
struct {
pthread_mutex_t mutex;
pthread_cond_t cond;
int nput;
int nval;
int nready;
} nready = { PTHREAD_MUTEX_INITIALIZER, PTHREAD_COND_INITIALIZER };
void *produce(void *), *consume(void *);
/* include main */
int
main(int argc, char **argv)
{
int i, nthreads, count[MAXNTHREADS];
pthread_t tid_produce[MAXNTHREADS], tid_consume;
if (argc != 3)
err_quit("usage: prodcons5 <#items> <#threads>");
nitems = min(atoi(argv[1]), MAXNITEMS);
nthreads = min(atoi(argv[2]), MAXNTHREADS);
Set_concurrency(nthreads + 1);
/* 4create all producers and one consumer */
for (i = 0; i < nthreads; i++) {
count[i] = 0;
Pthread_create(&tid_produce[i], NULL, produce, &count[i]);
}
Pthread_create(&tid_consume, NULL, consume, NULL);
/* wait for all producers and the consumer */
for (i = 0; i < nthreads; i++) {
Pthread_join(tid_produce[i], NULL);
printf("count[%d] = %d\n", i, count[i]);
}
Pthread_join(tid_consume, NULL);
exit(0);
}
/* end main */
void *
produce(void *arg)
{
for ( ; ; ) {
Pthread_mutex_lock(&nready.mutex);
if (nready.nput >= nitems) {
Pthread_mutex_unlock(&nready.mutex);
return(NULL); /* array is full, we're done */
}
buff[nready.nput] = nready.nval;
nready.nput++;
nready.nval++;
nready.nready++;
Pthread_cond_signal(&nready.cond);
Pthread_mutex_unlock(&nready.mutex);
*((int *) arg) += 1;
}
}
/* include consume */
void *
consume(void *arg)
{
int i;
for (i = 0; i < nitems; i++) {
Pthread_mutex_lock(&nready.mutex);
while (nready.nready == 0)
Pthread_cond_wait(&nready.cond, &nready.mutex);
nready.nready--;
Pthread_mutex_unlock(&nready.mutex);
if (buff[i] != i)
printf("buff[%d] = %d\n", i, buff[i]);
}
return(NULL);
}
/* end consume */

pthread_mutex_lock(&nready.mutex);
while (nready.nready == 0)
pthread_cond_wait(&nready.cond, &nready.mutex);
nready.nready--;
pthread_mutex_unlock(&nready.mutex);
The whole point of this structure is to guarantee that the condition (nready.nready == 0) is still true when you execute the corresponding action (nready.nready--) or - if the condition is not satisfied - to wait until it is without using CPU time.
You could use a mutex only, to check that the condition is correct and to perform the corresponding action atomically. But if the condition is not satisfied, you wouldn't know what to do. Wait? Until when? Check it again? Release the mutex and re-check immediately after? That would be wasting CPU time...
pthread_cond_signal() and pthread_cond_wait() are here to solve this problem. You should check their man pages.
Briefly, what pthread_cond_wait does, is it puts the calling thread to sleep and release the mutex in an atomic way until it's signaled. So this is a blocking function. The thread can then be re-scheduled by calling signal or broadcast from a different thread. When the thread is signaled, it grabs the mutex again and exit the wait() function.
Ath this point you know that
your condition is true and
you hold the mutex.
So you can do whatever you need to do with your data.
Be careful though, you shouldn't call wait, if you're not sure that another thread will signal. This is a very common source of deadlocks.
When a thread received a signal, it's put on the list of threads that are ready to be scheduled. By the time the thread is actually executed, your condition (i.e. nread.nready == 0) may be false again. Hence the while (to recheck if the thread is waked).

"But I don't really understand the nready.nready variable"
this results from the struct instance being named 'nready' and there
is a field within the struct named 'nready'
IMO: a very poor design to have two different objects being given the same name
the nready field of the nready struct seems to be keeping track of the number of
items that have been 'produced'

1) The nready filed of struct nready is used to tack how many tasks are ready to consume, i.e., the remaining tasks in array buff. The nready.nready++; statement is only executed when producers put one new item in array buff, and the nready.nready--; is only executed when consume gets item out of buff. With is variable, programmer can always track how many tasks are there left to process.
2)
pthread_mutex_lock(&nready.mutex);
while (nready.nready == 0)
pthread_cond_wait(&nready.cond, &nready.mutex);
nready.nready--;
pthread_mutex_unlock(&nready.mutex);
The statements above are common condition variable usage. You can check
POSIX Threads Programming and Condition Variables for more about condition variables.
Why can't use mutex only? You can poll a mutex lock again and again. Obviously, it is CPU time consuming and may be hugely affect system performance. Instead, you want the consume to wait in sleep when there is no more items in buff, and to be awaken when producer puts new item in buff. The condition variable is acting as this role here. When there is no items (nready.nready==0), pthread_cond_wait() function puts the current thread into sleep, save the precious cpu time. When new items are arriving, Pthread_cond_signal() awakes the consumer.

Related

Synchronising threads in a loop

I want each thread to synchronize at the end of every loop. I have a condition variable at the end, which sends the thread to sleep if the other threads have not reached the pseudo-barrier at the end of the thread. I keep getting a deadlock. Can you help me spot my mistake?
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <pthread.h>
pthread_cond_t continue_cond;
pthread_mutex_t continue_mut;
pthread_mutex_t waiting_threads_mut;
int num_waiting_threads = 0;
pthread_mutex_t working_threads_mut;
int num_working_threads = 0;
int AllThreadsHere() {
pthread_mutex_lock(&waiting_threads_mut);
pthread_mutex_lock(&working_threads_mut);
//printf("%d: %d\n", num_waiting_threads, num_working_threads);
int res = (num_waiting_threads == num_working_threads) ? 1 : 0;
pthread_mutex_unlock(&working_threads_mut);
pthread_mutex_unlock(&waiting_threads_mut);
return res;
}
// used to
void WorkerProcess(int* thread_id) {
int to_process_indices = 1000;
while (to_process_indices > 0) {
// do computation here
to_process_indices -= *(thread_id + 1);
//
pthread_cond_broadcast(&continue_cond);
// increment number of waiting threads
pthread_mutex_lock(&waiting_threads_mut);
++num_waiting_threads;
pthread_mutex_unlock(&waiting_threads_mut);
// this mutex is necessary so as to make the process of sleeping, and decrementing the number of waiting threads atomic.
// note that if a thread wakes up from sleeping, then this mutex is locked again, meaning, the process of decrementing the number of working threads cannot occur without it unlocking.
// this is very important, as an incoming thread may otherwise just finish its chunk
pthread_mutex_lock(&continue_mut);
while (AllThreadsHere() == 0) {
//printf("Thread %d sleeping\n", args->thread_id);
// waits for signal from incoming threads.
pthread_cond_wait(&continue_cond, &continue_mut);
//printf("Thread %d woken\n", args->thread_id);
}
pthread_mutex_unlock(&continue_mut);
// need to decrease the number of waiting threads.
pthread_mutex_lock(&waiting_threads_mut);
--num_waiting_threads;
pthread_mutex_unlock(&waiting_threads_mut);
}
pthread_mutex_lock(&continue_mut);
pthread_mutex_lock(&working_threads_mut);
--num_working_threads;
pthread_cond_broadcast(&continue_cond);
pthread_mutex_unlock(&working_threads_mut);
pthread_mutex_unlock(&continue_mut);
}
and here is my int main which simply initialises the mutexes, pthreads, and joins the launched threads at the end.
int main() {
const unsigned int NUM_THREADS = 3;
const double PRECISION = 0.1;
// make the space for worker threads.
pthread_t* worker_threads = malloc(NUM_THREADS * sizeof(pthread_t));
int* worker_ids = malloc(sizeof(int) * NUM_THREADS);
pthread_cond_init(&continue_cond, NULL);
pthread_mutex_init(&waiting_threads_mut, NULL);
pthread_mutex_init(&working_threads_mut, NULL);
pthread_mutex_init(&continue_mut, NULL);
for (unsigned int k = 0; k < NUM_THREADS; ++k) {
worker_ids[k] = k;
pthread_create(worker_threads + k, NULL, WorkerProcess, (void*)(worker_ids + k));
}
for (unsigned int k = 0; k < NUM_THREADS; ++k) {
pthread_join(worker_threads[k], NULL);
}
}
Throwing code together and moving it around until it seems to work is not software development, it is gambling.
Instead, consider the behaviour you want from your barrier.
In my opinion, the barrier needs a condition variable that threads can wait on, until the number of threads reaches the set number of threads, at which point all waiting threads are woken up and released to proceed.
If the iteration is fast enough, then it is possible that the first-released thread arrives again at the barrier before all threads waiting in the barrier have released. So, we need a second condition variable for such incoming threads, plus at least one flag to indicate when the barrier is still being released.
Let's create a structure to describe such a barrier. Because linux lets us initialize mutexes and condition variables statically, we'll also define a static initializer.
#include <stdlib.h>
#include <pthread.h>
#include <stdio.h>
enum {
BARRIER_RELEASED = (1 << 0),
};
typedef struct {
pthread_mutex_t lock;
// Threads that arrive at the barrier before it has released
// all threads blocked in it, will wait on 'incoming'.
pthread_cond_t incoming;
// Threads blocked on the barrier wait on 'waiting'.
pthread_cond_t waiting;
// Number of threads in the barrier, or release trigger limit.
int limit;
// Number of threads blocked on the barrier (waiting on 'waiting').
int count;
// Barrier state flags, one bit per flag. See BARRIER_ flag enums.
int state;
} barrier;
#define BARRIER_INITIALIZER(limit_) \
{ .lock = PTHREAD_MUTEX_INITIALIZER, \
.incoming = PTHREAD_COND_INITIALIZER, \
.waiting = PTHREAD_COND_INITIALIZER, \
.limit = limit_, \
.count = 0, \
.state = 0 }
Since we do not necessarily know the number of threads participating in the barrier at compile time, let's also define an init function:
void barrier_init(barrier *b, int limit)
{
if (!b) {
fprintf(stderr, "barrier_init(): No barrier (NULL) specified.\n");
exit(EXIT_FAILURE);
}
if (limit < 0) {
fprintf(stderr, "barrier_init(): Negative limit (%d) specified.\n", limit);
exit(EXIT_FAILURE);
}
pthread_mutex_init(&(b->lock), NULL); // Cannot fail in Linux
pthread_cond_init(&(b->incoming), NULL); // Cannot fail in Linux
pthread_cond_init(&(b->waiting), NULL); // Cannot fail in Linux
b->limit = limit;
b->count = 0;
b->state = 0;
}
In both the initializer and the init function, the limit is the (expected) number of threads participating in the barrier.
(If we have each thread register itself in the barrier, then the first thread might iterate several times by itself while the barrier thread count is 1, before any other threads have a chance of registering themselves also.)
Whenever a thread no longer wants to participate in a barrier, it needs to "leave", so that the other threads waiting in the barrier won't wait there forever for the missing thread.
// When a thread no longer participates in a barrier, it needs to leave,
// so that the rest of the threads can keep gathering at the barrier without hanging.
void barrier_leave(barrier *b)
{
if (!b) {
fprintf(stderr, "barrier_leave(): No barrier (NULL) specified.\n");
exit(EXIT_FAILURE);
}
pthread_mutex_lock(&(b->lock));
b->limit--;
// Decreasing the limit may release the threads waiting on the barrier!
if (!(b->state & BARRIER_RELEASED) && b->count >= b->limit) {
b->state |= BARRIER_RELEASED;
pthread_cond_broadcast(&(b->waiting));
}
pthread_mutex_unlock(&(b->lock));
}
For completeness, we can define a function so that if one does not know the limit beforehand, one can init the barrier to an impossibly large number of threads, say SIZE_MAX/2, and then re-set the limit to the actual number of threads. This way, the created threads will start normally but wait at the barrier:
void barrier_set_limit(barrier *b, int limit)
{
if (!b) {
fprintf(stderr, "barrier_set_limit(): No barrier (NULL) specified.\n");
exit(EXIT_FAILURE);
}
if (limit < 0) {
fprintf(stderr, "barrier_set_limit(): Invalid, negative limit (%d) set.\n", limit);
exit(EXIT_FAILURE);
}
pthread_mutex_lock(&(b->lock));
b->limit = limit;
// Decreasing the limit may release the threads waiting on the barrier!
if (!(b->state & BARRIER_RELEASED) && b->count >= b->limit) {
b->state |= BARRIER_RELEASED;
pthread_cond_broadcast(&(b->waiting));
}
pthread_mutex_unlock(&(b->lock));
}
The final function left is the waiting at the barrier.
// Wait/gather at a barrier.
void barrier_wait(barrier *b)
{
if (!b) {
fprintf(stderr, "barrier_wait(): No barrier (NULL) specified.\n");
exit(EXIT_FAILURE);
}
pthread_mutex_lock(&(b->lock));
// Wait, if the barrier is being released.
if (b->state & BARRIER_RELEASED)
pthread_cond_wait(&(b->incoming), &(b->lock));
b->count++;
if (b->count >= b->limit) {
// We filled the barrier: release.
b->state |= BARRIER_RELEASED;
pthread_cond_broadcast(&(b->waiting));
} else {
// Barrier wasn't full, so we wait.
pthread_cond_wait(&(b->waiting), &(b->lock));
}
// If we are the last thread out of the barrier, we need to update
// the barrier state, and let the incoming threads advance.
b->count--;
if (b->count <= 0) {
b->state &= ~(BARRIER_RELEASED);
pthread_cond_broadcast(&(b->incoming));
}
pthread_mutex_unlock(&(b->lock));
}
Quick testing indicates the above indeed works, as long as the barrier is first initialized to the number of threads, and each thread participating in the barrier calls barrier_leave() before it exits.
In other words, you have a global, say static barrier turnstile;, and before creating say n threads, you call barrier_init(&turnstile, n);. Within the thread worker function, inside your loop, you call barrier_wait(&turnstile); to synchronize those threads. If one of those threads wants or needs to exit, it should call barrier_leave(&turnstile); first.
As you can see, the code looks nothing like yours. Everything related to a barrier is contained within the barrier structure, and all functions needed to manipulate barriers start with a barrier_ prefix, and should be straightforward to understand.
The entire process here started at thinking about "What do I need?", then implementing each need step by step. While I was writing this answer, I did need to "go back one step", because I didn't initially remember that an incoming condition queue is also needed, in case the first released thread(s) reach the same barrier again before all waiting threads in it have been released. But, because I went about it constructively, one step at a time (instead of throwing a lot of code together and then trying to see if it compiles and works), it wasn't a huge change –– even though for simplicity I rewrote the functions from scratch, only keeping the error checks.

Trying to understand pthread_cond_lock and pthread_cond_signal

So I'm trying to understand exactly how pthread_mutex_lock works.
My current understanding is that it unlocks the mutex and puts whatever thread is going though it to sleep. Sleep meaning that the thread is inactive and consuming no resources.
It then waits for a signal to go from asleep to blocked, meaning that the thread can no longer change any variables.
thread 1:
pthread_mutex_lock(&mutex);
while (!condition){
printf("Thread wating.\n");
pthread_cond_wait(&cond, &mutex);
printf("Thread awakened.\n");
fflush(stdout);
}
pthread_mutex_unlock(&mutex);
pthread_cond_signal(&condVar);
pthread_mutex_unlock(&mutex);
So basically in the sample above, the loop runs and runs and each iteration pthread_cond_wait checks if the condition of the loop is true. If it is then the cond_signal is sent and the thread is blocked so it can't manipulate any more data.
I'm really having trouble wrapping my head around this, I'd appreciate some input and feedback about how this works and whether or not I am beginning to understand this based on what I have above.
I've gone over this post but am still having trouble
First, a summary:
pthread_mutex_lock(&mutex):
If mutex is free, then this thread grabs it immediately.
If mutex is grabbed, then this thread waits until the mutex becomes free, and then grabs it.
pthread_mutex_trylock(&mutex):
If mutex is free, then this thread grabs it.
If mutex is grabbed, then the call returns immediately with EBUSY.
pthread_mutex_unlock(&mutex):
Releases mutex.
pthread_cond_signal(&cond):
Wake up one thread waiting on the condition variable cond.
pthread_cond_broadcast(&cond):
Wake up all threads waiting on the condition variable cond.
pthread_cond_wait(&cond, &mutex):
This must be called with mutex grabbed.
The calling thread will temporarily release mutex and wait on cond.
When cond is broadcast on, or signaled on and this thread happens to be the one woken up, then the calling thread will first re-grab the mutex, and then return from the call.
It is important to note that at all times, the calling thread either has mutex grabbed, or is waiting on cond. There is no interval in between.
Let's look at a practical, running example code. We'll create it along the lines of OP's code.
First, we'll use a structure to hold the parameters for each worker function. Since we'll want the mutex and the condition variable to be shared between threads, we'll use pointers.
#define _POSIX_C_SOURCE 200809L
#include <stdlib.h>
#include <pthread.h>
#include <limits.h>
#include <string.h>
#include <stdio.h>
#include <errno.h>
/* Worker function work. */
struct work {
pthread_t thread_id;
pthread_mutex_t *lock; /* Pointer to the mutex to use */
pthread_cond_t *wait; /* Pointer to the condition variable to use */
volatile int *done; /* Pointer to the flag to check */
FILE *out; /* Stream to output to */
long id; /* Identity of this thread */
unsigned long count; /* Number of times this thread iterated. */
};
The thread worker function receives a pointer to the above structure. Each thread iterates the loop once, then waits on the condition variable. When woken up, if the done flag is still zero, the thread iterates the loop. Otherwise, the thread exits.
/* Example worker function. */
void *worker(void *workptr)
{
struct work *const work = workptr;
pthread_mutex_lock(work->lock);
/* Loop as long as *done == 0: */
while (!*(work->done)) {
/* *(work->lock) is ours at this point. */
/* This is a new iteration. */
work->count++;
/* Do the work. */
fprintf(work->out, "Thread %ld iteration %lu\n", work->id, work->count);
fflush(work->out);
/* Wait for wakeup. */
pthread_cond_wait(work->wait, work->lock);
}
/* *(work->lock) is still ours, but we've been told that all work is done already. */
/* Release the mutex and be done. */
pthread_mutex_unlock(work->lock);
return NULL;
}
To run the above, we'll need a main() as well:
#ifndef THREADS
#define THREADS 4
#endif
int main(void)
{
pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t wait = PTHREAD_COND_INITIALIZER;
volatile int done = 0;
struct work w[THREADS];
char *line = NULL, *p;
size_t size = 0;
ssize_t len = 0;
unsigned long total;
pthread_attr_t attrs;
int i, err;
/* The worker functions require very little stack, but the default stack
size is huge. Limit that, to reduce the (virtual) memory use. */
pthread_attr_init(&attrs);
pthread_attr_setstacksize(&attrs, 2 * PTHREAD_STACK_MIN);
/* Grab the mutex so the threads will have to wait to grab it. */
pthread_mutex_lock(&lock);
/* Create THREADS worker threads. */
for (i = 0; i < THREADS; i++) {
/* All threads use the same mutex, condition variable, and done flag. */
w[i].lock = &lock;
w[i].wait = &wait;
w[i].done = &done;
/* All threads output to standard output. */
w[i].out = stdout;
/* The rest of the fields are thread-specific. */
w[i].id = i + 1;
w[i].count = 0;
err = pthread_create(&(w[i].thread_id), &attrs, worker, (void *)&(w[i]));
if (err) {
fprintf(stderr, "Cannot create thread %d of %d: %s.\n", i+1, THREADS, strerror(errno));
exit(EXIT_FAILURE); /* Exits the entire process, killing any other threads as well. */
}
}
fprintf(stderr, "The first character on each line controls the type of event:\n");
fprintf(stderr, " e, q exit\n");
fprintf(stderr, " s signal\n");
fprintf(stderr, " b broadcast\n");
fflush(stderr);
/* Let each thread grab the mutex now. */
pthread_mutex_unlock(&lock);
while (1) {
len = getline(&line, &size, stdin);
if (len < 1)
break;
/* Find the first character on the line, ignoring leading whitespace. */
p = line;
while ((p < line + len) && (*p == '\0' || *p == '\t' || *p == '\n' ||
*p == '\v' || *p == '\f' || *p == '\r' || *p == ' '))
p++;
/* Do the operation mentioned */
if (*p == 'e' || *p == 'E' || *p == 'q' || *p == 'Q')
break;
else
if (*p == 's' || *p == 'S')
pthread_cond_signal(&wait);
else
if (*p == 'b' || *p == 'B')
pthread_cond_broadcast(&wait);
}
/* It is time for the worker threads to be done. */
pthread_mutex_lock(&lock);
done = 1;
pthread_mutex_unlock(&lock);
/* To ensure all threads see the state of that flag,
we wake up all threads by broadcasting on the condition variable. */
pthread_cond_broadcast(&wait);
/* Reap all threds. */
for (i = 0; i < THREADS; i++)
pthread_join(w[i].thread_id, NULL);
/* Output the thread statistics. */
total = 0;
for (i = 0; i < THREADS; i++) {
total += w[i].count;
fprintf(stderr, "Thread %ld: %lu events.\n", w[i].id, w[i].count);
}
fprintf(stderr, "Total: %lu events.\n", total);
return EXIT_SUCCESS;
}
If you save the above as example.c, you can compile it to example using e.g. gcc -Wall -O2 example.c -lpthread -o example.
To get the correct intuitive grasp of the operations, run the example in a terminal, with the source code in a window next to it, and see how the execution progresses as you provide input.
You can even run commands like printf '%s\n' s s s b q | ./example to run a sequence of events in a quick succession, or printf 's\ns\ns\nb\nq\n' | ./example with even less time in between events.
After some experimentation, you'll hopefully find out that not all input events cause their respective action. This is because the exit event (q above) is not synchronous: it does not wait for all pending work to be done, but tells the threads to exit right then and there. That is why the number of events may vary even for the exact same input.
(Also, if you signal on the condition variable, and immediately broadcast on it, the threads tend to only get woken up once.)
You can mitigate that by delaying the exit, using e.g. (printf '%s\n' s s b s s s ; sleep 1 ; printf 'q\n' ) | ./example.
However, there are better ways. A condition variable is not suitable for countable events; it is really flag-like. A semaphore would work better, but then you should be careful to not overflow the semaphore; it can only be from 0 to SEM_VALUE_MAX, inclusive. (So, you could use a semaphore to represent the number of pending job, but probably not for the number of iterations done by each/all thread workers.) A queue for the work to do, in thread pool fashion, is the most common approach.
pthread_cond_wait() simply means that the current thread shall release the mutex and then waits on a condition. The trick here is that both happens atomically, so it cannot happen, that the thread has released the mutex and is not yet waiting on the condition or is already waiting on the condition and has not yet released the mutex. Either both has happened or none has happened.
pthread_cond_signal() simply wakes up any thread that is currently waiting on the signaled condition. The first thing the woken up thread will do is obtaining the mutex again, if it cannot obtain it (e.g. as the signaling thread is currently owning the mutex), it will block until it can. If multiple threads are waiting on the condition, pthread_cond_signal() just wakes up one of them, which one is not defined. If you want to wake up all the waiting threads, you must use pthread_cond_broadcast() instead; but of course they won't run at the same time as now each of them first requires to obtain the mutex and that will only be possible one after another.
pthread_cond_t has no state. If you signal a condition no thread is waiting for, then nothing will happen. It's not like this will set a flag internally and if later on some thread calls pthread_cond_wait(), it will be woken up immediately as there is a pending signal. pthread_cond_signal() only wakes up threads that are already waiting, that means these threads must have called pthread_cond_wait() prior to you calling pthread_cond_signal().
Here's some simple sample code. First a reader thread:
// === Thread 1 ===
// We want to process an item from a list.
// To make sure the list is not altered by one
// thread while another thread is accessing it,
// it is protected by a mutex.
pthread_mutex_lock(&listLock);
// Now nobody but us is allowed to access the list.
// But what if the list is empty?
while (list->count == 0) {
// As long as we hold the mutex, no other thread
// thread can add anything to the list. So we
// must release it. But we want to know as soon
// as another thread has changed it.
pthread_cond_wait(&listCondition, &listLock);
// When we get here, somebody has signaled the
// condition and we have the mutex again and
// thus are allowed to access the list. The list
// may however still be empty, as another thread
// may have already consumed the new item in case
// there are multiple readers and all are woken
// up, thus the while-loop. If the list is still
// empty, we just go back to sleep and wait again.
}
// If we get here, the list is not empty.
processListItem(list);
// Finally we release the mutex again.
pthread_mutex_unlock(&listLock);
And then a writer thread:
// === Thread 2 ===
// We want to add a new item to the list.
// To make sure that nobody is accessing the
// list while we do, we need to obtain the mutex.
pthread_mutex_lock(&listLock);
// Now nobody but us is allowed to access the list.
// Check if the list is empty.
bool listWasEmpty = (list->count == 0);
// We add our item.
addListItem(list, newItem);
// If the list was empty, one or even multiple
// threads may be waiting for us adding an item.
// So we should wake them up here.
if (listWasEmpty) {
// If any thread is waiting for that condition,
// wake it up as now there is an item to process.
pthread_cond_signal(&listCondition);
}
// Finally we must release the mutex again.
pthread_mutex_unlock(&listLock);
The code is written so that there can be any number of reader/writer threads. Signaling only if the list was empty (listWasEmpty) is just a performance optimization, the code would also work correctly if you always signal the condition after adding an item.

pthreads C program hangs at execution

I'm writing a program that performs gaussian elimination given an A and B matrix. I first grab divisor and multipliers, create pthreads which execute in gauss function which perform their operations on a single 'column'. Then I call main which generates new divisor and multipliers and passes back for another round of operations by the same threads. Using condition pthread vars to accomplish this.
The code hangs until I create a breakpoint which it then proceeds and finishes. Not sure what's holding it up. Could use some help.
#include <stdio.h>
#include <pthread.h>
//Need one mutex variable and two condition variables (one c var for
//communicating between threads, and one c var for communicating with main).
pthread_mutex_t mut = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
pthread_cond_t condM = PTHREAD_COND_INITIALIZER;
float arr[3][4] = {{2,-3,1, -22},{7,9,-3, 14},{6,7,2,91}};
float mults[3];
float divisor;
int num_items = 3;
void* gauss(void *mine)
{
int thread_count=0;
int x = *((int *)mine);
for(int i=0;i<num_items;i++)
{
/*do something*/
arr[i][x] = arr[i][x] / divisor;
for(int k=0;k<num_items;k++){
if(k!=i)
arr[k][x] -= mults[k] * arr[i][x];
}
/*lock || wait || signal*/
pthread_mutex_lock(&mut);
thread_count++;
if(thread_count < num_items)
pthread_cond_wait(&cond,&mut);
else
{
pthread_cond_signal(&condM);
pthread_cond_wait(&cond,&mut);
thread_count = 0;
}
pthread_mutex_unlock(&mut);
}
return NULL;
}
int main(int argc, const char * argv[]) {
int i, j;
pthread_t threadr[num_items+1]; /*thread id array */
int is[num_items+1];
printf("Test");
// /*input num items*/
// printf("input the number of items ");
// scanf("%d",&num_items);
//
// /*input A array*/
// printf("input A array\n");
// for(i=0;i<num_items;i++)
// for(j=0;j<num_items;j++)
// scanf("%f",&arr[i][j]);
//
// /*input B array*/
// printf("input B array\n");
// for(i=0;i<num_items;i++)
// scanf("%f",&arr[i][num_items]);
/*grab first divisor & multipliers*/
divisor = arr[0][0];
for(i=0;i<num_items;i++)
{
mults[i] = arr[i][0];
}
for(i=0;i<num_items+1;i++)
{
is[i]=i;
if(pthread_create(&threadr[i],NULL,gauss,(void *)&is[i]) != 0)
perror("Pthread_create fails");
}
for(i=1;i<num_items;i++)
{
pthread_mutex_lock(&mut);
pthread_cond_wait(&condM,&mut);
divisor = arr[i][i];
for(j=0;j<num_items;j++)
{
mults[j] = 1;
if(j != i)
mults[j] = arr[j][i];
}
pthread_cond_broadcast(&cond);
pthread_mutex_unlock(&mut);
}
printf("The X values are:\n");
for(i=0;i<num_items; i++) {
printf("%0.3f \n", arr[i][num_items]);
}
/*wait for all threads*/
for(i=0;i<num_items+1; i++)
if (pthread_join(threadr[i],NULL) != 0)
perror("Pthread_join fails");
return 0;
}
You have a race condition (at least one), and your code does not use its condition variables correctly. You can probably fix the former by fixing the latter. Also, I suspect that you intend gauss()'s local variable thread_count to be shared, but variables with no linkage are not shared.
First, the race condition. Consider the main thread: it starts the three other threads, and then locks the mutex and waits for condition variable condM to be signaled. But suppose the threads all manage to signal condM before the main thread starts waiting? Condition variable operations are immediate -- any signals to condM that occur before main() is waiting on it are lost.
Now let's shift gears to talk about condition variables. As the Linux manual for pthread_cond_wait() puts it:
When using condition variables there is always a Boolean predicate involving shared variables associated with each condition wait that is true if the thread should proceed. Spurious wakeups from the pthread_cond_timedwait() or pthread_cond_wait() functions may occur. Since the return from pthread_cond_timedwait() or pthread_cond_wait() does not imply anything about the value of this predicate, the predicate should be re-evaluated upon such return.
In other words, condition variables are used to suspend thread operations pending a given condition becoming true. In abstract terms, that condition is always "it's ok for this thread to proceed", but that's realized in context-specific terms. Most importantly, the fact that a thread wakes from its wait never inherently communicates that the condition is true; it merely indicates that the newly-woken thread should check whether the condition is true. Generally, the thread should also check before waiting for the first time, as the condition may already be true.
In pseudocode, that looks like this:
Thread 1:
lock mutex;
loop
if is_ok_to_proceed then exit loop;
wait on condition variable;
end loop
// ... maybe do mutex-protected work ...
unlock mutex
Thread 2:
lock mutex
// ... maybe do mutex-protected work ...
is_ok_to_proceed = true;
signal condition variable;
unlock mutex
Generally speaking, there is also (mutex-protected) code somewhere else that make the CV predicate false, so that sometimes the threads indeed do execute their waits.
Now consider how that applies to the race condition in main(). How does main() know whether to wait on condM()? There needs to be a shared variable somewhere that answers that for it, and its wait must be conditioned on the value of that variable. Any thread that means to allow the main thread to proceed must both set the appropriate value for the variable and signal condM. The main thread itself should set the variable, too, as needed, to indicate that it is not ready, at that time, to proceed.
Of course, your other CV usage suffers from the same kind of problem, too.

While loop synchronization

I am working on a project with a user defined number of threads I am using 7 at the moment. I have a while loop that runs in each thread but I need all of the threads to wait for each other at the end of the while loop. The tricky thing is that some of the threads do not all end on the same number of times through the loop.
void *entryFunc(void *param)
{
int *i = (int *)param;
int nextPrime;
int p = latestPrime;
while(latestPrime < testLim)
{
sem_wait(&sem);
nextPrime = findNextPrime(latestPrime);
if(nextPrime != -1)
{
latestPrime = nextPrime;
p = latestPrime;
}
else
{
sem_post(&sem);
break;
}
sem_post(&sem);
if(p < 46341)
{
incrementNotPrimes(p);
}
/*
sem_wait(&sem2);
doneCount++;
sem_post(&sem2);
while(go != 1);
sem_wait(&sem2);
doneCount--;
//sem_post(&sem3);
sem_post(&sem2);
*/
}
return NULL;
}
where the chunk of code is commented out is part of my last attempt at solving this problem. That is where the functions all need to wait for each other. I have a feeling I am missing something simple.
If your problem is that on each thread, the while loop has a different numbers of iterations and some threads never reach the synchronization point after exiting the loop, you could use a barrier. Check here for an example.
However you need to decrease the number of threads at the barrier after you exit each thread. Waiting at the barrier will end after count number of threads reached that point.
So you need to update the barrier object each time a thread finishes. And make sure you do this atomically.
As I mentioned in the comments, you should use a barrier instead of a semaphore for this kind of situation, as it should be simpler to implement (barriers have been designed exactly to solve that problem). However, you may still use a semaphore with a little bit of arithmetic
arithmetic: your goal is to have all thread execute the same code path, but somehow the last thread to finish its task should wake all the other threads up. One way to achieve that is to have at the end of the function an atomic counter which each thread would decrement, and if the counter reaches 0, the thread simply calls as many time sem_post as necessary to release all the waiting threads, instead of issuing a sem_wait as the others.
A second method, this time using only a semaphore, is also possible. Since we cannot differentiate the last thread for the others, all the threads must do the same operations with the semaphore, ie try to release everyone, but also wait for the last one. So the idea is to initialize the semaphore to (1-n)*(n+1), so that each of the n-1 first threads would fail at waking up their friends with n+1 calls to sem_post, but still work toward getting the semaphore at exactly 0. Then the last thread would do the same, pushing the semaphore value to n+1, thus releasing the locked threads, and leaving room for it to also perform its sem_wait and be released immediately.
void *entryFunc(void *param)
{
int *i = (int *)param;
int nextPrime;
int p = latestPrime, j;
while(latestPrime < testLim){
nextPrime = findNextPrime(latestPrime);
if(nextPrime != -1)
{
latestPrime = nextPrime;
p = latestPrime;
}
if(p < 46341)
{
incrementNotPrimes(p);
}
}
for (j=0;j<=THREAD_COUNT;j++)
sem_post(&sem);
sem_wait(&sem);
return NULL;
}
The problem with this approach is that it doesn't deal with how the semaphore should be reset in between uses (if your program needs to repeat this mechanism, it will need to reset the semaphore value, since it will end up being 1 after this code has been executed successfully).

The version of pthread_join() that does not block main(): POSIX

I am trying to write a code that does not block main() when pthread_join() is called:
i.e. basically trying to implement my previous question mentioned below:
https://stackoverflow.com/questions/24509500/pthread-join-and-main-blocking-multithreading
And the corresponding explanation at:
pthreads - Join on group of threads, wait for one to exit
As per suggested answer:
You'd need to create your own version of it - e.g. an array of flags (one flag per thread) protected by a mutex and a condition variable; where just before "pthread_exit()" each thread acquires the mutex, sets its flag, then does "pthread_cond_signal()". The main thread waits for the signal, then checks the array of flags to determine which thread/s to join (there may be more than one thread to join by then).
I have tried as below:
My status array which keeps a track of which threads have finished:
typedef struct {
int Finish_Status[THREAD_NUM];
int signalled;
pthread_mutex_t mutex;
pthread_cond_t FINISHED;
}THREAD_FINISH_STATE;
The thread routine, it sets the corresponding array element when the thread finishes and also signals the condition variable:
void* THREAD_ROUTINE(void* arg)
{
THREAD_ARGUMENT* temp=(THREAD_ARGUMENT*) arg;
printf("Thread created with id %d\n",temp->id);
waitFor(5);
pthread_mutex_lock(&(ThreadFinishStatus.mutex));
ThreadFinishStatus.Finish_Status[temp->id]=TRUE;
ThreadFinishStatus.signalled=TRUE;
if(ThreadFinishStatus.signalled==TRUE)
{
pthread_cond_signal(&(ThreadFinishStatus.FINISHED));
printf("Signal that thread %d finished\n",temp->id);
}
pthread_mutex_unlock(&(ThreadFinishStatus.mutex));
pthread_exit((void*)(temp->id));
}
I am not able to write the corresponding parts pthread_join() and pthread_cond_wait() functions. There are a few things which I am not able to implement.
1) How to write corresponding part pthread_cond_wait() in my main()?
2) I am trying to write it as:
pthread_mutex_lock(&(ThreadFinishStatus.mutex));
while((ThreadFinishStatus.signalled != TRUE){
pthread_cond_wait(&(ThreadFinishStatus.FINISHED), &(ThreadFinishStatus.mutex));
printf("Main Thread signalled\n");
ThreadFinishStatus.signalled==FALSE; //Reset signalled
//check which thread to join
}
pthread_mutex_unlock(&(ThreadFinishStatus.mutex));
But it does not enter the while loop.
3) How to use pthread_join() so that I can get the return value stored in my arg[i].returnStatus
i.e. where to put below statement in my main:
`pthread_join(T[i],&(arg[i].returnStatus));`
COMPLETE CODE
#include <stdio.h>
#include <pthread.h>
#include <time.h>
#define THREAD_NUM 5
#define FALSE 0
#define TRUE 1
void waitFor (unsigned int secs) {
time_t retTime;
retTime = time(0) + secs; // Get finishing time.
while (time(0) < retTime); // Loop until it arrives.
}
typedef struct {
int Finish_Status[THREAD_NUM];
int signalled;
pthread_mutex_t mutex;
pthread_cond_t FINISHED;
}THREAD_FINISH_STATE;
typedef struct {
int id;
void* returnStatus;
}THREAD_ARGUMENT;
THREAD_FINISH_STATE ThreadFinishStatus;
void initializeState(THREAD_FINISH_STATE* state)
{
int i=0;
state->signalled=FALSE;
for(i=0;i<THREAD_NUM;i++)
{
state->Finish_Status[i]=FALSE;
}
pthread_mutex_init(&(state->mutex),NULL);
pthread_cond_init(&(state->FINISHED),NULL);
}
void destroyState(THREAD_FINISH_STATE* state)
{
int i=0;
for(i=0;i<THREAD_NUM;i++)
{
state->Finish_Status[i]=FALSE;
}
pthread_mutex_destroy(&(state->mutex));
pthread_cond_destroy(&(state->FINISHED));
}
void* THREAD_ROUTINE(void* arg)
{
THREAD_ARGUMENT* temp=(THREAD_ARGUMENT*) arg;
printf("Thread created with id %d\n",temp->id);
waitFor(5);
pthread_mutex_lock(&(ThreadFinishStatus.mutex));
ThreadFinishStatus.Finish_Status[temp->id]=TRUE;
ThreadFinishStatus.signalled=TRUE;
if(ThreadFinishStatus.signalled==TRUE)
{
pthread_cond_signal(&(ThreadFinishStatus.FINISHED));
printf("Signal that thread %d finished\n",temp->id);
}
pthread_mutex_unlock(&(ThreadFinishStatus.mutex));
pthread_exit((void*)(temp->id));
}
int main()
{
THREAD_ARGUMENT arg[THREAD_NUM];
pthread_t T[THREAD_NUM];
int i=0;
initializeState(&ThreadFinishStatus);
for(i=0;i<THREAD_NUM;i++)
{
arg[i].id=i;
}
for(i=0;i<THREAD_NUM;i++)
{
pthread_create(&T[i],NULL,THREAD_ROUTINE,(void*)&arg[i]);
}
/*
Join only if signal received
*/
pthread_mutex_lock(&(ThreadFinishStatus.mutex));
//Wait
while((ThreadFinishStatus.signalled != TRUE){
pthread_cond_wait(&(ThreadFinishStatus.FINISHED), &(ThreadFinishStatus.mutex));
printf("Main Thread signalled\n");
ThreadFinishStatus.signalled==FALSE; //Reset signalled
//check which thread to join
}
pthread_mutex_unlock(&(ThreadFinishStatus.mutex));
destroyState(&ThreadFinishStatus);
return 0;
}
Here is an example of a program that uses a counting semaphore to watch as threads finish, find out which thread it was, and review some result data from that thread. This program is efficient with locks - waiters are not spuriously woken up (notice how the threads only post to the semaphore after they've released the mutex protecting shared state).
This design allows the main program to process the result from some thread's computation immediately after the thread completes, and does not require the main wait for all threads to complete. This would be especially helpful if the running time of each thread varied by a significant amount.
Most importantly, this program does not deadlock nor race.
#include <pthread.h>
#include <semaphore.h>
#include <stdlib.h>
#include <stdio.h>
#include <queue>
void* ThreadEntry(void* args );
typedef struct {
int threadId;
pthread_t thread;
int threadResult;
} ThreadState;
sem_t completionSema;
pthread_mutex_t resultMutex;
std::queue<int> threadCompletions;
ThreadState* threadInfos;
int main() {
int numThreads = 10;
int* threadResults;
void* threadResult;
int doneThreadId;
sem_init( &completionSema, 0, 0 );
pthread_mutex_init( &resultMutex, 0 );
threadInfos = new ThreadState[numThreads];
for ( int i = 0; i < numThreads; i++ ) {
threadInfos[i].threadId = i;
pthread_create( &threadInfos[i].thread, NULL, &ThreadEntry, &threadInfos[i].threadId );
}
for ( int i = 0; i < numThreads; i++ ) {
// Wait for any one thread to complete; ie, wait for someone
// to queue to the threadCompletions queue.
sem_wait( &completionSema );
// Find out what was queued; queue is accessed from multiple threads,
// so protect with a vanilla mutex.
pthread_mutex_lock(&resultMutex);
doneThreadId = threadCompletions.front();
threadCompletions.pop();
pthread_mutex_unlock(&resultMutex);
// Announce which thread ID we saw finish
printf(
"Main saw TID %d finish\n\tThe thread's result was %d\n",
doneThreadId,
threadInfos[doneThreadId].threadResult
);
// pthread_join to clean up the thread.
pthread_join( threadInfos[doneThreadId].thread, &threadResult );
}
delete threadInfos;
pthread_mutex_destroy( &resultMutex );
sem_destroy( &completionSema );
}
void* ThreadEntry(void* args ) {
int threadId = *((int*)args);
printf("hello from thread %d\n", threadId );
// This can safely be accessed since each thread has its own space
// and array derefs are thread safe.
threadInfos[threadId].threadResult = rand() % 1000;
pthread_mutex_lock( &resultMutex );
threadCompletions.push( threadId );
pthread_mutex_unlock( &resultMutex );
sem_post( &completionSema );
return 0;
}
Pthread conditions don't have "memory"; pthread_cond_wait doesn't return if pthread_cond_signal is called before pthread_cond_wait, which is why it's important to check the predicate before calling pthread_cond_wait, and not call it if it's true. But that means the action, in this case "check which thread to join" should only depend on the predicate, not on whether pthread_cond_wait is called.
Also, you might want to make the while loop actually wait for all the threads to terminate, which you aren't doing now.
(Also, I think the other answer about "signalled==FALSE" being harmless is wrong, it's not harmless, because there's a pthread_cond_wait, and when that returns, signalled would have changed to true.)
So if I wanted to write a program that waited for all threads to terminate this way, it would look more like
pthread_mutex_lock(&(ThreadFinishStatus.mutex));
// AllThreadsFinished would check that all of Finish_Status[] is true
// or something, or simpler, count the number of joins completed
while (!AllThreadsFinished()) {
// Wait, keeping in mind that the condition might already have been
// signalled, in which case it's too late to call pthread_cond_wait,
// but also keeping in mind that pthread_cond_wait can return spuriously,
// thus using a while loop
while (!ThreadFinishStatus.signalled) {
pthread_cond_wait(&(ThreadFinishStatus.FINISHED), &(ThreadFinishStatus.mutex));
}
printf("Main Thread signalled\n");
ThreadFinishStatus.signalled=FALSE; //Reset signalled
//check which thread to join
}
pthread_mutex_unlock(&(ThreadFinishStatus.mutex));
Your code is racy.
Suppose you start a thread and it finishes before you grab the mutex in main(). Your while loop will never run because signalled was already set to TRUE by the exiting thread.
I will echo #antiduh's suggestion to use a semaphore that counts the number of dead-but-not-joined threads. You then loop up to the number of threads spawned waiting on the semaphore. I'd point out that the POSIX sem_t is not like a pthread_mutex in that sem_wait can return EINTR.
Your code appears fine. You have one minor buglet:
ThreadFinishStatus.signalled==FALSE; //Reset signalled
This does nothing. It tests whether signalled is FALSE and throws away the result. That's harmless though since there's nothing you need to do. (You never want to set signalled to FALSE because that loses the fact that it was signalled. There is never any reason to unsignal it -- if a thread finished, then it's finished forever.)
Not entering the while loop means signalled is TRUE. That means the thread already set it, in which case there is no need to enter the loop because there's nothing to wait for. So that's fine.
Also:
ThreadFinishStatus.signalled=TRUE;
if(ThreadFinishStatus.signalled==TRUE)
There's no need to test the thing you just set. It's not like the set can fail.
FWIW, I would suggest re-architecting. If the existing functions like pthread_join don't do exactly what you want, just don't use them. If you're going to have structures that track what work is done, then totally separate that from thread termination. Since you will already know what work is done, what different does it make when and how threads terminate? Don't think of this as "I need a special way to know when a thread terminates" and instead think of this "I need to know what work is done so I can do other things".

Resources