Ending pthreads with deferred cancelation - c

I'm new to pthreads and signal handling and I am working on a project that will create x amount of pthreads as either a producer or consumer, that execute forever, and I want to end all threads and then the main in a systematic way.
To do this, I am attempting to catch the ^c with a signal handler and then setting some global flag to then end the threads.
I have tried doing the following but I am unsure if it is working or not and I would like advice on my thought process and implementation.
Here is what I am working with, omitting error checking:
#include stdio, pthread, stdlib, semaphore, time, string, unistd, signal
sem_t empty, full;
pthread_mutex_t mutex;
int flag = 0;
void *producer();
void *consumer();
void sig_handler(int sig);
int main(int argc, char *argv[]){
signal(handler(SIGINT, sig_handler);
//init locks and semas
pthread_mutex_init(&mutex, NULL);
sem_init(&empty, 0, SOME_BUFFER_SIZE);
sem_init(&full, 0, 0);
pthread_t prod_threads[5]; //5 for example, can be any amount passed in
pthread_t cons_threads[3];
//start up threads
for(i = 0; i < 5; i++)
pthread_create(&prod_threads[i], NULL, producer, NULL)
for(i = 0; i < 3; i++)
pthread_create(&cons_threads[i], NULL, consumer, NULL)
//join threads at end
for(i = 0; i < PROD; i++)
pthread_join(&prod_threads[i], NULL);
for(i = 0; i < CONS; i++)
pthread_join(&cons_threads[i], NULL);
sleep(4);//could be any amount of time
exit(EXIT_SUCCESS);
}
//not sure if I am going about this the right way
void sig_handler(int sig){
if(sig == SIGINT)
flag = 1;
}
void *producer(){
while(1){
sleep(x);//where x is some random time
//part I am concerned about:
if(flag)
exit(EXIT_SUCCESS);
//get locks and semaphore stuff
//enter crit section
//release locks, semaphore
}
void *consumer(){
while(1){
sleep(x);//sleep some random time
//again not sure if this is right or not
if(flag)
exit(EXIT_SUCCESS);
//get locks
//enter crit
//release locks
}
}
}
My program seems to execute properly, I'm just not sure if I am using pthreads and signals correctly and would like some guidance there. If you need more complete code just let me know
Thanks

You're execution seems appropriate, however using the docs #yano provided, you're pthread_join() is off ever so slightly.
Instead of:
for(i = 0; i < PROD; i++)
pthread_join(&prod_threads[i], NULL);
for(i = 0; i < CONS; i++)
pthread_join(&cons_threads[i], NULL);
You need:
for(i = 0; i < PROD; i++)
pthread_join(prod_threads[i], NULL);
for(i = 0; i < CONS; i++)
pthread_join(cons_threads[i], NULL);
Notice the ampersand. Whether or not this is what you are looking for, I am not sure, but I hope it helps.

I want to end all threads and then the main in a systematic way.
As Martin James suggests, post a special thread message into the queue telling the worker threads to terminate. This message should not be removed from the queue, so that all worker threads have a chance to receive it and terminate.
The other point is that you should not call exit in your worker threads, since this function terminates the entire process, defeating your original goal. Just return from the thread function.
Another point is that in a multi-threaded application it is good practice to handle all signals in one specific thread (often the main thread is the best candidate for that). And block the signals you expect to receive in all other threads. It should be done by blocking the expected signals before creating other threads, so that they inherit the signal mask, and eventually unblocking these signals in the signal handling thread when it is ready to handle them (e.g. when all extra threads have been created). This is because a signal targeted for a process (such as SIGINT) can be delivered to any one of its threads, but we want it to be handled by one specific thread. See Signal Generation and Delivery for more details.

Related

pthread_cond_signal() not giving enough time for the signaled thread to run

The thread calling pthread_cond_signal is re-grabbing the mutex before the signaled thread can be released.
The code below shows a simple example of the issue at hand. The main thread will hold the lock, create the worker thread, and then enter a loop that prints data as it comes in. It is signaled to run via a conditional variable.
The worker thread will enter a loop that generates data, grabs the lock, writes the data, and then signals the main thread.
#include <stdio.h>
#include <pthread.h>
pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
int data = 0;
void *thread(void *arg) {
int length = *(int *) arg;
for (int i = 0; i < length; i++) {
// Do some work
pthread_mutex_lock(&lock);
data = i;
fprintf(stdout, "Writing\n");
pthread_cond_signal(&cond);
pthread_mutex_unlock(&lock);
}
pthread_exit(0);
}
int main(int argc, const char *argv[]) {
pthread_t worker;
int length = 4;
pthread_mutex_lock(&lock);
pthread_create(&worker, 0, thread, &length);
for (int i = 0; i < length; i++) {
fprintf(stdout, "Waiting\n");
pthread_cond_wait(&cond, &lock);
fprintf(stdout, "read data: %d\n", data);
}
pthread_mutex_unlock(&lock);
pthread_join(worker, NULL);
return 0;
}
This will give the following output:
Waiting
Writing
Writing
Writing
Writing
read data: 3
Waiting
Expected behavior:
Main thread holds the mutex and only releases it once its waiting. The worker thread will then write its data and signal the main thread. The main thread will immediately lock the mutex on signal, then read the data and go back to waiting, releasing the mutex. Meanwhile the worker thread will do its work and be waiting until the main thread is waiting again to write its data and signal it.
Instead, it appears that the worker thread is getting the mutex immediately after calling signal, rarely letting the main thread get access. If I put a sleep in the worker thread in place of the // Do some work then it will give the expected output.
Signalling the condition variable does not give any kind of priority on locking the mutex to a thread that was waiting on that condition variable. All it means is that at least one thread waiting on the condition variable will start trying to acquire the mutex so it can return from the pthread_cond_wait(). The signalling thread will keep executing and can easily re-acquire the mutex first, as you've seen.
You should never have a condition variable without an actual condition over some shared state that you're waiting for - returning from a pthread_cond_wait() doesn't mean a thread should definitely proceed, it means that it should check if the condition it was waiting for is true. That's why they're called condition variables.
In this case, the state your writing thread wants to wait for is "the main thread has consumed the last data I wrote.". However, your reading (main) thread also needs to wait on a condition - "the writing thread has written some new data". You can achieve both these conditions with a flag variable that indicates that some new, unconsumed data has been written to the data variable. The flag starts out unset, is set by the writing thread when it updates data, and is unset by the main thread when it reads from data. The writing thread waits for the flag to be unset, and the reading thread waits for the flag to be set.
With this arrangement, you also don't need to have the mutex locked when you start the writing thread - it doesn't matter which order the threads start, because everything is consistent either way.
The updated code looks like:
#include <stdio.h>
#include <pthread.h>
pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
int data = 0;
int data_available = 0;
void *thread(void *arg)
{
int length = *(int *) arg;
for (int i = 0; i < length; i++) {
// Do some work
pthread_mutex_lock(&lock);
fprintf(stdout, "Waiting to write\n");
while (data_available)
pthread_cond_wait(&cond, &lock);
fprintf(stdout, "Writing\n");
data = i;
data_available = 1;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&lock);
}
pthread_exit(0);
}
int main(int argc, const char *argv[])
{
pthread_t worker;
int length = 4;
pthread_create(&worker, 0, thread, &length);
for (int i = 0; i < length; i++) {
pthread_mutex_lock(&lock);
fprintf(stdout, "Waiting to read\n");
while (!data_available)
pthread_cond_wait(&cond, &lock);
fprintf(stdout, "read data: %d\n", data);
data_available = 0;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&lock);
}
pthread_join(worker, NULL);
return 0;
}
Of course, the threads end up working in lockstep - but essentially you have a producer-consumer with a maximum queue length of 1, so that's expected.

pthread_cond_broadcast not signalling threads

I'm really struggling with understanding how to lock and unlock mutexes in a way that lets threads run at the same time. Right now I'm trying to have every thread do something, then wait until all threads are ready, and then repeat it. This should happen repeatedly, so I have it all in a conditioned loop. My problem is, it seems like my broadcast is never received by any threads, as every thread that waits for the signal waits forever, while the thread that sent the signal continues.
I've dumbed my code down to make it as simple as possible while still being compileable and runnable:
#include <stdio.h>
#include <pthread.h>
#include <unistd.h>
#define num_threads 4
int repeat = 1;
int timesLooped = 0;
int waitingThreads = 0;
int threadNo = -1;
pthread_mutex_t mutex1;
pthread_cond_t condition;
//thread body
void* threadBody(void* arg){
//sensitive info: requires lock
if(pthread_mutex_lock(&mutex1)){ printf("error1\n"); }
threadNo++;
int threadId = threadNo;
//other info is set here as well in my code
if(pthread_mutex_unlock(&mutex1)){ printf("error2\n"); }
//main loop in the thread body
while(repeat == 1){
if(pthread_mutex_lock(&mutex1)){ printf("error3\n"); }
//wait until all threads are ready
while(waitingThreads < num_threads - 1){
printf(" %d! ", threadId);
waitingThreads++;
pthread_cond_wait(&condition, &mutex1);
printf(" %d ", threadId);
if(pthread_mutex_unlock(&mutex1)){ printf("error4\n"); }
}
//last thread will broadcast
if(waitingThreads == num_threads - 1){
printf("\n\nthread %d was last! broadcasting to let everyone proceed...", threadId);
timesLooped++;
waitingThreads = 0;
if(timesLooped == 3){
repeat = 0;
}
sleep(1);
pthread_cond_broadcast(&condition);
if(pthread_mutex_unlock(&mutex1)){ printf("error5\n"); }
}
}
printf("\n\nexiting thread %d\n", threadId);
pthread_exit((void*) arg);
}
int main(int argc, char** argv){
pthread_t threads[num_threads];
void* retval;
//create threads
for(long i = 0; i < num_threads; i++){
pthread_create(&threads[i], NULL, threadBody, (void*) i);
}
//join threads
for(long j = 0; j < num_threads; j++){
pthread_join(threads[j], &retval);
}
printf("\n\nDONE\n");
}
This gives me an output of:
thread 3 was last! broadcasting to let everyone proceed...
thread 2 was last! broadcasting to let everyone proceed...
thread 1 was last! broadcasting to let everyone proceed...
exiting thread 1 (deadlock, other threads never exit)
There are at least two bugs in your program.
You have a data race (see this blog post).
When threads 0 through num_threads - 1 access waitingThreads in if (waitingThreads == num_threads - 1) ..., they do so outside of the lock (i.e. they race).
You don't allow threads other than the last one to run (this is your primary question).
pthread_cond_wait returns when both condition has been signaled and the mutex can be re-acquired.
But your last thread immediately re-acquires the mutex upon releasing it, so no other thread can proceed.
I expect that if you add sleep or usleep after pthread_mutex_unlock in the last thread, your program will start working as you expect it to.

Why some threads don't receive pthread_cond_broadcast?

I have a threadpool of workers. Each worker executes this routine:
void* worker(void* args){
...
pthread_mutex_lock(&mtx);
while (queue == NULL && stop == 0){
pthread_cond_wait(&cond, &mtx);
}
el = pop(queue);
pthread_mutex_unlock(&mtx);
...
}
main thread:
int main(){
...
while (stop == 0){
...
pthread_mutex_lock(&mtx);
insert(queue, el);
pthread_cond_signal(&cond);
pthread_mutex_unlock(&mtx);
...
}
...
}
Then I have a signal handler that executes this code when it receives a signal:
void exit_handler(){
stop = 1;
pthread_mutex_lock(&mtx);
pthread_cond_broadcast(&cond);
pthread_mutex_unlock(&mtx);
}
I have omitted declarations and initialization, but the original code has them.
After a signal is received most of the time it's all ok, but sometimes it seems that some worker threads stay in the wait loop because they don't see that the variable stop is changed and/or they are not waken up by the broadcast.
So the threads never end.
What I am missing?
EDIT: stop=1 moved inside the critical section in exit_handler. The issue remains.
EDIT2: I was executing the program on a VM with Ubuntu. Since the code appears to be totally right I tried to change VM and OS (XUbuntu) and now it seems to work correctly. Still don't know why, anyone has an idea?
Some guessing here, but it's too long for a comment, so if this is wrong, I will delete. I think you may have a misconception about how pthread_cond_broadcast works (at least something I've been burned with in the past). From the man page:
The pthread_cond_broadcast() function shall unblock all threads
currently blocked on the specified condition variable cond.
Ok, that make sense, _broadcast awakens all threads currently blocked on cond. However, only one of the awakened threads will then be able to lock the mutex after they're all awoken. Also from the man page:
The thread(s) that are unblocked shall contend for the mutex according
to the scheduling policy (if applicable), and as if each had called
pthread_mutex_lock().
So this means that if 3 threads are blocked on cond and _broadcast is called, all 3 threads will wake up, but only 1 can grab the mutex. The other 2 will still be stuck in pthread_cond_wait, waiting on a signal. Because of this, they don't see stop set to 1, and exit_handler (I'm assuming a Ctrl+c software signal?) is done signaling, so the remaining threads that lost the _broadcast competition are stuck in limbo, waiting on a signal that will never come, and unable to read that the stop flag has been set.
I think there are 2 options to work-around/fix this:
Use pthread_cond_timedwait. Even without being signaled, this will return from waiting at the specified time interval, see that stop == 1, and then exit.
Add pthread_cond_signal or pthread_cond_broadcast at the end of your worker function. This way, right before a thread exits, it will signal the cond variable allowing any other waiting threads to grab the mutex and finish processing. There is no harm in signaling a conditional variable if no threads are waiting on it, so this should be fine even for the last thread.
EDIT: Here is an MCVE that proves (as far as I can tell) that my answer above is wrong, heh. As soon as I press Ctrl+c, the program exits "immediately", which says to me all the threads are quickly acquiring the mutex after the broadcast, seeing that stop is false, and exiting. Then main joins on the threads and it's process over.
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <stdbool.h>
#include <signal.h>
#include <unistd.h>
#define NUM_THREADS 3
#define STACK_SIZE 10
pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t c = PTHREAD_COND_INITIALIZER;
volatile bool stop = false;
int stack[STACK_SIZE] = { 0 };
int sp = 0; // stack pointer,, also doubles as the current stack size
void SigHandler(int sig)
{
if (sig == SIGINT)
{
stop = true;
}
else
{
printf("Received unexcepted signal %d\n", sig);
}
}
void* worker(void* param)
{
long tid = (long)(param);
while (stop == false)
{
// acquire the lock
pthread_mutex_lock(&m);
while (sp <= 0) // sp should never be < 0
{
// there is no data in the stack to consume, wait to get signaled
// this unlocks the mutex when it is called, and locks the
// mutex before it returns
pthread_cond_wait(&c, &m);
}
// when we get here we should be guaranteed sp >= 1
printf("thread %ld consuming stack[%d] = %d\n", tid, sp-1, stack[sp-1]);
sp--;
pthread_mutex_unlock(&m);
int sleepVal = rand() % 10;
printf("thread %ld sleeping for %d seconds...\n", tid, sleepVal);
sleep(sleepVal);
}
pthread_exit(NULL);
}
int main(void)
{
pthread_t threads[NUM_THREADS];
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
srand(time(NULL));
for (long i=0; i<NUM_THREADS; i++)
{
int rc = pthread_create(&threads[i], &attr, worker, (void*)i);
if (rc != 0)
{
fprintf(stderr, "Failed to create thread %ld\n", i);
}
}
while (stop == false)
{
// produce data in bursts
int numValsToInsert = rand() % (STACK_SIZE - sp);
printf("main producing %d values\n", numValsToInsert);
// acquire the lock
pthread_mutex_lock(&m);
for (int i=0; i<numValsToInsert; i++)
{
// produce values for the stack
int val = rand() % 10000;
// I think this should already be guaranteed..?
if (sp+1 < STACK_SIZE)
{
printf("main pushing stack[%d] = %d\n", sp, val);
stack[sp++] = val;
// signal the workers that data is ready
//printf("main signaling threads...\n");
//pthread_cond_signal(&c);
}
else
{
printf("stack full!\n");
}
}
pthread_mutex_unlock(&m);
// signal the workers that data is ready
printf("main signaling threads...\n");
pthread_cond_broadcast(&c);
int sleepVal = 1;//rand() % 5;
printf("main sleeping for %d seconds...\n", sleepVal);
sleep(sleepVal);
}
for (long i=0; i<NUM_THREADS; i++)
{
pthread_join(threads[i], NULL);
}
return 0;
}

two threads, first adds second substracts

This is a classic example of mutex locks. I don't know why the following code doesn't work, ie. it doesn't print "ctr = 0" every time (but, for example, ctr = 535).
int ctr;
pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
void * add (void * arg_wsk)
{
int i;
for (i = 0; i < 100000; i++) {
pthread_mutex_lock (&m);
ctr++;
pthread_mutex_unlock (&m);
}
return(NULL);
}
void * sub(void * arg_wsk)
{
int i;
for (i = 0; i < 100000; i++) {
pthread_mutex_lock (&m);
ctr--;
pthread_mutex_unlock (&m);
}
return(NULL);
}
int main()
{
pthread_t tid1, tid2;
int i;
void *res;
ctr = 0;
pthread_mutex_init(&m, NULL);
pthread_create(&tid1, NULL, add, NULL);
pthread_detach(tid1);
pthread_create(&tid2, NULL, sub, NULL);
pthread_detach(tid2);
pthread_join(tid1, &res);
pthread_join(tid2, &res);
pthread_mutex_destroy(&m);
printf("ctr = %d", ctr);
pthread_exit(NULL);
}
I think you are misusing the POSIX API. If you detach the threads, you shouldn't join them. Remove the detach and see if this improves things. I think you'll see that main() now blocks until the threads have completed.
Note also, from the link for the join call
ESRCH No thread could be found corresponding to that specified by the
given thread ID.
If you're lucky, you'll hit this. If you're not lucky, crazy things will happen. Don't mix detach and join calls on the same thread.
https://computing.llnl.gov/tutorials/pthreads/#Joining
The value of your counter ctr depends on both the threads completing their full execution.
According to pthread_detach(3)
Once a thread has been detached, it can't be joined with
pthread_join(3) or be made joinable again.
If you remove pthread_detach call from your program, you will get the expected output.
Also, consider explicitly creating your thread joinable(or detached) for portability.

How to make main thread wait for all child threads finish?

I intend to fire 2 threads in the main thread, and the main thread should wait till all the 2 child threads finish, this is how I do it.
void *routine(void *arg)
{
sleep(3);
}
int main()
{
for (int i = 0; i < 2; i++) {
pthread_t tid;
pthread_create(&tid, NULL, routine, NULL);
pthread_join(&tid, NULL); //This function will block main thread, right?
}
}
In the above code, pthread_join indeed makes main thread wait for the child threads, but the problem is, the second thread won't be created untill the first one finishes. This is not what I want.
What I want is, the 2 threads get created immediatly in the main thread, and then main thread waits for them to finish. Seems like pthread_join cannot do the trick, can it?
I thought, maybe via a semaphore I can do the job, but any other way?
int main()
{
pthread_t tid[2];
for (int i = 0; i < 2; i++) {
pthread_create(&tid[i], NULL, routine, NULL);
}
for (int i = 0; i < 2; i++)
pthread_join(tid[i], NULL);
return 0;
}
First create all the threads, then join all of them:
pthread_t tid[2];
/// create all threads
for (int i = 0; i < 2; i++) {
pthread_create(&tid[i], NULL, routine, NULL);
}
/// wait all threads by joining them
for (int i = 0; i < 2; i++) {
pthread_join(tid[i], NULL);
}
Alternatively, have some pthread_attr_t variable, use pthread_attr_init(3) then pthread_attr_setdetachedstate(3)
on it, then pass its address to pthread_create(3) second argument. Thos would create the threads in detached state. Or use pthread_detach as explained in Jxh's answer.
Remember to read some good Pthread tutorial. You may want to use mutexes and condition variables.
You could use frameworks wrapping them, e.g. Qt or POCO (in C++), or read a good C++ book and use C++ threads.
Conceptually, threads have each their call stack and are related to continuations. They are "heavy".
Consider some agent-oriented programming approach: as a rule of thumb, you don't want to have a lot of threads (e.g. 20 threads on a 10 core processor is reasonable, 200 threads won't be unless a lot of them are sleeping or waiting) and and do want threads to synchronize using mutex and condition variables and communicate and/or synchronize with other threads quite often (several times per second). See also poll(2), fifo(7), unix(7), sem_overview(7) with shm_overview(7) as another way of communicating between threads. In general, avoid using signal(7) with threads (read signal-safety(7)...), and use dlopen(3) with caution (probably only in the main thread).
A pragmatical approach would be to have most of your threads running some event loop (using poll(2), pselect(2), perhaps eventfd(2), signalfd(2), ....), perhaps communicating using pipe(7) or unix(7) sockets. See also socket(7).
Don't forget to document (on paper) the communication protocols between threads. For a theoretical approach, read books about π-calculus and be aware of Rice's theorem : debugging concurrent programs is difficult.
You could start the threads detached, and not worry about joining.
for (int i = 0; i < 2; i++) {
pthread_t tid;
pthread_create(&tid, NULL, routine, NULL);
pthread_detach(tid);
}
pthread_exit(0);
Or, alternatively, you can have the thread that dies report back to the main thread who it is, so that the threads are joined in the order they exited, rather than in the order you created them in.
void *routine(void *arg)
{
int *fds = (int *)arg;
pthread_t t = pthread_self();
usleep((rand()/(1.0 + RAND_MAX)) * 1000000);
write(fds[1], &t, sizeof(t));
}
int main()
{
int fds[2];
srand(time(0));
pipe(fds);
for (int i = 0; i < 2; i++) {
pthread_t tid;
pthread_create(&tid, NULL, routine, fds);
printf("created: %llu\n", (unsigned long long)tid);
}
for (int i = 0; i < 2; i++) {
pthread_t tid;
read(fds[0], &tid, sizeof(tid));
printf("joining: %llu\n", (unsigned long long)tid);
pthread_join(tid, 0);
}
pthread_exit(0);
}
#include<stdio.h>
#include<pthread.h>
int icnt = 0; //in non_bss data segment
pthread_mutex_t lock; //lock variable created stored into bss data segment
void *Thread_count(void* args) //syncronization
{
pthread_mutex_lock(&lock); //lock aquire
icnt++;
for(int x = 1; x <= icnt; x++)
{
printf("Hello from Thread_count : %d \n",icnt);
}
printf("\n");
pthread_mutex_unlock(&lock); //release lock
pthread_exit(NULL); //exit from child thread
}
int main()
{
pthread_t threads[4]; //created array of {unsigned long int}
int status = 0;
//creating threads in loop
for(int i = 1; i <= sizeof(threads)/sizeof(threads[0]); i++)
{
pthread_create(&threads[i], NULL, &Thread_count, NULL);
}
//waiting for threads in loop
for(int j = 1; j <= sizeof(threads)/sizeof(threads[0]); j++)
{
pthread_join(threads[j], &status);
printf("Thread number : %d <--> Thread status : %d\n",j, status);
}
pthread_exit(0); //end of main thread
}

Resources