How to properly suspend multiple threads with posix signals? - c

Question

In the context of an existing multi-threaded application I want to suspend a list of threads for a specific duration then resume their normal execution. I know some of you wil say that I should not do that but I know that and I don't have a choice.
I came up with the following code that sort of work but randomly failed. For each thread I want to suspend, I send a signal and wait for an ack via a semaphore. The signal handler when invoked, post the semaphore and sleep for the specified duration.
The problem is when the system is fully loaded, the call to sem_timedwait sometimes fails with ETIMEDOUT and I am left with an inconsistent logic with semaphore used for the ack: I don't know if the signal has been dropped or is just late.
// compiled with: gcc main.c -o test -pthread
#include <pthread.h>
#include <stdio.h>
#include <signal.h>
#include <errno.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <semaphore.h>
#include <sys/types.h>
#include <sys/syscall.h>
#define NUMTHREADS 40
#define SUSPEND_SIG (SIGRTMIN+1)
#define SUSPEND_DURATION 80 // in ms
static sem_t sem;
void checkResults(const char *msg, int rc) {
if (rc == 0) {
//printf("%s success\n", msg);
} else if (rc == ESRCH) {
printf("%s failed with ESRCH\n", msg);
} else if (rc == EINVAL) {
printf("%s failed with EINVAL\n", msg);
} else {
printf("%s failed with unknown error: %d\n", msg, rc);
}
}
static void suspend_handler(int signo) {
sem_post(&sem);
usleep(SUSPEND_DURATION*1000);
}
void installSuspendHandler() {
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sa.sa_handler = suspend_handler;
int rc = sigaction(SUSPEND_SIG, &sa, NULL);
checkResults("sigaction SUSPEND", rc);
}
void *threadfunc(void *param) {
int tid = *((int *) param);
free(param);
printf("Thread %d entered\n", tid);
// this is an example workload, the real app is doing many things
while (1) {
int rc = sleep(30);
if (rc != 0 && errno == EINTR) {
//printf("Thread %d got a signal delivered to it\n", tid);
} else {
//printf("Thread %d did not get expected results! rc=%d, errno=%d\n", tid, rc, errno);
}
}
return NULL;
}
int main(int argc, char **argv) {
pthread_t threads[NUMTHREADS];
int i;
sem_init(&sem, 0, 0);
installSuspendHandler();
for(i=0; i<NUMTHREADS; ++i) {
int *arg = malloc(sizeof(*arg));
if ( arg == NULL ) {
fprintf(stderr, "Couldn't allocate memory for thread arg.\n");
exit(EXIT_FAILURE);
}
*arg = i;
int rc = pthread_create(&threads[i], NULL, threadfunc, arg);
checkResults("pthread_create()", rc);
}
sleep(3);
printf("Will start to send signals...\n");
while (1) {
printf("***********************************************\n");
for(i=0; i<NUMTHREADS; ++i) {
int rc = pthread_kill(threads[i], SUSPEND_SIG);
checkResults("pthread_kill()", rc);
printf("Waiting for Semaphore for thread %d ...\n", i);
// compute timeout abs timestamp for ack
struct timespec ts;
clock_gettime(CLOCK_REALTIME, &ts);
const int TIMEOUT = SUSPEND_DURATION*1000*1000; // in nano-seconds
ts.tv_nsec += TIMEOUT; // timeout to receive ack from signal handler
// normalize timespec
ts.tv_sec += ts.tv_nsec / 1000000000;
ts.tv_nsec %= 1000000000;
rc = sem_timedwait(&sem, &ts); // try decrement semaphore
if (rc == -1 && errno == ETIMEDOUT) {
// timeout
// semaphore is out of sync
printf("Did not received signal handler sem_post before timeout of %d ms for thread %d", TIMEOUT/1000000, i);
abort();
}
checkResults("sem_timedwait", rc);
printf("Received Semaphore for thread %d.\n", i);
}
sleep(1);
}
for(i=0; i<NUMTHREADS; ++i) {
int rc = pthread_join(threads[i], NULL);
checkResults("pthread_join()\n", rc);
}
printf("Main completed\n");
return 0;
}
Questions?
Is it possible for a signal to be dropped and never delivered?
What causes the timeout on the semaphore at random time when the system is loaded?

Answer 1

usleep() is not among the async-signal-safe functions (though sleep() is, and there are other async-signal-safe functions by which you can produce a timed delay). A program that calls usleep() from a signal handler is therefore non-conforming. The specifications do not describe what may happen -- neither with such a call itself nor with the larger program execution in which it occurs. Your questions can be answered only for a conforming program; I do that below.
Is it possible for a signal to be dropped and never delivered?
It depends on what exactly you mean:
If a normal (not real-time) signal is delivered to a thread that already has that signal queued then no additional instance is queued.
A thread can die with signals still queued for it; those signals will not be handled.
A thread can change a given signal's disposition (to SIG_IGN, for example), though this is a per-process attribute, not a per-thread one.
A thread can block a signal indefinitely. A blocked signal is not dropped -- it remains queued for the thread and will eventually be received some time after it is unblocked, if that ever happens.
But no, having successfully queued a signal via the kill() or raise() function, that signal will not be randomly dropped.
What causes the timeout on the semaphore at random time when the system is loaded?
A thread can receive a signal only when it is actually running on a core. On a system with more runnable processes than cores, some runnable processes must be suspended, without a timeslice on any core, at any given time. On a heavily-loaded system, that's the norm. Signals are asynchronous, so you can send one to a thread that is currently waiting for a timeslice without the sender blocking. It is entirely possible, then, that the thread you have signaled does not get scheduled to run before the timeout expires. If it does run, it may have the signal blocked for one reason or another, and not get around to unblocking it before it uses up its timeslice.
Ultimately, you can use your semaphore-based approach to check whether the target thread handled the signal within any timeout of your choice, but you cannot predict in advance how long it will take for the thread to handle the signal, nor even whether it will do so in any finite amount of time (for example, it could die for one reason or another before doing so).