Pthreads signals getting lost when used as IPC inLinux - c

Question

We are developing a simple application in Linux for Desktop computer. The scheme is very simple and as below:
A main process which deals with the external world interface (which gives some data periodically) and spawns and keeps track of child processes.
Various child processes which process this data and report back to main process from time to time.
The data coming in from external world interface is in the chunks of around 240 KBytes and 1 chunk comes in at the rate of roughly once per milliseconds. All the child processes use and work on the same data i.e. the complete chunk is sent to every child process when it arrives.
The number of child processes is not fixed, can vary from 4 to 20. The scheme adopted for inter process communication is as follows:
Shared memory capable of holding multiple data chunks is created by all the processes using common key and using shmget() and shmat() APIs. Main process writes to this shared memory and all the child processes read from this memory.
To inform the child processes that new data chunk have arrived I have used pthread_cond_broadcast() API. The conditional variable and the corresponding mutex used for this purpose reside in a small separate shared memory and are initialized to default attributes in the main process.
So whenever new data chunk arrives (roughly once per 1 ms) main process calls pthread_cond_broadcast() and the child processes which are waiting on pthread_cond_wait() read this data from the shared memory and process it. The problem I am facing is:
Depending on the processor load, sometimes the pthread signals are getting lost i.e. either delivered to only some or none of the waiting child processes. This severly affects the data processing as the data continuity is lost (and the child process is not even aware of it). Processing time of the child process is average 300 micro seconds and this application is running on a multicore processor.
To pin down the problem I even created a dummy application with 1 main process and several dummy child processes who does nothing but to wait on pthread_cond_wait(). From main process I called pthread_cond_broadcast every 1 ms and a count was incremented and printed, similarly every time a pthread signal was received in a child process another count was incremented and printed. When I ran this test program I found after some time the receiver's count began to lag the sender's count and the gap between their count went on increasing. Am I right in my understanding that this was due to some pthread signals not delivered? Are there any other fast plus secure IPC mechanisms.
I even tried the same thing using internet domain sockets using UDP datagrams in broadcast (only for the synchronization purpose while the data was still read from the shared memory). But here also I noticed as the number of child processes increased the synchronization signals were getting lost. Please give your thoughts and ideas.
Consider the test program as below:
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <pthread.h>
#define SHM_KEY 3579
#define NumOfChildProc 20
int Packets_Tx = 0, Packets_Rx = 0;
void ChildProc(void)
{
/* Create the shared memory with same key as SHM_KEY
* Declare the condition and mutex and assign them the shared memory
address */
while(1)
{
pthread_mutex_lock(PTMutex);
pthread_cond_wait(PTCond, PTMutex);
pthread_mutex_unlock(PTMutex);
printf("From CP [%d]: Packets Received = %d\n",getpid(), Packets_Rx++);
}
}
int main(void)
{
int shmid, i;
pid_t l_pid;
char* SigBlock;
pthread_condattr_t condattr;
pthread_mutexattr_t mutexattr;
pthread_cond_t* PTCond;
pthread_mutex_t* PTMutex;
shmid = shmget(SHM_KEY, (sizeof(pthread_cond_t) + sizeof(pthread_mutex_t)), IPC_CREAT | 0666);
if(shmid < 0)
{
perror("shmget");
}
SigBlock = (char *)shmat(shmid, NULL, 0);
if(SigBlock == (char *) -1)
{
perror("shmat");
}
PTCond = (pthread_cond_t*) SigBlock;
PTMutex = (pthread_mutex_t*)(SigBlock + sizeof(pthread_cond_t));
pthread_condattr_init(&condattr);
pthread_condattr_setpshared(&condattr, PTHREAD_PROCESS_SHARED);
pthread_cond_init(PTCond, &condattr);
pthread_condattr_destroy(&condattr);
pthread_mutexattr_init(&mutexattr);
pthread_mutexattr_setpshared(&mutexattr, PTHREAD_PROCESS_SHARED);
pthread_mutex_init(PTMutex, &mutexattr);
pthread_mutexattr_destroy(&mutexattr);
for(i=0; i<NumOfChildProc; i++)
{
l_pid = fork();
if(l_pid == 0)
ChildProc();
}
sleep(1);
while(1)
{
/* Send pthread broadcast and increment the packets count */
printf("From Main Process : Packets Sent = %d\n", Packets_Tx++);
pthread_cond_broadcast(PTCond);
usleep(1000);
}
}

Answer 1

pthread_cond_broadcast() signals do not get "lost". Every thread which is waiting in a pthread_cond_wait() call at the point where the broadcast is sent will be woken - your problem is almost certainly that every thread is not waiting in the pthread_cond_wait() call at the point where pthead_cond_broadcast() is called - some threads may still be processing that last lot of data when the broadcast is sent, in which case they will "miss" the broadcast.
A pthread conditional variable should always be paired with a suitable condition (or predicate) over shared state, and a thread should only call pthread_cond_wait() after checking the state of that predicate.
For example, in your case you might have a shared variable which is the block number of the latest chunk to have arrived. In the main thread, it would increment this (while holding the mutex) before broadcasting the condition variable:
pthread_mutex_lock(&lock);
latest_block++;
pthread_cond_broadcast(&cond);
pthread_mutex_unlock(&lock);
In the worker threads, each thread would keep track of the last block it has processed in a local variable, and check to see if another block has arrived before calling pthread_cond_wait():
pthread_mutex_lock(&lock);
while (latest_block <= my_last_block)
pthread_cond_wait(&cond, &lock);
pthread_mutex_unlock(&lock);
This will cause the worker to wait until the main thread has incremented latest_block to be greater than my_last_block (the last block that was processed by this worker).
Your example test code has the same problem - sooner or later the main thread will call pthread_cond_broadcast() when a child thread is locking or unlocking the mutex, or inside the printf() call.
A version of your example code, updated to use the fix I mentioned, does not show this problem:
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <pthread.h>
#define SHM_KEY 9753
#define NumOfChildProc 20
int Packets_Tx = 0, Packets_Rx = 0;
struct {
pthread_cond_t PTCond;
pthread_mutex_t PTMutex;
int last_packet;
} *shared_data;
void ChildProc(void)
{
int my_last_packet = 0;
/* Create the shared memory with same key as SHM_KEY
* Declare the condition and mutex and assign them the shared memory
address */
while(1)
{
pthread_mutex_lock(&shared_data->PTMutex);
while (shared_data->last_packet <= my_last_packet)
pthread_cond_wait(&shared_data->PTCond, &shared_data->PTMutex);
pthread_mutex_unlock(&shared_data->PTMutex);
printf("From CP [%d]: Packets Received = %d\n",getpid(), Packets_Rx++);
my_last_packet++;
}
}
int main(void)
{
int shmid, i;
pid_t l_pid;
pthread_condattr_t condattr;
pthread_mutexattr_t mutexattr;
shmid = shmget(SHM_KEY, sizeof *shared_data, IPC_CREAT | 0666);
if(shmid < 0)
{
perror("shmget");
}
shared_data = shmat(shmid, NULL, 0);
if(shared_data == (void *) -1)
{
perror("shmat");
}
pthread_condattr_init(&condattr);
pthread_condattr_setpshared(&condattr, PTHREAD_PROCESS_SHARED);
pthread_cond_init(&shared_data->PTCond, &condattr);
pthread_condattr_destroy(&condattr);
pthread_mutexattr_init(&mutexattr);
pthread_mutexattr_setpshared(&mutexattr, PTHREAD_PROCESS_SHARED);
pthread_mutex_init(&shared_data->PTMutex, &mutexattr);
pthread_mutexattr_destroy(&mutexattr);
shared_data->last_packet = 0;
for(i=0; i<NumOfChildProc; i++)
{
l_pid = fork();
if(l_pid == 0)
ChildProc();
}
sleep(1);
while(1)
{
/* Send pthread broadcast and increment the packets count */
printf("From Main Process : Packets Sent = %d\n", Packets_Tx++);
pthread_mutex_lock(&shared_data->PTMutex);
shared_data->last_packet++;
pthread_cond_broadcast(&shared_data->PTCond);
pthread_mutex_unlock(&shared_data->PTMutex);
usleep(30);
}
}