gcc 4.7.2
c89
APR 1.4
Hello,
I am using APR thread pools, event loop (polling), and message queue (msgget, msgsnd, msgrcv).
Currently, I am just testing that if I press ctrl-c. It will set the while to false. So the message will be sent in my signal handler to be received in my while loop.
However, the msgrcv seems to block in my event loop. So I can't get any other events from my library. In my event loop I am checking if a message has been recevied. However, I think it blocks waiting for a message. This will cause my loop to stop looping and I won't be able to capture any event from my library.
This is the first time I have used message queues.
Here is my code, I am only copied the relevent parts so I have removed all error checking to keep it short.
Currently I am just testing that my event loop can receive a message from the queue, so it can stop the loop.
Many thanks for any suggestions,
#define MSG_KEY 0001L
static int msg_id = 0;
static volatile apr_status_t is_looping = FALSE;
/* Wait for these message commands */
typedef enum tag_msg_cmd {END_LOOP, START_LOOP} msg_cmd_e;
/* message queue data */
typedef struct tag_msg_data msg_data_t;
struct tag_msg_data {
long mtype;
msg_cmd_e msg_cmd;
};
int main(void)
{
/* Create message queue */
msg_id = msgget(MSG_KEY, 0666 | IPC_CREAT);
/* Create thread pool */
has_error = apr_thread_pool_create(&thd_pool,
init_threads,
max_threads,
mem_pool_app);
/* Start event loop */
LOG_CHECK(apr_thread_pool_schedule(thd_pool,
(apr_thread_start_t)event_loop,
NULL,
0,
NULL) == APR_SUCCESS, "Failed to schedule event loop");
/* Pause until ctrl-c */
pause();
/* Destroy all memory pools and threading pools */
}
/* Poll for incoming events */
static apr_status_t event_loop(void)
{
msg_data_t msg_data;
int evt_id = 0;
int msg_id = 0;
/* Connect to the message queue */
msg_id = msgget(MSG_KEY, 0644);
assert(msg_id != -1 && "Failed to connect to the message queue");
LOG_DEBUG("Connected to message queue with msg ID [ %d ]", msg_id);
while(is_looping) {
/* Under development - Process incoming event from shared library */
/* evt_id = sr_waitevt(timeout_ms); */
/* process_event(evt_id); */
LOG_DEBUG("Looping....");
/* Should not be blocking here, as I need to receive other events */
msgrcv(msg_id, &msg_data, (sizeof msg_data) - (sizeof(long)), 2, 0);
LOG_DEBUG("Waiting...");
if(msg_data.msg_cmd == END_LOOP) {
LOG_DEBUG("END_LOOP event queue message has been received");
break;
}
sleep(1);
}
return 1;
}
/* Signal handler will terminated application on ctrl-c */
static void signal_handler(int sig)
{
msg_data_t msg_data = {2, END_LOOP};
int rc = 0;
rc = msgsnd(msg_id,
&msg_data,
(sizeof msg_data) - (sizeof(long)),
IPC_NOWAIT);
assert(rc == 0 && "Failed to send data to the message queue");
}
Set the flag parameter on msgrcv (the 5th parm) to IPC_NOWAIT. This makes the queue receive non-blocking.
Related
while(1) {
char message_buffer[SIZE];
ssize_t message_length = mq_receive(mq_identifier, message_buffer, _mqueue_max_msg_size NULL);
if(message_len == -1) { /* error handling... */}
pthread_t pt1;
int ret = pthread_create(&pt1, NULL, handle_message, message_buffer);
if(ret) { /* error handling ... */}
}
void * handle_message (void * message) {
puts((char *) message);
return NULL;
}
The above example is not an MRE but it is extremely simple:
I've got a main thread with a loop that constantly consumes messages from a message queue. Once a new message is received, it is stored in the local message_buffer buffer. Then, a new thread is spawned to "take care" of said new message, and thus the message buffer's address is passed into handle_message, which the new thread subsequently executes.
The problem
Often, 2 threads will print the same message, even though I can verify with a 100% certainty that the messages in the queue were not the same.
I am not completely certain, but I think I understand why this is happening:
say that I push 2 different messages to the mqueue and only then I begin consuming them.
In the first iteration of the while loop, the message will get consumed from the queue and saved to message_buffer. A new thread will get spawned and the address of message_length passed to it. But that thread may not be fast enough to print the buffer's contents to the stream before the next message gets consumed (on the next iteration of the loop), and the contents of message_buffer subsequently overridden. Thus the first and second thread now print the same value.
My question is: what is the most efficient way to solve this? I'm pretty new to parallel programming and threading/pthreads and I'm pretty overwhelmed by the different synchronization primitives.
Mutex trouble
static pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
while(1) {
char message_buffer[SIZE];
pthread_mutex_lock(&m);
ssize_t message_length = mq_receive(mq_identifier, message_buffer, _mqueue_max_msg_size NULL);
pthred_mutex_unlock(&m);
if(message_len == -1) { /* error handling... */}
pthread_t pt1;
int ret = pthread_create(&pt1, NULL, handle_message, message_buffer);
if(ret) { /* error handling ... */}
}
void * handle_message (void * message) {
char own_buffer[SIZE];
pthread_mutex_lock(&m);
strncpy(own_buffer, (char *) message, SIZE);
pthread_mutex_unlock(&m);
puts(own_buffer);
return NULL;
}
I don't think my current mutex implementation is right as the threads are still receiving duplicate messages. The main thread can lock the mutex, consume a message into the buffer, unlock the mutex, spawn a thread, but that thread still may hang and the main one could just rewrite the buffer again (as the buffer mutex was never locked by the new thread), effectively making my current mutex implementation useless? How do I overcome this?
The problem is that you end the loop that contains message_buffer before guaranteeing that the thread has finished with that memory.
while (1) {
char message_buffer[SIZE];
ssize_t message_length = mq_receive(...);
if (message_len == -1) { /* error handling */ }
pthread_t pt1;
int ret = pthread_create(&pt1, NULL, handle_message, message_buffer);
if (ret) { /* error handling */ }
/****** Can't go beyond here until thread is done with message_buffer. ******/
}
void * handle_message (void * message) {
char own_buffer[SIZE];
strncpy(own_buffer, (char *) message, SIZE);
/******* Only now can the caller loop back. ******/
puts(own_buffer);
return NULL;
}
You could use a semaphore or similar.
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
static int copied = 0;
while (1) {
char message_buffer[SIZE];
ssize_t message_length = mq_receive(...);
if (message_len == -1) { /* error handling */ }
pthread_t pt1;
int ret = pthread_create(&pt1, NULL, handle_message, message_buffer);
if (ret) { /* error handling */ }
// Wait until threads is done with message_buffer.
pthread_mutex_lock(&mutex);
while (!copied) pthread_cond_wait(&cond, &mutex);
copied = 0;
pthread_mutex_unlock(&mutex);
}
void * handle_message (void * message) {
char own_buffer[SIZE];
strncpy(own_buffer, (char *) message, SIZE);
// Done with caller's buffer.
// Signal caller to continue.
pthread_mutex_lock(&mutex);
copied = 1;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&mutex);
puts(own_buffer);
return NULL;
}
(The added chunks effectively perform semaphore operations. See the last snippet of this answer for a more generic implementation.)
But there's a simpler solution: Make the copy before creating the thread.
while (1) {
char message_buffer[SIZE];
ssize_t message_length = mq_receive(...);
if (message_len == -1) { /* error handling */ }
pthread_t pt1;
int ret = pthread_create(&pt1, NULL, handle_message, strdup(message_buffer));
if (ret) { /* error handling */ }
}
void * handle_message (void * message) {
char * own_buffer = message;
puts(own_buffer);
free(own_buffer);
return NULL;
}
I have a program in c which is supposed to send and receive ipc messages through msgq.
The problem I have is that when I run msgrcv() it sets my global int msqid to 0. And of course I need it at other methods, like in a signal handler.
here is some code:
/* all the includes and some variables*/
#include "msg.h" // include the one I made
int msgQ; // global int
int main(int argc, char *argv[])
{
key = ftok("progfile", 65);
msgQ = msgget(key, 0666 | IPC_CREAT);
printf("msg queue id: %d \n", msgQ);
start_tik_tok(); // setting up the timer and the signal handler
/* irrelevant code */
void read_msgs(msgQ);
}
void read_msgs(int msgQid)
{
while (1)
{
printf("before the read local:%d goval:%d\n", msgQid, msgQ);
int ret = msgrcv(msgQid, &message, sizeof(message), 1, 0);
printf("after the read local:%d global :%d\n", msgQid, msgQ);
if (ret == -1)
/* error handling */
switch (message.action_type)
{
/* mesage handling */
}
}
void signal_handler(int signo)
{
/*I need the global int here to send some messages */
}
void start_tik_tok()
{
//timer interval for setitimer function
struct itimerval timer;
timer.it_interval.tv_sec = 1; //every 1 seconds
timer.it_interval.tv_usec = 0;
timer.it_value.tv_sec = 1; //start in 1 seconds
timer.it_value.tv_usec = 0;
//action for the signal
struct sigaction new_sa;
memset(&new_sa, 0, sizeof(new_sa));
new_sa.sa_handler = &signal_handler;
sigaction(SIGALRM, &new_sa, NULL);
setitimer(ITIMER_REAL, &timer, NULL);
}
the msg.h file:
#include <sys/msg.h>
struct msg_buff{
long mesg_type; //reciver
int sender; //sender
char action_type;
char time_tiks; //time in tiks
} message;
output:
msg queue id: 45416448
before the read local:45416448 global:45416448
after the read local:45416448 global:0
...
you can see that after I run msgrcv(), the value of msgQ turns to 0, even though I'm using a variable to pass the value to the method read_msgs().
The msgrcv function takes a pointer to a structure that starts with a "header" of type long, followed by the message data. The third argument to msgrcv, msgsz, is the size of the message data body, not including the long that's the header. So you should pass something like sizeof message - sizeof(long). By passing sizeof message, you're asking it to overflow the buffer sizeof(long) bytes, and this is clobbering some other global variable.
I found the solution, I'm not sure why is that but it solved it.
I just initialized the int from the beginning.
changed:
int msgQ; // global int
for:
int msgQ = 0; // global int
There's something going on in my code that I didn't intend.
I've got a program with several threads. One thread reads a rfid chip and saves the rfid code string into a global struct variable. There are also some other threads that are opened when a client, which is supposed to get the string, connects to the program on a certain socket. They are on a timed waiting (pthread_cond_timedwait) for the reading thread to broadcast a pthread condition variable (thread->done) that signals that a rfid chip was read and there is a new string to be sent to the clients. Now to the problem: the pthread_cond_timedwait in the sending thread shall only be satisfied when a new string is ready to be sent, but it also triggers when a new client thread is opened. The client threads start with the function below. I hope I explained my problem clearly and anyone can help me with the problem.
code of the sending function:
void* send_data(void* arg)
{
int client_id;
int status_timeout;
struct timespec timeout;
struct Thread_parameter *thread = (struct Thread_parameter*)arg;
pthread_condattr_t attr;
pthread_condattr_init(&attr);
pthread_condattr_setclock(&attr, CLOCK_MONOTONIC);
pthread_cond_init(&thread->done, &attr);
// read the client id from struct
pthread_mutex_lock(&thread->mutex);
client_id = thread->t_client_id;
pthread_mutex_unlock(&thread->mutex);
while (1)
{
// wait for string
while (1)
{
pthread_mutex_lock(&thread->mutex);
status_timeout = ETIMEDOUT;
while (1)
{
clock_gettime(CLOCK_MONOTONIC, &timeout);
timeout.tv_sec += 1;
status_timeout = pthread_cond_timedwait(&thread->done, &thread->mutex, &timeout);
// Timeout
if (status_timeout == ETIMEDOUT)
{
if (is_connected(client_id))
continue;
else
{
log_output("Lost connection to client '%i'", client_id);
pthread_mutex_unlock(&thread->mutex);
pthread_exit(NULL);
}
}
else if (status_timeout == 0)
{
// condition satisfied
break;
}
}
pthread_mutex_unlock(&thread->mutex);
break;
}
// send the string to the client when the condition is satisfied
while (1)
{
pthread_mutex_lock(&thread->mutex);
if (send(client_id, thread->t_chip_id, CHIP_ID_LENGTH, MSG_NOSIGNAL) >= 0)
{
pthread_mutex_unlock(&thread->mutex);
log_output("Successfully sent data to client '%i'", client_id);
break;
}
else
{
pthread_mutex_unlock(&thread->mutex);
// sending not successful, check if client is still connected
if (is_connected(client_id))
continue;
else
{
close(client_id);
log_output("Lost connection to client '%i'", client_id);
pthread_exit(NULL);
}
}
}
}
}
I would like to wake up a pthread from another pthread - but after some time. I know signal or pthread_signal with pthread_cond_wait can be used to wake another thread, but I can't see a way to schedule this. The situation would be something like:
THREAD 1:
========
while(1)
recv(low priority msg);
dump msg to buffer
THREAD 2:
========
while(1)
recv(high priority msg);
..do a little bit of processing with msg ..
dump msg to buffer
wake(THREAD3, 5-seconds-later); <-- **HOW TO DO THIS? **
//let some msgs collect for at least a 5 sec window.
//i.e.,Don't wake thread3 immediately for every msg rcvd.
THREAD 3:
=========
while(1)
do some stuff ..
Process all msgs in buffer
sleep(60 seconds).
Any simple way to schedule a wakeup (short of creating a 4th thread that wakes up every second and decides if there is a scheduled entry for thread-3 to wakeup). I really don't want to wakeup thread-3 frequently if there are only low priority msgs in queue. Also, since the messages come in bursts (say 1000 high priority messages in a single burst), I don't want to wake up thread-3 for every single message. It really slows things down (as there is a bunch of other processing stuff it does every time it wakes up).
I am using an ubuntu pc.
How about the use of the pthread_cond_t object available through the pthread API ?
You could share such an object within your threads and let them act on it appropriately.
The resulting code should look like this :
/*
* I lazily chose to make it global.
* You could dynamically allocate the memory for it
* And share the pointer between your threads in
* A data structure through the argument pointer
*/
pthread_cond_t cond_var;
pthread_mutex_t cond_mutex;
int wake_up = 0;
/* To call before creating your threads: */
int err;
if (0 != (err = pthread_cond_init(&cond_var, NULL))) {
/* An error occurred, handle it nicely */
}
if (0 != (err = pthread_mutex_init(&cond_mutex, NULL))) {
/* Error ! */
}
/*****************************************/
/* Within your threads */
void *thread_one(void *arg)
{
int err = 0;
/* Remember you can embed the cond_var
* and the cond_mutex in
* Whatever you get from arg pointer */
/* Some work */
/* Argh ! I want to wake up thread 3 */
pthread_mutex_lock(&cond_mutex);
wake_up = 1; // Tell thread 3 a wake_up rq has been done
pthread_mutex_unlock(&cond_mutex);
if (0 != (err = pthread_cond_broadcast(&cond_var))) {
/* Oops ... Error :S */
} else {
/* Thread 3 should be alright now ! */
}
/* Some work */
pthread_exit(NULL);
return NULL;
}
void *thread_three(void *arg)
{
int err;
/* Some work */
/* Oh, I need to sleep for a while ...
* I'll wait for thread_one to wake me up. */
pthread_mutex_lock(&cond_mutex);
while (!wake_up) {
err = pthread_cond_wait(&cond_var, &cond_mutex);
pthread_mutex_unlock(&cond_mutex);
if (!err || ETIMEDOUT == err) {
/* Woken up or time out */
} else {
/* Oops : error */
/* We might have to break the loop */
}
/* We lock the mutex again before the test */
pthread_mutex_lock(&cond_mutex);
}
/* Since we have acknowledged the wake_up rq
* We set "wake_up" to 0. */
wake_up = 0;
pthread_mutex_unlock(&cond_mutex);
/* Some work */
pthread_exit(NULL);
return NULL;
}
If you want your thread 3 to exit the blocking call to pthread_cond_wait() after a timeout, consider using pthread_cond_timedwait() instead (read the man carefully, the timeout value you supply is the ABSOLUTE time, not the amount of time you don't want to exceed).
If the timeout expires, pthread_cond_timedwait() will return an ETIMEDOUT error.
EDIT : I skipped error checking in the lock / unlock calls, don't forget to handle this potential issue !
EDIT² : I reviewed the code a little bit
You can have the woken thread do the wait itself. In the waking thread:
pthread_mutex_lock(&lock);
if (!wakeup_scheduled) {
wakeup_scheduled = 1;
wakeup_time = time() + 5;
pthread_cond_signal(&cond);
}
pthread_mutex_unlock(&lock);
In the waiting thread:
pthread_mutex_lock(&lock);
while (!wakeup_scheduled)
pthread_cond_wait(&cond, &lock);
pthread_mutex_unlock(&lock);
sleep_until(wakeup_time);
pthread_mutex_lock(&lock);
wakeup_scheduled = 0;
pthread_mutex_unlock(&lock);
Why not just compare the current time to one save earlier?
time_t last_uncond_wakeup = time(NULL);
time_t last_recv = 0;
while (1)
{
if (recv())
{
// Do things
last_recv = time(NULL);
}
// Possible other things
time_t now = time(NULL);
if ((last_recv != 0 && now - last_recv > 5) ||
(now - last_uncond_wakeup > 60))
{
wake(thread3);
last_uncond_wakeup = now;
last_recv = 0;
}
}
Objective: N nodes (running on different machines) should communicate with each other by establishing TCP connections with each other. Sending and receiving messages are done by 2 threads created by the process. Initially the main process connects all nodes with each other, creates the 2 threads and gives it a list of file descriptors which can be used by threads to send and receive data. The below structure is filled by the main process and passed to the threads.
typedef struct
{
char hostName[MAXIMUM_CHARACTERS_IN_HOSTNAME]; /* Host name of the node */
char portNumber[MAXIMUM_PORT_LENGTH]; /* Port number of the node */
char nodeId[MAXIMUM_NODE_ID_LENGTH]; /* Node ID of the node */
int socketFd; /* Socket file descriptor */
int socketReady; /* Flag to indicate if socket information is filled */
}SNodeInformation;
PS: socketFd is the socket descriptor received by either accept() or by socket() depending on how the connection was established (Either listening to connections from a node or connecting to a node).
An array of SNodeInformation of size MAX_NUM_OF_NODES is used.
The send thread goes through the nodeInformation and sends a message "Hello" to all nodes as except itself show below.
void *sendMessageThread(void *pNodeInformation) {
int i;
int ownNodeId;
int bytesSent = 0;
char ownHostName[MAXIMUM_CHARACTERS_IN_HOSTNAME];
SNodeInformation *nodeInformation = (SNodeInformation *) pNodeInformation;
SNodeInformation *iterNodeInformation;
printf("SendMessageThread: Send thread created\n");
if(gethostname(ownHostName, MAXIMUM_CHARACTERS_IN_HOSTNAME) != 0) {
perror("Error: sendMessageThread, gethostname failed\n");
exit(1);
}
for(i=0, iterNodeInformation=nodeInformation ; i<MAXIMUM_NUMBER_OF_NODES ; i++, iterNodeInformation++) {
if(strcmp((const char*) iterNodeInformation->hostName, (const char*) ownHostName) != 0) {
/* Send message to all nodes except yourself */
bytesSent = send(iterNodeInformation->socketFd, "Hello", 6, 0);
if(bytesSent == -1) {
printf("Error: sendMessageThread, sending failed, code: %s FD %d\n", strerror(errno), iterNodeInformation->socketFd);
}
}
}
pthread_exit(NULL);
}
The receive thread goes through the nodeInformation, sets up a file descriptor set and uses select to wait for incoming data as show below.
void *receiveMessageThread(void *pNodeInformation)
{
int i;
int fileDescriptorMax = -1;
int doneReceiving = 0;
int numberOfBytesReceived = 0;
int receiveCount = 0;
fd_set readFileDescriptorList;
char inMessage[6];
SNodeInformation *nodeInformation = (SNodeInformation *) pNodeInformation;
SNodeInformation *iterNodeInformation;
printf("ReceiveMessageThread: Receive thread created\n");
/* Initialize the read file descriptor */
FD_ZERO(&readFileDescriptorList);
for(i=0, iterNodeInformation=nodeInformation ; i<MAXIMUM_NUMBER_OF_NODES ; i++, iterNodeInformation++) {
FD_SET(iterNodeInformation->socketFd, &readFileDescriptorList);
if(iterNodeInformation->socketFd > fileDescriptorMax) {
fileDescriptorMax = iterNodeInformation->socketFd;
}
}
printf("ReceiveMessageThread: fileDescriptorMax:%d\n", fileDescriptorMax);
while(!doneReceiving) {
if (select(fileDescriptorMax+1, &readFileDescriptorList, NULL, NULL, NULL) == -1) {
perror("Error receiveMessageThread, select failed \n");
return -1;
}
for(i=0 ; i<fileDescriptorMax ; i++) {
if (FD_ISSET(i, &readFileDescriptorList)) {
/* Check if any FD was set */
printf("ReceiveThread: FD set %d\n", i);
/* Receive data from one of the nodes */
if ((numberOfBytesReceived = recv(i, &inMessage, 6, 0)) <= 0) {
/* Got error or connection closed by client */
if (numberOfBytesReceived == 0) {
/* Connection closed */
printf("Info: receiveMessageThread, node %d hung up\n", i);
}
else {
perror("Error: receiveMessageThread, recv FAILED\n");
}
close(i);
/* Remove from Master file descriptor set */
FD_CLR(i, &readFileDescriptorList);
doneReceiving = 1;
}
else {
/* Valid data from a node */
inMessage[6] = '\0';
if(++receiveCount == MAXIMUM_NUMBER_OF_NODES-1) {
doneReceiving = 1;
}
printf("ReceiveThread: %s received, count: %d\n", inMessage, rece iveCount);
}
}
}
}
pthread_exit(NULL);
}
Expected Output: I tried with just 2 processes, P1 (Started first) and P2 running on machine1 and another on machine2. Both the processes in the machines should first connect and then the threads should send and receive the message "Hello" and exit.
Observed Output: The P1 is able to send the message and P2 (receiver thread) is able to receive the message "Hello". But P1 (receiver thread) is not able to get the message from P2 (Sending thread). Application code is the same in both the machines but every time, the process started first does not get the message from the other process. I added a print to just check if some file descriptor was set, but I don't see it for P1 but only for the P2. The send in the receiving process is not failing, it returns with 6. I checked the maximum value of file descriptors, its correct.
If I start P2 first and then P1 then I can see that P1 receives the message from P2 and exists while P2 waits infinitely for the message from P1.
I am not sure if the problem is because of incorrect use of socket descriptors or because of threads ?
Two issues:
1 The loop testing for a file descriptor being set, does not include all file descriptors put into the set. (This programming error is expected to be the reason for the malfunction described in the OP.)
2 The sets of file descriptors passed to select() are modified by select(), so the set need to be re-initialized before for select() again. (The programming error would only be notable if from more than one socket data sall be received.)
Please see the following mod/s to the OP's code:
void *receiveMessageThread(void *pNodeInformation)
{
...
printf("ReceiveMessageThread: Receive thread created\n");
while(!doneReceiving) {
/* Initialize the read-set of file descriptors */
/* Issue 2 fixed from here ... */
FD_ZERO(&readFileDescriptorList);
for(i=0, iterNodeInformation=nodeInformation ; i<MAXIMUM_NUMBER_OF_NODES ; i++, iterNodeInformation++) {
FD_SET(iterNodeInformation->socketFd, &readFileDescriptorList);
if (iterNodeInformation->socketFd > fileDescriptorMax) {
fileDescriptorMax = iterNodeInformation->socketFd;
}
}
/* ... up to here. */
printf("ReceiveMessageThread: fileDescriptorMax:%d\n", fileDescriptorMax);
if (select(fileDescriptorMax+1, &readFileDescriptorList, NULL, NULL, NULL) == -1) {
perror("Error receiveMessageThread, select failed \n");
return -1;
}
for(i=0 ; i <= fileDescriptorMax ; i++) { /* Issue 1 fixed here. */
...