I am quit confused regarding this problem, I have 10 threads which try to insert into a queue.
I want to prevent more than one inserting at the same time using mutex defined globally in queue.h.
I have 2 queues waiting and working and I just gave it a second though that it's not efficient to give same DS same mutex since this will prevent working on them in parallel even though no problem with that (please correct me if I am wrong).
This means I need mutex for each copy of queue, I tried defining it inside this:
struct queue
{
pthread_mutex_t m; //new
pthread_cond_t c; //new
int queue_size;
struct queue_node *head, *tail;
};
but in this case it won't be global so threads can't relate to the same mutex, how can I fix that easily?
If you need this:
waiting_buffer = create_queue();
working_buffer = create_queue();
struct queue *create_queue() {
pthread_mutex_init(&m, NULL); /** initialize mutex **/
pthread_cond_init(&c, NULL);/** initialize cond **/
struct queue *q = (struct queue *) malloc(sizeof(struct queue));
q->head = q->tail = NULL;
q->queue_size = 0;
return q;
}
Sounds like you want a per-queue mutex and cond var, so you want to include them in the structure returned by create_queue. You did this; you just didn't reference them in the create_queue.
// Sets `errno` and returns `NULL` on error.
struct queue *create_queue(void) {
struct queue *q = (struct queue *)malloc(sizeof(struct queue));
if (!q)
return NULL;
pthread_mutex_init(&(q->m), NULL);
pthread_cond_init(&(q->c), NULL);
q->head = q->tail = NULL;
q->queue_size = 0;
return q;
}
Related
How do you delegate tasks for N threads such that the workload is evenly distributed?
Say we have a queue
[the] -> [quick] -> [brown] -> [fox] -> [jumps] -> [over] -> [the] -> [lazy] -> [dog]
And we have N threads to split up the workload of dequeuing the queue and output the words where at least one word is only printed by one thread.
Here's my attempt (Updated, fixed null printing):
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#define threadCount 8 // Set number of threads
pthread_t* thread;
pthread_mutex_t lock;
//========= Setup for Queue =========
struct node
{
char *key;
struct node *next;
};
struct Q
{
struct node *front, *rear;
};
struct node* newNode(char *key)
{
struct node* temp = (struct node*)malloc(sizeof(struct node));
temp->key = key;
temp->next = NULL;
return temp;
}
struct Q* q;
void enqueue(char* key)
{
struct node* temp = newNode(key);
if(q->rear == NULL)
{
q->front = q->rear = temp;
return;
}
q->rear->next = temp;
q->rear = temp;
}
char* dequeue()
{
if (q->front == NULL)
{
return NULL;
}
struct node* temp = q->front;
char *key = temp->key;
q->front = q->front->next;
if(q->front == NULL)
{
q->rear = NULL;
}
free(temp);
return key;
}
//========= Setup for Queue =========
void *run(void* arg)
{
int id = *(int*)arg;
char* node;
while(q->front != NULL)
{
pthread_mutex_lock(&lock);
node = dequeue();
pthread_mutex_unlock(&lock);
if(node == NULL)
{
return NULL;
}
printf("Thread %d: %s\n", id, node);
}
return 0;
}
int main()
{
q = (struct Q*)malloc(sizeof(struct Q));
q->front = NULL;
q->rear = NULL;
enqueue("the");
enqueue("quick");
enqueue("brown");
enqueue("fox");
enqueue("jumps");
enqueue("over");
enqueue("the");
enqueue("lazy");
enqueue("dog");
thread = malloc(sizeof(pthread_t)*threadCount);
// Should output lines be only N-1 due to how the id is generated?
for(int id = 0; id < threadCount; id++)
{
pthread_create(&thread[id], NULL, (void *) run, &id);
}
for(int id = 0; id < threadCount; id++)
{
pthread_join(thread[id], NULL);
}
free(thread);
free(q);
return 0;
}
Here is my unresolved problem:
Sometimes there are lines also that print Thread N, but according to how I implemented (see main), max output for thread number should be N-1.
Lock the mutex
Take an item from the queue
Unlock the mutex
Do the work with the item
You don't want to hold the mutex while doing the work (e.g. printf) because then only one thread can do work at a time - you aren't really using multiple threads.
The reason you get (null) printed is that your code checks q->front != NULL while it hasn't locked the mutex. Then by the time it locks the mutex, the queue is empty. The solution is that the thread should dequeue, and then after it dequeues and unlocks the mutex, check whether it dequeued NULL.
You can't expect the work to be done in the "correct" order when using threads. You use threads when you don't care about the order.
If some threads don't print any words, that's normal - your "work" is very quick and you have too many threads for the amount of work. It's understandable that some threads don't get any work because all the work is already done by the time they start up. You can fix this for the demonstration by putting a call like sleep(1) before or after the printf, to slow the thread down.
Sometimes your thread IDs are wrong. When each thread starts, it accesses the id variable in main. The pthread_create function doesn't wait for the thread to read the variable. main sets id to 0, and it starts a thread, and it sets id to 1, and it starts a thread, ..., and it sets id to N and stops looping. When does the thread read the variable? No idea, could be any time, so the number it reads could be higher than the right number.
You can fix this by either making a new variable per thread to hold its ID (with new or just with an array of 8 separate ints) or you can use a little trick, and cast the ID to a pointer since it doesn't have to be a real pointer:
// No &. Just take the number and use that number as a pointer.
// vvvvvvvv
pthread_create(&thread[id], NULL, (void *) run, (void *)id);
// ^^^^^^^^
// don't know why you wrote (void *) here
in the run function:
int id = (int)arg;
// no *. This isn't a pointer to an int, it's just an int hiding in a pointer variable
Right now, I have an unbounded synchronous queue that enqueues data and sends a dequeueReady signal. The main thread will spawn a thread pool that will start to dequeue from this queue. However, there will come a point when the queue is closed and no more data will be coming in, at that point, I want all the threads to wake up and terminate.
Here is my queue implementation:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include "unbounded_queue.h"
struct Node { // These are private and not meant to be accessed by client code.
void *data;
struct Node *next;
};
struct Queue *initQueue() {
struct Queue *queue = (struct Queue *) malloc(sizeof(struct Queue));
if (queue == NULL) {
perror("Init Queue Memory Allocation Failed:");
exit(2);
}
queue->head = queue->tail = NULL;
queue->queueSize = 0;
queue->jobComplete = false;
pthread_mutex_init(&queue->lock, NULL);
pthread_cond_init(&queue->dequeueReady, NULL);
return queue;
}
void *enqueue(struct Queue *queue, void *item, size_t itemSize) {
pthread_mutex_lock(&queue->lock); // FIXME: CHECK RETURN VALUES OF ALL THIS! IF IT FAILS,
struct Node *tempNode = (struct Node *) malloc(sizeof(struct Node));
tempNode->data = malloc(itemSize); // Don't forget your null terminator
memcpy(tempNode->data, item, itemSize);
tempNode->next = NULL;
if (isEmpty(queue)) {
queue->head = queue->tail = tempNode; // If empty, head and tail point to the same thing.
pthread_mutex_unlock(&queue->lock);
return tempNode->data;
}
queue->tail->next = tempNode; // tail points to old tempNode. This says old tempNode now points to new tempNode
queue->tail = tempNode; // Current tail also needs to point to new tempNode
queue->queueSize++;
pthread_cond_signal(&queue->dequeueReady);
pthread_mutex_unlock(&queue->lock); // FIXME: Check return value
return item;
}
void *dequeue(struct Queue *queue) {
pthread_mutex_lock(&queue->lock); // FIXME: CHECK RETURN VALUES OF ALL THIS! IF IT FAILS,
while (isEmpty(queue)) {
if (queue->jobComplete) {
pthread_mutex_unlock(&queue->lock);
return NULL;
}
pthread_cond_wait(&queue->dequeueReady, &queue->lock);
}
struct Node *tempNode = queue->head; // Get node to dequeue
char *data = tempNode->data; // Need to store the data so we can use it later. Client is responsible for freeing this because it points to dynamic memory. This could be improved so the client doesn't have to do anything, but it's fine for now.
queue->head = queue->head->next;
if (queue->head == NULL) {
queue->tail = NULL;
}
queue->queueSize--;
free(tempNode);
pthread_mutex_unlock(&queue->lock); // FIXME: CHECK RETURN VALUES OF ALL THIS! IF IT FAILS,
return data;
}
void jobComplete(struct Queue *queue) {
pthread_mutex_lock(&queue->lock);
queue->jobComplete = true;
pthread_cond_broadcast(&queue->dequeueReady);
pthread_mutex_unlock(&queue->lock);
return;
}
bool isEmpty(struct Queue *queue) {
return (queue->head == NULL);
}
From the main thread, I am creating the thread pool and starting up the threads by dequeuing from the queue. However, eventually, some of the threads will be waiting for a dequeue signal that will never come. Hence, I created the jobComplete function. Once the main thread realizes that queue1 is empty, that means no more data will be coming in and we can call jobComplete and stop.
Here is how I am spawning the worker threads:
while (!isEmpty(queue1)) {
// The main thread here will call a function that enqueues data to queue1 and queue2.
// Data from queue1 is dequeued here and used.
while (!isEmpty(queue2)) { // While the queue is not empty, keep dequeuing and spawn a new worker thread.
for (int x = 0; x < threadNum; x++) {
pthread_create(&threads[x], NULL, wrapFile, dequeue(queue2));
}
for (int x = 0; x < threadNum; x++) {
if (isEmpty(queue1)) { // If queue1 is empty, that means nothing else will be added to queue2.
jobComplete(queue2);
}
pthread_join(threads[x], NULL); // Wait for thread to finish.
}
}
}
When I try to run this, it won't detect that the queue is empty and therefore jobCompleted won't run. I don't know how to join threads that are done and broadcast to all the other threads that are asleep to terminate. As of right now, the queue is infinitely waiting for more data to be enqueued, but I know more data won't be enqueued because queue1 is empty at this point.
I'm new to C and even newer to multithreading and attempting to create a threadsafe queue datastructure. It's getting a seg fault error at line 116 in dequeue() (commented in the code), and I was wondering if anyone could see any obvious issues with my implementation. Thanks so much
typedef struct node{
char *data;
struct node *next;
} node;
extern int directoryThreads;
//unbounded queue for file and directory queue
typedef struct {
node *head;;
node *tail;
int activeThreads;
int open;
int count;
pthread_mutex_t lock;
pthread_cond_t read_ready;
//pthread_cond_t write_ready;
} queue_t;
char *dequeue(queue_t *Q)
{
pthread_mutex_lock(&Q->lock); //lock queue
if(isempty(Q)){
Q->activeThreads--;
if(Q->activeThreads == 0){
pthread_mutex_unlock(&Q->lock);
return NULL;
}
while (isempty(Q) && Q->activeThreads>0) {
pthread_cond_wait(&Q->read_ready, &Q->lock);
}
if (isempty(Q)){
pthread_mutex_unlock(&Q->lock);
return NULL;
}
Q->activeThreads++;
}
//printf("%s", "Dequeued: ");
//display(Q->head);
char *item = (char *) malloc(strlen(Q->head->data) + 1); //segault here
item = Q->head->data;
if(Q->count>1){
Q->head = Q->head->next;
}
else{
Q->head = NULL;
}
Q->count--;
pthread_mutex_unlock(&Q->lock);
return item;
}
The problem is elsewhere. (I didn't rule out a problem with the signaling protocol that leads to a race condition, but I doubt it. More on that later.)
If the signal is thrown by Q->head->data, Q->head contains garbage, or isempty(Q) and Q->head != NULL are in disagreement.
If the signal is thrown by strlen, Q->head->data contains garbage, or the string isn't properly NUL-terminated.
That doesn't mean there are no issues with dequeue.
You allocate memory in dequeue for no reason.
Worse, you overwrite the pointer returned by malloc on the very next line, causing a memory leak.
dequeue never changes Q->tail even though it should be NULL when Q->head is NULL.
The signaling protocol is overly complex.
Fixed:
// Call done() when nothing will be added to the queue anymore.
// This is done to unblock calls to dequeue, and
// to the cause future calls to return immediately.
void Queue_done(queue_t *Q) {
pthread_mutex_lock(&Q->lock);
Q->done = 1;
// In case another thread is blocked in dequeue().
pthread_cond_signal(&Q->read_ready);
pthread_mutex_unlock(&Q->lock);
}
char *Queue_dequeue(queue_t *Q) {
pthread_mutex_lock(&Q->lock);
while (!Q->head && !Q->done)
pthread_cond_wait(&Q->read_ready, &Q->lock);
char *rv;
if (Q->head) {
rv = Q->head->data;
Q->head = Q->head->next;
if (!Q->head)
Q->tail = NULL;
--Q->count;
} else {
// done() was called and queue is empty.
rv = NULL;
}
// In case another thread is blocked in dequeue().
pthread_cond_signal(&Q->read_ready);
pthread_mutex_unlock(&Q->lock);
return rv;
}
Example program:
static queue_t Q;
void consumer(void) {
while (1) {
char *job = Queue_dequeue(&Q);
if (!job)
break;
// Do something with `job`.
free(job);
}
}
int main(void) {
Queue_init(&Q);
// Creates consumer threads here.
// Add stuff to queue here.
// -or-
// Create producer threads wait for them to complete here.
Queue_done(&Q);
// Wait for consumer threads to complete here.
Queue_destroy(&Q);
}
I have previously posted a working thread-safe queue implementation with demo. (It uses a fixed-sized circular buffer rather than a linked list, though.)
I have used multithreading library before in Python, but this is the first time I am trying threading in C. I want to create pool of workers. In turn, these workers supposed to push to or pop from queue.Following code is not quite there yet, but is what I have done so far:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#define NUMTHREADS 20 /* number of threads to create */
typedef struct node node;
typedef struct queue queue;
struct node {
char *name;
node *next;
};
struct queue {
node *head;
node *tail;
};
/* pop: remove and return first name from a queue */
char *pop(queue *q)
{
if (q->head == NULL)
return NULL;
char *name = q->head->name;
node *tmp = q->head;
q->head = q->head->next;
free(tmp);
return name;
}
/* push: add name to the end of the queue */
int push(queue *q, char *name)
{
node *new = malloc(sizeof(node));
if (new == NULL)
return -1;
new->name = name;
new->next = NULL;
if (q->tail != NULL)
q->tail->next = new;
q->tail = new;
if (q->head == NULL) /* first value */
q->head = new;
return 0;
}
/* printname: get a name from the queue, and print it. */
void *printname(void *sharedQ)
{
queue *q = (queue *) sharedQ;
char *name = pop(q);
if (name == NULL)
pthread_exit(NULL);
printf("%s\n",name);
pthread_exit(NULL);
}
int main()
{
size_t i;
int rc;
pthread_t threads[NUMTHREADS];
char *names[] = {
"yasar",
"arabaci",
"osman",
"ahmet",
"mehmet",
"zeliha"
};
queue *q = malloc(sizeof(queue));
q->head = NULL;
q->tail = NULL;
/* number of elements in the array */
size_t numelems = sizeof(names) / sizeof(char *);
for (i = 0; i < numelems; i++) /* push each name */
push(q, names[i]);
for (i = 0; i < NUMTHREADS; i++) { /* fire up threads */
rc = pthread_create(&threads[i], NULL, printname,
(void *)q);
if (rc) {
printf("Error, return code from pthread is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}
I tried above code, and it always printed each name exactly once. It didn't skip any names, or printed same name twice. On the other hand, I am not sure how thread safe this queue implementation is. So my question is, Is this a threadsafe queue? If not, why not? And how to make it thread safe?
The code is not thread safe.
The push and pop functions are not thread safe. In the code, the push is only being executed by a single thread, so it doesn't matter, but the pops are being executed by multiple threads.
1. char *name = q->head->name;
2. node *tmp = q->head;
3. q->head = q->head->next;
4. free(tmp);
Imagine thread A executes up to and including line 2. Thread B then executes up to and including line 4. Thread A resumes execution. It finds that q->head has already been free()ed.
Now, this so far discusses logical issues.
However, there are physical issues to consider.
Imagine we had a locking mechanism whereby threads could synchronize their behaviour, such that only one thread at a time could execute the code in the lines 1 to 4, e.g. a mutex, which is an object only one thread can 'hold' at a time, and where attempting to get the mutex blocks the thread until the holding thread releases.
0. get mutex
1. char *name = q->head->name;
2. node *tmp = q->head;
3. q->head = q->head->next;
4. free(tmp);
5. release mutex
We would still have a problem, in that the writes performed by any given CPU core (not thread) are visible immediately only to threads on that core; not to threads on other cores.
It is not enough merely to sychronize execution; at the same time, we must also ensure the writes performed by a core become visible to other cores.
(Un)fortunately, all modern sychronization methods also perform this write flushing (e.g. when you get a mutex, you also flush all writes to memory). I say unfortunately, because you don't -always- need this behaviour and it is harmful to performance.
It is not thread-safe since multiple threads may modify the pointers in the linked list at the same time, potentially corrupting it.
Here you have an answer for a very similar question:
Multiple-writer thread-safe queue in C
There you can see how to make the queue thread-safe.
I am tasked with making a queue data structure in C, as a linked list. Our lecturer gave us a large amount of code to implement a stack, but we have to adapt it to create a queue. The code our lecturer gave us ends up not compiling and segfaulting at the exact same point as the code I wrote for the queue. I'm very new to structs, malloc and C in general, so there could be something painfully obvious I've overlooked.
Here is the code I am using:
#include <stdio.h>
#include <stdlib.h>
struct node{
int data; //contains the actual data
struct node *prev; //pointer to previous node (Closer to front)
struct node *next; //pointer to next node (Closer to back)
};
typedef struct node *Nodepointer;
struct queue{
Nodepointer front;
Nodepointer back;
};
typedef struct queue *Queuepointer;
main(){
Queuepointer myqueue; //create a queue called myqueue
init(myqueue); //initialise the queue
Nodepointer new = (Nodepointer)malloc(sizeof(struct node));
myqueue->front = new;
}
int init(Queuepointer q){
q = (Queuepointer)malloc(sizeof(struct queue));
q->front = NULL;
q->back = NULL;
}
The idea is that the queue struct 'contains' the first and last nodes in a queue, and when a node is created, myqueue is updated. However, I cannot even get to that part (pop and push are written but omitted for brevity). The code is segfaulting at the line
myqueue->front = new;
with the following gdb output:
Program received signal SIGSEGV, Segmentation fault.
0x08048401 in main () at queue.c:27
27 myqueue->front = new;
Any idea what I'm doing wrong?
When you call init:
int init(Queuepointer q){
q = (Queuepointer)malloc(sizeof(struct queue));
q->front = NULL;
q->back = NULL;
}
You're passing a pointer to a queue into the function, and initializing where that pointer points (in memory) within the function. By setting q = ..., you're assigning a new value to q.
Unfortunately, the calling function does not see this. You need to pass a pointer to a pointer instead:
int init(Queuepointer * qp){
Queuepointer q = (Queuepointer)malloc(sizeof(struct queue));
q->front = NULL;
q->back = NULL;
// Set qp:
*qp = q;
}
Then change the calling function:
init(&myqueue);
init(myqueue); passes by value a pointer to unallocated memory.
init does nothing on it, consequently (instead, writing random things at random location).
Then, myqueue->stuff does it again.
You should have used pointer to pointer.
Init will receive queue**, and called as init(&myqueue).
Inside, *myqueue=()malloc stuff
Also, I recommend you against these typedefs. They are rather bad style.
The first problem I see is that the "init" function writes the allocated pointer in "q", that is NOT your original "myqueue". Remember that C passes its arguments by value. A possible correction (not perfect, just a hint) is
Queuepointer init(void)
Queuepointer q;
q = (Queuepointer)malloc(sizeof(struct queue));
q->front = NULL;
q->back = NULL;
return q;
}
`
And in "main":
myqueue = init();
Also beware that in your program you don't initialize the element allocated by malloc. malloc doesn't in general clean the memory it allocates.
Regards
You are passing myqueue by value so the allocation happened at init() is for the copy of myqueue not to myqueue.
So the correct version is:
int init(Queuepointer* q){
*q = (Queuepointer)malloc(sizeof(struct queue));
*q->front = NULL;
*q->back = NULL;
}
and you can call init() from main
init(&myqueue);
int init(Queuepointer q){
q = (Queuepointer)malloc(sizeof(struct queue));
q->front = NULL;
q->back = NULL;
}
Minor nitpick, but your init function has no return value so perhaps change it to:
void init(Queuepointer *q) {
or
int init(Queuepointer * qp){
Queuepointer q = (Queuepointer)malloc(sizeof(struct queue));
q->front = NULL;
q->back = NULL;
*qp = q;
if(q) {
return 1;
} else return 0;
}
Adjust according to how you want to perform error checking.