I'm trying to make a producer/ consumer application. The problem is, my producer is fillin the whole buffer THEN the consumer is removing the whole buffer... wasn't the consumer supposed to remove an item from the buffer as soon as the producer make a post to the semaphore ?
my code:
void* producerFunc(void* arg)
{
while(n_insertions < N_PRODUCTS)
{
sem_wait(&sem_postAvaliable);
sem_wait(&mutex);
//Insert Item
sem_post(&mutex);
sem_post(&sem_posTaken);
}
pthread_exit(NULL);
}
void* consumerFunc(void* arg)
{
while(n_consumed < N_PRODUCTS)
{
sem_wait(&sem_posTaken);
sem_wait(&mutex);
//Remove from bufer
sem_post(&mutex);
sem_post(&sem_postAvaliable);
}
pthread_exit(NULL);
}
n_insertionsis a global integer that counts the number of items inserted into the buffer;
n_consumed is a global integer that counts the number of times the consumer consumed an item from the buffer.
At a very high level, when you do a sem_post(&sem_posTaken) you are telling the consumers "Hey!, you can consume something", but after that depending on the processor scheduler the producers may continue producing until the counter of the semaphore doesn't allow them, so maybe a consumer consume right after the producer insert something or maybe after 3 inserts, who knows? The thread is blocked when you try to make a sem_wait of a semaphore whose counter is 0.
If you want to alternate between an insert and a remove, try initializing sem_postAvaliable to 1, that way you would allowing only one insertion at first, because the other threads would be blocked waiting a sem_post(&sem_postAvaliable), then only one consumer could consume whatever you inserted in the buffer and so on, but note that in this case a buffer with size larger than 1 would be meaningless.
Related
I'm currently learning concurrent programming. I have two threads and I want them to act like this:
the first thread runs N times while the second one waits
when the first one is done, the second does its job and the first waits
when the second thread is done, repeat.
I'm trying to do it in C using pthread library. Here's some pseudocode (hopefully understandable)
int cnt = 0;
void* thread1(){
while(1){
// thread 1 code
cnt ++;
if(cnt == N){
let_thread2_work();
wait_thread2();
}
}
}
void* thread2(){
while(1){
wait_thread1();
// thread2 code
cnt = 0;
let_thread1_work();
}
}
Can anyone please help me ?
Like David Schwartz commented, this doesn't make a ton of sense with just thread1 and thread2 waiting on each other, not actually doing any work in parallel. But maybe eventually you want multiple "thread1"s, all processing jobs at the same time until they finish a batch of N jobs, then they all stop and wait for "thread2" to do some kind of post-processing before the pool of worker threads start back up.
In this situation I would consider using a couple condition variables, one for your worker threads to communicate to your post-processing thread that they're waiting, and one for your post-processing thread to tell the workers to start working again. You can declare a condition variable and a helper mutex in the global scope right next to your "cnt" variable, which I'm calling "jobs_done" for clarity:
#include<pthread.h>
#DEFINE NUM_WORKERS 1 // although it doesn't make sense to just have 1
#DEFINE BATCH_SIZE 50 // this is "N"
// we're going to keep track of how many jobs we have done in
// this variable
int jobs_done = 0;
// when a worker thread checks jobs_done and it's N or greater,
// that means we have to wait for the post-processing thread to set
// jobs_done back to 0. so the worker threads "wait" on the
// condition variable, and the post-processing thread "broadcasts"
// to the condition variable to wake them all up again once it's
// done its work
pthread_cond_t jobs_ready_cv;
// we're going to use this helper mutex. whenever any thread
// reads or writes to the jobs_done variable, we have to lock this
// mutex. that includes the worker threads when they check to see if
// they're ready to wake up again.
pthread_mutex_t jobs_mx;
// here's how the worker threads will communicate to the post-process
// thread that a batch is done. to make sure that all N jobs are fully
// complete before postprocessing happens, we'll use this variable to
// keep track of how many threads are waiting for postprocessing to finish.
int workers_done = 0;
// we'll also use a separate condition variable and separate mutex to
// communicate to the postprocess thread.
pthread_cond_t workers_done_cv;
pthread_mutex_t workers_done_mx;
Then in your setup code, initialize the condition variables and helper mutexes:
int main() { // or something
pthread_cond_init(&jobs_ready_cv, NULL);
pthread_mutex_init(&jobs_mx, NULL);
pthread_cond_init(&workers_done_cv, NULL);
pthread_mutex_init(&workers_done_mx, NULL);
...
}
So, your worker threads (or "thread1"), before taking a job, will check to see how many jobs have been taken. If N (here, BATCH_SIZE) have been taken, then it updates a variable to indicate that it has no work left to do. If it finds that all of the worker threads are done, then it signals the postprocess thread ("thread2") through workers_done_cv. Then, the thread waits for a signal from the postprocess thread through `
void* worker_thread(){
while(1){
/* first, we check if the batch is complete. we do this first
* so we don't accidentally take an extra job.
*/
pthread_mutex_lock(&jobs_mx);
if (jobs_done == BATCH_SIZE) {
/* if BATCH_SIZE jobs have been done, first let's increment workers_done,
* and if all workers are done, let's notify the postprocess thread.
* after that, we release the workers_done mutex so the postprocess
* thread can wake up from the workers_done condition variable.
*/
pthread_mutex_lock(&workers_done_mx);
++workers_done;
if (workers_done == NUM_WORKERS) {
pthread_cond_broadcast(&workers_done_cv);
}
pthread_mutex_unlock(&workers_done_mx);
/* now we wait for the postprocess thread to do its work. this
* unlocks the mutex. when we get the signal to start doing jobs
* again, the mutex will relock automatically when we wake up.
*
* note that we use a while loop here to check the jobs_done
* variable after we wake up. That's because sometimes threads
* can wake up on accident even if no signal or broadcast happened,
* so we need to make sure that the postprocess thread actually
* reset the variable. google "spurious wakeups"
*/
while (jobs_done == BATCH_SIZE) {
pthread_cond_wait(&jobs_ready_cv, &jobs_mx);
}
}
/* okay, now we're ready to take a job.
*/
++jobs_done;
pthread_mutex_unlock(&jobs_mx);
// thread 1 code
}
}
Meanwhile, your postprocess thread waits on the workers_done_cv immediately, and doesn't wake up until the last worker thread is done and calls pthread_cond_broadcast(&workers_done_cv). Then, it does whatever it needs to, resets the counts, and broadcasts to the worker threads to wake them back up.
void* postprocess_thread(){
while(1){
/* first, we wait for our worker threads to be done
*/
pthread_mutex_lock(&workers_done_mx);
while (workers_done != NUM_WORKERS) {
pthread_cond_wait(&workers_done_cv, &workers_done_mx);
}
// thread2 code
/* reset count of stalled worker threads, release mutex */
workers_done = 0;
pthread_mutex_unlock(&workers_done_mx);
/* reset number of jobs done and wake up worker threads */
pthread_mutex_lock(&jobs_mx);
jobs_done = 0;
pthread_mutex_unlock(&jobs_mx);
pthread_cond_broadcast(&jobs_ready_cv);
}
}
Also take heed of David Schwartz's advice that you probably don't actually need the postprocessing thread to wait on the worker threads. If you don't need this, then you can get rid of the condition variable that makes the worker threads wait for the postprocess thread, and this implementation becomes a lot simpler.
edit: mutex protected the assignment to jobs_done in postprocess_thread(), added a forgotten ampersand
One solution is to use mutex:
https://www.geeksforgeeks.org/mutex-lock-for-linux-thread-synchronization/
Another solution, but using semaphore:
https://www.geeksforgeeks.org/difference-between-binary-semaphore-and-mutex/?ref=rp
Or creating own lightthread schedule.
for exemple:
http://www.dunkels.com/adam/pt/ [edited]
I'm tring to solve producer consumer problem by using monitoring thread.
Basically, when we try to solve this problem, we can easly solve it by using semaphore and mutex(semaphore detect empty or full state keep stable critical section, and mutex keep shared part where producer producing, consumer consuming)
In this problem I want to using monitoring thread to check how many item is in shared memory and this thread must enter critical section after producer or consumer doing their work. It kind of acknowledgement thing
For example
Produce
Ack, buffer have 1 item
Consume
Ack, buffer have 0 item
Produce
Ack, buffer have 1 item
Produce
Ack, buffer have 2 item
I was thinking use two more sem_t,
sem_t m1;
sem_t m2;
Void *produce()
{
sem_wait(&m1);
sem_wait(&m2);
....
sem_signal(&m1);
}
Void *consumer()
{
sem_wait(&m1);
sem_wait(&m2);
....
sem_signal(&m2);
}
Void *monitor()
{
if(m1==o && m2==1)
sem_wait();
else if(m1==1 && m2==0)
sem_wait(m1);
....
sem_signal(m1);
sem_signal(m2);
}
producer and consumer only can enter critical section when both m1 and m2 is bigger then 0, and when they done their tasks they call sem_signal just one time for m1 or m2. this block producer and consumer, only monitoring thread can enter critical section. after monitoring thread finishing own task, it calls sem_signal for m1, m2, then producer or consumer can enter the critical section (each of sem object is kind of mutex, binary semaphore)
but it doesn't work..
I can't find what is my problem.
Is Sem object can't use if statement?
or, my idea is have something problem?
thanks for reading my long text
The Producer-Consumer problem is solved using this Semaphore algorithm where the Semaphore is decremented by the buffer size then the Semaphore is decremented by 1 signifying the critical section.
If these operations are happening consecutively back-to-back why is it not correct to decrement by 1 (the mutex) and then decrement the buffer size?
I know both the producer and consumer will go to sleep at the same time creating a deadlock but why does this minor switch cause the entire algorithm to fail?
BufferSize = 3;
semaphore mutex = 1; // Controls access to critical section
semaphore empty = BufferSize; // counts number of empty buffer slots
semaphore full = 0; // counts number of full buffer slots
Producer()
{
int widget;
while (TRUE) { // loop forever
make_new(widget); // create a new widget to put in the buffer
down(&empty); // decrement the empty semaphore
down(&mutex); // enter critical section
put_item(widget); // put widget in buffer
up(&mutex); // leave critical section
up(&full); // increment the full semaphore
}
}
Consumer()
{
int widget;
while (TRUE) { // loop forever
down(&full); // decrement the full semaphore
down(&mutex); // enter critical section
remove_item(widget); // take a widget from the buffer
up(&mutex); // leave critical section
up(&empty); // increment the empty semaphore
consume_item(widget); // consume the item
}
}
Code Source: Resource
Its not correct because what if the producer enters the code section by wait(mutex) but then it finds out there are no empty blocks left so it has to wait on the empty semaphore. At the same time when a consumer tries to enter it has to wait on mutex as producer has locked mutex. Thus there is a deadlock as producer is waiting for consumer to consume and consumer is waiting for producer to signal(mutex)
This is not the case with original arrangement of mutex and semaphores where a producer or consumer is only allowed to wait on mutex when it has decremented the semaphore value. Hence no deadlock.
Lets say I have a buffer that has 3 producer threads and 5 consumer threads inserting and consuming to/from the buffer.
I only want to allow 1 producer or up to 3 consumer threads access the buffer at any given time.
Up to 3 consumers can peek at the top element in the buffer, only, if no producer is accessing it. If more than 1 consumer thread does access the buffer, the last thread to leave must delete the top element.
Now this is part of a class assignment, and the assignment explicitly states to use semaphores. However, I can't think of a way to really implement this wording exactly using only semaphores.
The pseudo code -I think- should look like this: (I'm not worrying about an empty or full buffer, just this sub-part of the problem)
sem_init(&binary, 0, 1); //Init binary semaphore to 1
sem_init(&consumerCount, 0 , 3); //Allows 3 consumers to access
producer()
{
createItem()
sem_wait(&binary)
appendItem()
sem_post(&binary)
}
//The above assures nothing else can access buffer while appending an item
consumer()
{
while( binary unlocked)
{
sem_wait(&binary) and sem_wait(&consumerCount) //Locks the producers out
//Allows 3 consumers in
peek() //Gets copy of top item
if( last consumer out )
{
delete() //Deletes top item
sem_post(&binary) //Allow producer access back since last one out
}
sem_post(&consumerCount)
}
}
I think that's the gist of the logic, problem is how to implement this with just semaphores. How do I allow only 1 producer in with a semaphore but allow 3 consumers in on the other side? It seems like I would need to use something besides a semaphore.
Also, please correct any of the logic if needed, this is meant to just be a general idea.
You can solve the problem with two semaphores. The first semaphore is used for exclusive access by producers. The second semaphore is used for the shared access. The producer tries to acquire all three permits in order to lock out the consumers.
sem_init(&exclusive, 0, 1);
sem_init(&shared, 0, 3);
void producer()
{
sem_wait(&exclusive);
sem_wait(&shared);
sem_wait(&shared);
sem_wait(&shared);
// critical section
sem_post(&shared);
sem_post(&shared);
sem_post(&shared);
sem_post(&exclusive);
}
void consumer()
{
sem_wait(&shared);
// critical section
sem_post(&shared);
}
I have assignment to work on producer and consumer problem by use thread and semaphore. The task is that allow user to define # of producer,# of consumer and buffer size. The program always lock if producer reach the buffersize. But the requirement says if producer reach buffersiz the consumer thread should start and take things from buffer. I am out of idea how to fix this problem and my teacher refuse to help. I am a totally beginner of the C language, please give me some suggestion. Thank you very much
My program can run when Producer = Consumer, or Producer < Consumer, except Producer > Buffer Size, it seems to appear deadlock, and I think I understand the reason why but I don't know how to fix the code to let Consumer thread run first than back to Producer thread.
Here is the running result when producer =3 consumer = 1 and buffersize = 2
./Task2 3 1 2
Producer 0 has started
Producer 0:Put item 0.
Producer 1 has started
Producer 1:Put item 1.
Producer 2 has started
The requirement says the result should looks like
Started
Producer 0 has started
Producer 0: Put item 0.
Producer 1 has started
Producer 1: Put item 1.
Producer 2 has started
Consumer 0 has started
Consumer 0: Taked item 0.
Producer 2: Put item 2.
Terminated!
Here is my origional code, I have discard some input error check code
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <semaphore.h>
pthread_t *pid, *cid;
void *producer(void *param);
void *consumer(void *param);
void init();
int Remove();
struct prot_buffer{
int Producer;
int Consumer;
int *buffer;
int buffersize;
int front;
int rear;
int item;
sem_t mutex;
sem_t slots;
sem_t items;
}b;
main(int argc, char *argv[]){
int c1;
b.Producer = atoi(argv[1]);
b.Consumer = atoi(argv[2]);
b.buffersize = atoi(argv[3]);
init();
pid = (pthread_t *)malloc(b.Producer *sizeof(pthread_t));
cid = (pthread_t *)malloc(b.Consumer *sizeof(pthread_t));
for (c1=0; c1< b.Producer; c1++){
printf("Producer %d has started\n", c1);
pthread_create(&(pid[c1]),NULL, producer, NULL);
pthread_join(pid[c1], NULL);
printf("Producer %d:Create item %d.\n", c1,c1);
}
/* Create the consumer threads */
for (c1=0; c1<b.Consumer; c1++){
printf("Consumer %d has started\n", c1);
pthread_create(&(cid[c1]),NULL, consumer, NULL);
if (b.front==b.rear){
printf("Terminated!\n");
exit(0);
}
pthread_join(cid[c1], NULL);
printf("Consumer %d:Taked item %d.\n", c1, c1);
}
free(b.buffer);
free(pid);
free(cid);
sem_destroy(&b.items);
sem_destroy(&b.slots);
sem_destroy(&b.mutex);
printf("Threads terminated!\n");
exit(0);
}
void *producer(void *param){
sem_wait(&b.slots); sem_wait(&b.mutex);
if(b.rear<=b.buffersize){
b.buffer[b.rear] = b.item;
b.rear++;
sem_post(&b.mutex); sem_post(&b.items);
}else{
sem_post(&b.mutex); sem_post(&b.items); }
}
void *consumer(void *param){
Remove();
}
void init(){
b.buffer = (int *) malloc(b.buffersize *sizeof(int));
b.buffersize = b.buffersize;
b.front = b.rear =0;
sem_init(&b.items, 0, 0);
sem_init(&b.slots,0,b.buffersize);
sem_init(&b.mutex, 0, 1);
}
int Remove(){
sem_wait(&b.items);
sem_wait(&b.mutex);
b.item = b.buffer[b.front];
b.front++;
sem_post(&b.mutex);
sem_post(&b.slots);
return b.item;
}
My new code
main(int argc, char *argv[]){
...
pthread_create(&pid,NULL, producer, NULL);
pthread_create(&cid,NULL, consumer, NULL);
....
}
void *producer(void *param){
int c2;
for (c2=0; c2 < b.Producer; c2++) {
printf("Producer %d has started\n", c2);
b.item = c2;
sem_wait(&b.slots);
sem_wait(&b.mutex);
b.buffer[b.rear] = b.item;
b.rear = (b.rear+1)%b.buffersize;
printf("Producer %d:Put item %d.\n", c2,c2);
sem_post(&b.mutex);
sem_post(&b.items);
}
return NULL;
}
void *consumer(void *param){
int c2;
for (c2=0; c2 < b.Consumer; c2++) {
printf("Consumer %d has started\n", c2,c2);
b.item = c2;
sem_wait(&b.items);
sem_wait(&b.mutex);
b.buffer[b.front] = b.item;
b.front = (b.front+1)%b.buffersize;
printf("Consumer %d:take item %d.\n", c2, c2);
sem_post(&b.mutex);
sem_post(&b.slots);
}
return NULL;
}
To avoid trobule at school I remove some code and some description.
The program result is correct now, thanks for help. In this case I use b.item as the variable to display the item left inside the buffer, but its wrong. Use other variable like front or rear also not work too.
Program result-
Producer=2, Consumer=2, Buffer=2
./F 2 2 2
started
Producer 0 has started
Producer 0:Put item 0.
Producer 1 has started
Producer 1:Put item 1.
Consumer 0 has started
Consumer 0:Take item 0.
Consumer 1 has started
Consumer 1:Take item 1.
1 item(s) left in the buffer! //This is wrong!
Terminated!
Producer=3, Consumer=1, Buffer=2
./F 3 1 2
started
Producer 0 has started
Producer 0:Deposited item 0.
Producer 1 has started
Producer 1:Deposited item 1.
Producer 2 has started
Consumer 0 has started
Consumer 0:Removed item 0.
Producer 2:Deposited item 2.
0 item(s) left in the buffer! //Still wrong!
Terminated!
Producer =2, Consumer = 5, Buffer =3
./F 2 5 3
started
Producer 0 has started
Producer 0:Put item 0.
Producer 1 has started
Producer 1:Put item 1.
Consumer 0 has started
Consumer 0:Take item 0.
Consumer 1 has started
Consumer 1:Take item 1.
Consumer 2 has started
2 item(s) left in the buffer! //Wrong again!
Terminated!
Your buffer size is 2. The first 2 producers fill up this buffer. Hence the third one is waiting for a consumer to take one item so that it can add to buffer. But your pthread_join inside the producer loop never allows for consumers to be created at all! pthread_join is suspending main process until 3rd producer terminates. Hence the deadlock where in the third Producer is indefinitely waiting for the buffer to be freed by a consumer who never arrives.
I suggest you go through Exercise 3 of this article which deals with exactly the same problem and the same data structure as yours. They've clearly articulated how to prevent buffer overflow, where to print producer data. May be its a standard semaphore tutorial exercise for grads.
Your initialization and use of the semaphores seems correct - they are a natural choice for producer-consumer queues - it's condvars that are not, (condvars have no count and so require while loops for correct operation even if your OS does not support the spurious wakeup feature).
Your use of the buffer, however, seems a bit 'off'. I presume that the buffer is supposed to be circular? If so, you should be setting the index back to 0 whenever the front, or rear, index reaches the end of the array. You should be doing this inside the mutex lock in both the producer and consumer code. The check on the indexes should be solely to reset them to the start of the buffer and should not change the operation of the semaphore signal/wait in any way.
That, and you seem to be joining the threads as soon as they are created. Usually, to test/demonstrate producer/consumer operation, threads are not continually created, immediately joined, (so that the constructing thread has to immediately wait), and then left to terminate. It's more common to run up some producer and consumer threads that loop, continually producing/consuming messages, (perhaps with a very short sleep loop - those kind of loops that some developers say have no valid use and are an anti-pattern).
Continually creating/terminating/destroying threads is very wasteful and will result, in your test code, in an app that is performing nearly all overhead and very little producer/consumer.
Last thing - don't join to threads without thinking about it for a second or two first. It may not be optimal, it may not be necessary, it may be a hindrance, it may be a disaster. If the join is solely to keep your main thread, (and so process), from terminating early, it may be better to find another way - wait on keyboard input or use a long-term sleep() loop.