Pool Threads in pthread C - c

Question

In my Advance OS work, I need to read data from file and assign it to threads for further processing..
this is my code to read data from file and pass it to thread function
int main() {
FILE *fp = fopen(fileName, "r");
char str_pass[80][MAX_CHAR];
if (fp == NULL){
printf("Could not open file");
return 1;
}
int i=0;
pthread_t thread[THREADS];
int rc;
for(int th=1; th<=THREADS; th++)
{
if(fgets(str_pass[i], MAX_CHAR, fp) != NULL){
//printf("Hello, Thread %d\n",i);
rc = pthread_create(&thread[i], NULL, create_thread, (void*)str_pass[i]);
pthread_join(thread[i],NULL);
if(rc)
{
printf("ERROR; return code from pthread_create() is %d\n",rc);
exit(-1);
}
pthread_join(thread[i],NULL);
i++;
}
else{
printf("End of File");
exit(0);
}
}
pthread_exit(NULL);
fclose(fp);
return 0;
}
and here is my thread code;
void * create_thread(void *hash_string)
{
gen_combinations(hash_string);
//sleep(1);
pthread_exit(NULL);
return NULL;
}
This code is working fine and it is creating threads as much as I define value in THREADS variable, until unless it does not find any record in file. But now I've to do it with thread pooling concept. Because I can't generate as many threads as data in file.
So I need to implement multi-threading using thread pooling. I did
some search on it but didn't get any clearance on it. And now I'm
totally stuck here and not getting any idea from where to start and
how to do this work???
Any kind of help will be appreciated.

Answer 1

Look up the "producer / consumer problem" if you are not already familiar with it, specifically the variation with multiple consumers. Your main thread plays the role of the producer, reading data and parameters from the input file, and packaging them into tidy units of work -- these are the widgets being produced. The worker threads play the role of the consumers, accepting units of work and "consuming" them, as it were, by actually performing the work described. That's a thread pool.
Your particular implementation probably does not need to be generic. It can be designed and tuned as appropriate to specifically serve the problem at hand.

Answer 2

rc = pthread_create(&thread[i], NULL, create_thread, (void*)str_pass[i]);
pthread_join(thread[i],NULL);
if(rc)
{
printf("ERROR; return code from pthread_create() is %d\n",rc);
exit(-1);
}
pthread_join(thread[i],NULL);
Two major bugs here:
You call pthread_join right after you create the thread. That means you wait for the thread to complete. What's the point of creating the thread then?
Once pthread_join returns, the thread no longer exists and its ID is now invalid. Yet you call pthread_join again and pass it the thing that was the ID before but is now junk. The consequences of doing this are entirely unpredictable.

Answer 3

In your example you only have one thread active at a time. You start it and wait till it ends up before reading the next line from the file. It is caused by the pthread_join which follows the thread creation. Instead you need to move the join statement outside of the loop to join all the threads you created.
Now, you can limit the number of thread without creating a pool. You just need an atomic counter which you would increment before the thread starts and decrement when the thread finishes. This way you can check the value of the counter before creating the next thread and wait on a conditional variable. This would be a kind of a semaphore.
Schematically something like the following:
counter = 0;
while {
lock counter;
while (counter > 8)
wait_on_conditional_variable
counter ++;
unlock counter;
run thread;
}
join all remaining threads.
in the thread
do work;
lock counter;
counter --;
signal cond var;
unlock counter;
return;
For the pool you would need to start several threads. Every thread has a loop where it waits for some job to be available. Most likely getting it from a fifo. It has to wait on a conditional variable to check.
thread
do {
lock fifo;
while (fifo.size == 0)
wait on conditional variable.
read job from fifo;
unlock fifo;
do work;
} while (!exiting);
while you reading the file, the following should be done
while ... {
lock fifo;
push line into fifo;
signal var;
unlock fifo;
}
set-exit-condition;
join the pools.
I hope this would help.
But there are multiple ways and optimizations which you could do there.