Threading across multiple files - c

Question

My program is reading in files and using thread to compute the highest prime number, when I put a print statement into the getNum() function my numbers are printing out. However, it seems to just lag no matter how many threads I input. Each file has 1 million integers in it. Does anyone see something apparently wrong with my code? Basically the code is giving each thread 1000 integers to check before assigning a new thread. I am still a C noobie and am just learning the ropes of threading. My code is a mess right now because I have been switching things around constantly.
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <string.h>
#include <pthread.h>
#include <math.h>
#include <semaphore.h>
//Global variable declaration
char *file1 = "primes1.txt";
char *file2 = "primes2.txt";
char *file3 = "primes3.txt";
char *file4 = "primes4.txt";
char *file5 = "primes5.txt";
char *file6 = "primes6.txt";
char *file7 = "primes7.txt";
char *file8 = "primes8.txt";
char *file9 = "primes9.txt";
char *file10 = "primes10.txt";
char **fn; //file name variable
int numberOfThreads;
int *highestPrime = NULL;
int fileArrayNum = 0;
int loop = 0;
int currentFile = 0;
sem_t semAccess;
sem_t semAssign;
int prime(int n)//check for prime number, return 1 for prime 0 for nonprime
{
int i;
for(i = 2; i <= sqrt(n); i++)
if(n % i == 0)
return(0);
return(1);
}
int getNum(FILE* file)
{
int number;
char* tempS = malloc(20 *sizeof(char));
fgets(tempS, 20, file);
tempS[strlen(tempS)-1] = '\0';
number = atoi(tempS);
free(tempS);//free memory for later call
return(number);
}
void* findPrimality(void *threadnum) //main thread function to find primes
{
int tNum = (int)threadnum;
int checkNum;
char *inUseFile = NULL;
int x=1;
FILE* file;
while(currentFile < 10){
if(inUseFile == NULL){//inUseFIle being used to check if a file is still being read
sem_wait(&semAccess);//critical section
inUseFile = fn[currentFile];
sem_post(&semAssign);
file = fopen(inUseFile, "r");
while(!feof(file)){
if(x % 1000 == 0 && tNum !=1){ //go for 1000 integers and then wait
sem_wait(&semAssign);
}
checkNum = getNum(file);
/*
*
*
*
* I think the issue is here
*
*
*
*/
if(checkNum > highestPrime[tNum]){
if(prime(checkNum)){
highestPrime[tNum] = checkNum;
}
}
x++;
}
fclose(file);
inUseFile = NULL;
}
currentFile++;
}
}
int main(int argc, char* argv[])
{
if(argc != 2){ //checks for number of arguements being passed
printf("To many ARGS\n");
return(-1);
}
else{//Sets thread cound to user input checking for correct number of threads
numberOfThreads = atoi(argv[1]);
if(numberOfThreads < 1 || numberOfThreads > 10){
printf("To many threads entered\n");
return(-1);
}
time_t preTime, postTime; //creating time variables
int i;
fn = malloc(10 * sizeof(char*)); //create file array and initialize
fn[0] = file1;
fn[1] = file2;
fn[2] = file3;
fn[3] = file4;
fn[4] = file5;
fn[5] = file6;
fn[6] = file7;
fn[7] = file8;
fn[8] = file9;
fn[9] = file10;
sem_init(&semAccess, 0, 1); //initialize semaphores
sem_init(&semAssign, 0, numberOfThreads);
highestPrime = malloc(numberOfThreads * sizeof(int)); //create an array to store each threads highest number
for(loop = 0; loop < numberOfThreads; loop++){//set initial values to 0
highestPrime[loop] = 0;
}
pthread_t calculationThread[numberOfThreads]; //thread to do the work
preTime = time(NULL); //start the clock
for(i = 0; i < numberOfThreads; i++){
pthread_create(&calculationThread[i], NULL, findPrimality, (void *)i);
}
for(i = 0; i < numberOfThreads; i++){
pthread_join(calculationThread[i], NULL);
}
for(i = 0; i < numberOfThreads; i++){
printf("this is a prime number: %d \n", highestPrime[i]);
}
postTime= time(NULL);
printf("Wall time: %ld seconds\n", (long)(postTime - preTime));
}
}
Yes I am trying to find the highest number over all. So I have made some head way the last few hours, rescucturing the program as spudd said, currently I am getting a segmentation fault due to my use of structures, I am trying to save the largest individual primes in the struct while giving them the right indices. This is the revised code. So in short what the first thread is doing is creating all the threads and giving them access points to a very large integer array which they will go through and find prime numbers, I want to implement semaphores around the while loop so that while they are executing every 2000 lines or the end they update a global prime number.
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <string.h>
#include <pthread.h>
#include <math.h>
#include <semaphore.h>
//Global variable declaration
char *file1 = "primes1.txt";
char *file2 = "primes2.txt";
char *file3 = "primes3.txt";
char *file4 = "primes4.txt";
char *file5 = "primes5.txt";
char *file6 = "primes6.txt";
char *file7 = "primes7.txt";
char *file8 = "primes8.txt";
char *file9 = "primes9.txt";
char *file10 = "primes10.txt";
int numberOfThreads;
int entries[10000000];
int entryIndex = 0;
int fileCount = 0;
char** fileName;
int largestPrimeNumber = 0;
//Register functions
int prime(int n);
int getNum(FILE* file);
void* findPrimality(void *threadNum);
void* assign(void *num);
typedef struct package{
int largestPrime;
int startingIndex;
int numberCount;
}pack;
//Beging main code block
int main(int argc, char* argv[])
{
if(argc != 2){ //checks for number of arguements being passed
printf("To many threads!!\n");
return(-1);
}
else{ //Sets thread cound to user input checking for correct number of threads
numberOfThreads = atoi(argv[1]);
if(numberOfThreads < 1 || numberOfThreads > 10){
printf("To many threads entered\n");
return(-1);
}
int threadPointer[numberOfThreads]; //Pointer array to point to entries
time_t preTime, postTime; //creating time variables
int i;
fileName = malloc(10 * sizeof(char*)); //create file array and initialize
fileName[0] = file1;
fileName[1] = file2;
fileName[2] = file3;
fileName[3] = file4;
fileName[4] = file5;
fileName[5] = file6;
fileName[6] = file7;
fileName[7] = file8;
fileName[8] = file9;
fileName[9] = file10;
FILE* filereader;
int currentNum;
for(i = 0; i < 10; i++){
filereader = fopen(fileName[i], "r");
while(!feof(filereader)){
char* tempString = malloc(20 *sizeof(char));
fgets(tempString, 20, filereader);
tempString[strlen(tempString)-1] = '\0';
entries[entryIndex] = atoi(tempString);
entryIndex++;
free(tempString);
}
}
//sem_init(&semAccess, 0, 1); //initialize semaphores
//sem_init(&semAssign, 0, numberOfThreads);
time_t tPre, tPost;
pthread_t coordinate;
tPre = time(NULL);
pthread_create(&coordinate, NULL, assign, (void**)numberOfThreads);
pthread_join(coordinate, NULL);
tPost = time(NULL);
}
}
void* findPrime(void* pack_array)
{
pack* currentPack= pack_array;
int lp = currentPack->largestPrime;
int si = currentPack->startingIndex;
int nc = currentPack->numberCount;
int i;
int j = 0;
for(i = si; i < nc; i++){
while(j < 2000 || i == (nc-1)){
if(prime(entries[i])){
if(entries[i] > lp)
lp = entries[i];
}
j++;
}
}
return (void*)currentPack;
}
void* assign(void* num)
{
int y = (int)num;
int i;
int count = 10000000/y;
int finalCount = count + (10000000%y);
int sIndex = 0;
pack pack_array[(int)num];
pthread_t workers[numberOfThreads]; //thread to do the workers
for(i = 0; i < y; i++){
if(i == (y-1)){
pack_array[i].largestPrime = 0;
pack_array[i].startingIndex = sIndex;
pack_array[i].numberCount = finalCount;
}
pack_array[i].largestPrime = 0;
pack_array[i].startingIndex = sIndex;
pack_array[i].numberCount = count;
pthread_create(&workers[i], NULL, findPrime, (void *)&pack_array[i]);
sIndex += count;
}
for(i = 0; i< y; i++)
pthread_join(workers[i], NULL);
}
//Functions
int prime(int n)//check for prime number, return 1 for prime 0 for nonprime
{
int i;
for(i = 2; i <= sqrt(n); i++)
if(n % i == 0)
return(0);
return(1);
}
Here is my latest update, having issues with my threads running, the only thread is thread 0 that is completing
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <string.h>
#include <pthread.h>
#include <math.h>
#include <semaphore.h>
//Global variable declaration
char *file1 = "primes1.txt";
char *file2 = "primes2.txt";
char *file3 = "primes3.txt";
char *file4 = "primes4.txt";
char *file5 = "primes5.txt";
char *file6 = "primes6.txt";
char *file7 = "primes7.txt";
char *file8 = "primes8.txt";
char *file9 = "primes9.txt";
char *file10 = "primes10.txt";
sem_t semHold;
int numberOfThreads;
long unsigned int entries[10000000];
unsigned int entryIndex = 0;
int fileCount = 0;
char** fileName;
long unsigned int largestPrimeNumber = 0;
//Register functions
int prime(unsigned int n);
int getNum(FILE* file);
void* findPrimality(void *threadNum);
void* assign(void *num);
typedef struct package{
long unsigned int largestPrime;
unsigned int startingIndex;
unsigned int numberCount;
}pack;
pack pack_array[10];
//Beging main code block
int main(int argc, char* argv[])
{
if(argc != 2){ //checks for number of arguements being passed
printf("To many threads!!\n");
return(-1);
}
else{ //Sets thread cound to user input checking for correct number of threads
numberOfThreads = atoi(argv[1]);
if(numberOfThreads < 1 || numberOfThreads > 10){
printf("To many threads entered\n");
return(-1);
}
int threadPointer[numberOfThreads]; //Pointer array to point to entries
int i;
fileName = malloc(10 * sizeof(char*)); //create file array and initialize
fileName[0] = file1;
fileName[1] = file2;
fileName[2] = file3;
fileName[3] = file4;
fileName[4] = file5;
fileName[5] = file6;
fileName[6] = file7;
fileName[7] = file8;
fileName[8] = file9;
fileName[9] = file10;
FILE* filereader;
long unsigned int currentNum;
sem_init(&semHold, 0, 1);
for(i = 0; i < 10; i++){
filereader = fopen(fileName[i], "r");
while(fscanf(filereader, "%lu" , &currentNum)!= EOF){
entries[entryIndex] = currentNum;
// while(entryIndex < 5){
//char* tempString = malloc(20 *sizeof(long unsigned int));
//fgets(tempString, 20, filereader);
//tempString[strlen(tempString)-1] = '\0';
//currentNum = atoi(tempString);
//printf("Test %lu\n",currentNum);
//entries[entryIndex] = atoi(tempString);
//entryIndex++;
//free(tempString);
//}
entryIndex++;
}
}
printf("Test %lu\n",entries[9999999]);
//sem_init(&semAccess, 0, 1); //initialize semaphores
//sem_init(&semAssign, 0, numberOfThreads);
time_t tPre, tPost;
pthread_t coordinate;
tPre = time(NULL);
pthread_create(&coordinate, NULL, assign, (void**)numberOfThreads);
pthread_join(coordinate, NULL);
tPost = time(NULL);
printf("Largest prime = %lu , time: %ld\n", largestPrimeNumber,(long)(tPost-tPre));
}
}
void* findPrime(void* pack_array)
{
pack* currentPack = pack_array;
unsigned int lp = currentPack->largestPrime;
unsigned int si = currentPack->startingIndex;
unsigned int nc = currentPack->numberCount;
int i;
printf("Starting index Count: %d\n", si);
for(i = si; i < nc; i++){
if(i%100000==0)
printf("Here is i: %d\n", i);
if(entries[i]%2 != 0){
if(entries[i] > currentPack->largestPrime){
if(prime(entries[i])){
currentPack->largestPrime = entries[i];
printf("%lu\n", currentPack->largestPrime);
if(currentPack->largestPrime > largestPrimeNumber)
sem_wait(&semHold);
largestPrimeNumber = currentPack->largestPrime;
sem_post(&semHold);
}
}
}
}
}
void* assign(void* num)
{
int y = (int)num;
int i;
int count = 10000000/y;
int finalCount = count + (10000000%y);
int sIndex = 0;
printf("This is count: %d\n", count);
printf("This is final count: %d\n", finalCount);
pthread_t workers[y]; //thread to do the workers
for(i = 0; i < y; i++){
printf("for thread %d Starting index: %d\n", i, sIndex);
if(i == (y-1)){
pack_array[i].largestPrime = 0;
pack_array[i].startingIndex = sIndex;
pack_array[i].numberCount = finalCount;
}
pack_array[i].largestPrime = 0;
pack_array[i].startingIndex = sIndex;
pack_array[i].numberCount = count;
pthread_create(&workers[i], NULL, findPrime, (void *)&pack_array[i]);
printf("thread created\n");
sIndex += count;
}
for(i = 0; i < y; i++)
pthread_join(workers[i], NULL);
}
//Functions
int prime(unsigned int n)//check for prime number, return 1 for prime 0 for nonprime
{
int i;
for(i = 2; i <= sqrt(n); i++)
if(n % i == 0)
return(0);
return(1);
}

Answer 1

OK here's part of my solution, it's missing most of main, and has some other simple stuff missing, if you choose to base your code around this you can do one of two things load all the data before starting your workers, or have the main thread load it while the workers are running, I did the latter in my complete version. However you'll have to do some work to get that to be handled correctly because currently the workers will never exit.
Also you might want to try adapting your single array code above based on this.
So if you load all the data before starting the workers you don't need the condition variable and they can just exit when next_chunk is NULL. I recommend you figure out how to get loading while the workers are running working because it'll be more efficient.
Hint: pthread_cond_broadcast()
Also missing is the actual worker function.
// A singly linked list of chunks of 1000 numbers
// we use it as a queue of data to be processed
struct number_chunk
{
struct number_chunk *next;
int size;
int nums[1000];
};
pthread_mutex_t cnklst_mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t data_available = PTHREAD_COND_INITIALIZER;
struct number_chunk *next_chunk = NULL;
void load_chunks(char *filename)
{
FILE *in = fopen(filename, "r");
int done = 0;
int i;
if(in == NULL) {
fprintf(stderr, "Failed to open file %s\n", filename);
return;
}
// read in all the chunks of 1000 numbers from the file
while(!done) {
struct number_chunk *cnk = malloc(sizeof(struct number_chunk)); // allocate a new chunk
cnk->next = NULL;
for(i=0; i < 1000; i++) { // do the actual reading
char tmp[20];
if(fgets(tmp, 20, in) == NULL) { // end of file, leave the read loop
done = 1;
break;
}
cnk->nums[i] = atoi(tmp);
}
// need to do this so that the last chunk in a file can have less than 1000 numbers in it
cnk->size = i;
// add it to the list of chunks to be processed
pthread_mutex_lock(&cnklst_mutex);
cnk->next = next_chunk;
next_chunk = cnk;
pthread_cond_signal(&data_available); // wake a waiting worker
pthread_mutex_unlock(&cnklst_mutex);
}
fclose(in);
}
struct number_chunk *get_chunk()
{
struct number_chunk *cnk = NULL;
pthread_mutex_lock(&cnklst_mutex);
//FIXME: if we finish we will never exit the thread
// need to return NULL when all the work that there will ever be
// is done, altertitively load everything before starting the workers and
// get rid of all the condition variable stuff
while(next_chunk == NULL)
pthread_cond_wait(&data_available, &cnklst_mutex);
cnk = next_chunk;
if(next_chunk != NULL) next_chunk = next_chunk->next;
pthread_mutex_unlock(&cnklst_mutex);
return cnk;
}
The way my workers report the final max prime is to just do it at the end by looking at a single global variable and setting it or not based on the highest prime they found during their run. Obviously you'll need to synchronize for that.
Also note it uses a mutex rather than a semaphore because of the use of pthread_cond_wait() If you haven't covered condition variables yet just drop that stuff and load everything before starting your workers.
Also since this is homework, read my code try to understand it then without looking at it again try to write your own.
I would have changed it more but I'm not sure how because it's already basically a really generic producer/consumer example that's missing some bits :P
Another thing to try if you do decide to adopt the same strategy I did and have the loading running in the main thread while the workers work you could add a second condition variable and a counter to limit the number of chunks in the queue and have your workers wake up the main thread if they run out of work.