Basically, I sloppily coded an OpenCL program for an assignment using these global variables:
int devType = CL_DEVICE_TYPE_GPU;
cl_int err; /* Error code returned from api calls. */
size_t global; /* Global domain size for our calculation. */
size_t local; /* Local domain size for our calculation. */
cl_platform_id cpPlatform; /* openCL platform. */
cl_device_id device_id; /* Compute device id. */
cl_context context; /* Compute context. */
cl_command_queue commands; /* Compute command queue. */
cl_program program; /* Compute program. */
cl_kernel kernel; /* Compute kernel. */
/* Create data for the run. */
float *data = NULL; /* Original data set given to device. */
float *results = NULL; /* Results returned from device. */
unsigned int correct; /* Number of correct results returned. */
cl_mem input; /* Device memory used for the input array. */
cl_mem output; /* Device memory used for the output SUM. */
int rc = EXIT_FAILURE;
Now I'm trying to make them all local in order to tidy the program up.
I converted a global variable N by just moving it away from the variables above into the main() function. I then updated every function header that used N to have 'int N' as a parameter, and passed N into any function calls that needed it as an argument. The program worked as expected.
So I suppose what I'm asking is, for the rest of these variables, will it be that simple? I understand the concepts of passing by reference and value and realise some functions may change variables, so I'll need to use pointer referencing/dereferencing. My concern is that my pointer theory is a little rough and I'm worried I'll run into problems. I also am unsure whether my defined functions can take all of these cl variables.
Also, is there anything wrong with using the same variable names within the functions?
EDIT:
As I feared, a problem does occur in the following functions when trying to localise device_id:
void deviceSetup(int devType) {
cl_platform_id cpPlatform; /* openCL platform. */
/* Connect to a compute device. */
if (CL_SUCCESS != clGetPlatformIDs (1, &cpPlatform, NULL))
die ("Error: Failed to find a platform!");
/* Get a device of the appropriate type. */
if (CL_SUCCESS != clGetDeviceIDs (cpPlatform, devType, 1, &device_id, NULL))
die ("Error: Failed to create a device group!");
}
/* Create a compute context. */
void createContext(cl_int err){
context = clCreateContext (0, 1, &device_id, NULL, NULL, &err);
if (!context || err != CL_SUCCESS)
die ("Error: Failed to create a compute context!");
}
/* Create a command commands. */
void createCommandQueue(cl_int err) {
commands = clCreateCommandQueue (context, device_id, 0, &err);
if (!commands || err != CL_SUCCESS)
die ("Error: Failed to create a command commands!");
}
void createAndCompile(cl_int err){
/* Create the compute program from the source buffer. */
program = clCreateProgramWithSource (context, 1,
(const char **) &KernelSource,
NULL, &err);
if (!program || err != CL_SUCCESS)
die ("Error: Failed to create compute program!");
/* Build the program executable. */
err = clBuildProgram (program, 0, NULL, NULL, NULL, NULL);
if (err != CL_SUCCESS)
{
size_t len;
char buffer[2048];
clGetProgramBuildInfo (program, device_id, CL_PROGRAM_BUILD_LOG,
sizeof (buffer), buffer, &len);
die ("Error: Failed to build program executable!\n%s", buffer);
}
}
You've answered your own question really. Yes, that really is all there is to it. You may want to consider combining a large number of related variables into a struct and pass just a pointer to that struct if you find you've generated massive parameter lists for your functions but that's about it. (There is a tiny degree of performance consideration relating to the number of parameters you pass to any function, but I think for now that's an unnecessary level of complication you could do without!)
There's no getting away from understanding pointers in C though (the only way to pass by reference) so a small project like this might well be an ideal time to strengthen that knowledge!
OK, let's have an example, life's always better explained that way.
We have:
int cheddar;
int montereyjack;
int brie;
void print_cheeses(void)
{
printf("I have %d cheddar %d montereyjack and %d brie\n", cheddar, montereyjack, brie);
}
void add_cheeses(void)
{
cheddar = cheddar + 1;
montereyjack = montereyjack + 1;
brie = brie + 1;
print_cheeses();
}
int main(int argc, char *argv[])
{
add_cheeses();
printf ("Now I have %d cheddars %d jacks %d bries\n", cheddar, montereyjack, brie);
}
What we need to get to is:
// By value here because we're not changing anything
void print_cheeses(int cheds, int jacks, int bries)
{
printf("I have %d cheddar %d montereyjack and %d brie\n", cheds, jacks, bries);
}
// Pointers here because we need to change the values in main
void add_cheeses(int *cheese_one, int *cheese_two, int *cheese_three)
{
*cheese_one = *cheese_one + 1; // We're following the pointer to get to the data we want to change
*cheese_two = *cheese_two + 1;
*cheese_three = *cheese_three + 1;
print_cheeses(*cheese_one, *cheese_two, *cheese_three); // We're following the pointer to get to the data we want to print
}
int main(int argc, char *argv[])
{
int cheddar = 0;
int montereyjack = 0;
int brie = 0;
add_cheeses(&cheddar, &montereyjack, &brie);
printf ("Now I have %d cheddars %d jacks %d bries\n", cheddar, montereyjack, brie);
}
But it can be a pain passing all three values each time, and since they're related you could bundle them together in one struct and just pass a pointer to that struct about.
Related
I am working on a client/server program that is supposed to take user input (two integers) and allow the user to calculate those to receive an answer.
When I run my program there seems to be an issue with either connecting to the client or opening the directory and I'm not sure what it could be. I'm entirely new to setting up servers and utilizing directories to read and write .txt files to.
Here are the parts of code that I think might be wrong and causing the issues I am facing, which is the program asks for a port number to connect to (2000 is what is recommended to use) so I enter that and then nothing happens.
// PURPOSE: To run the server by 'accept()'-ing client requests from
// 'listenFd' and doing them.
void doServer (int listenFd)
{
/* Application validity check: */
/* Server clients: */
pthread_t threadId;
pthread_attr_t threadAttr;
int threadCount = 0;
int *iPtr;
/* YOUR CODE HERE: */
while(1)
{
/* Accept connection to client: */
int connfd = accept(listenFd, NULL, NULL); //if I change this to getServerFileDescriptor (another function within the code, it will continuously loop through threadNumbers and do nothing as well).
/* Malloc memory for two integers: */
iPtr = (int *)calloc(2, sizeof(int));
/*Put file descriptor in the first space: */
iPtr[0] = listenFd; // or just listenFd not sure
/* put threadCount into the second space and increment: */
iPtr[1] = threadCount++;
/* Creates detached thread for handleClient and passes the address of iPtr */
pthread_attr_init(&threadAttr);
pthread_attr_setdetachstate(&threadAttr, PTHREAD_CREATE_DETACHED);
pthread_create(&threadId, &threadAttr, handleClient, (void*)iPtr);
pthread_join(threadId, NULL);
pthread_attr_destroy(&threadAttr);
}
}
void* handleClient(void* vPtr)
{
/* Read command: */
char buffer[BUFFER_LEN];
char command;
int fileNum;
char text[BUFFER_LEN];
int shouldContinue = 1;
int threadNum;
int fd;
/* Cast void* vPtr back to an int */
int *iPtr = (int *)vPtr;
/* Assign file descriptor to a local value named 'fd'*/
fd = iPtr[0];
/* Assign thread number to local value named 'threadNum'*/
threadNum = iPtr[1];
free(iPtr);
while (shouldContinue)
{
memset(buffer,'\0',BUFFER_LEN);
memset(text ,'\0',BUFFER_LEN);
read(fd,buffer,BUFFER_LEN);
printf("Thread %d received: %s\n",threadNum,buffer);
sscanf(buffer,"%c %d \"%[^\"]\"",&command,&fileNum,text);
/* YOUR CODE HERE: */
if(command == DIR_CMD_CHAR)
{
/* 1. Open the current directory (named "."). If an error occurs then just send STD_ERROR_MSG back to the client: */
DIR* dirPtr = opendir(".");
struct dirent* entryPtr;
/* If error occurs send STD_ERROR_MSG to client: */
if ((dirPtr = opendir (".")) == NULL) {
{
write(fd, STD_ERROR_MSG, sizeof(STD_ERROR_MSG));
//return(EXIT_FAILURE);
}
/* Read as many entries that will fit into BUFFER_LEN
put as many entries into the buffer and send the buffer to client
d_name=entryPtr into the bufffer using strcat_s,
make sure buffer starts empty
buffer[0]='\n';
add new line char using stringcat "\n"
make sure do not go over buffer lengh */
if (dirPtr)
{
while ((entryPtr = readdir(dirPtr)) != NULL)
{
buffer[0]='\0';
int i;
int sizebuf = sizeof(buffer);
for (i = 0; i < sizebuf; i++)
{
strcat(buffer,entryPtr->d_name);
strcat(buffer,"\n");
}
}
}
/* 3. Close directory */
closedir(dirPtr);
}
Here's how the correct output should look.
$ ./mathClient Machine name [localhost.localdomain]? (I just pressed enter)
Port number? 2000
What would you like to do:
(1) List files
(2) Read a math file
(3) Write a math file
(4) Calculate a math file
(5) Delete a math file
(0) Quit
Your choice? 1
Sending "l"
0.bc
Here are the instructions for the code that I am having trouble with.
Implementing doServer(int listenFd) (10 Points):
doServer() should have a loop in which it waits for a client to connect to listenFd. When a client does, it should:
malloc() enough memory for 2 integers
put the file descriptor from accept() in one of those spaces
put the value of threadCount in the other space, and increment threadCount
Make a detached thread to handle this new client. I called my function handleClient(), but you may call yours whatever. Pass the address of your malloc()-ed array.
The loop should then go back for another accept().
void* handleClient(void* vPtr) (10 Points):
(Or whatever you call your function that runs a thread for the client.)
The thread id and the file descriptor are passed, but they come in as a void* pointer.
Use another pointer to cast back to int*
Save the file descriptor and thread number in local vars
free() the memory
Print the thread number and do a loop like this:
// II.B. Read command:
char buffer[BUFFER_LEN];
char command;
int fileNum;
char text[BUFFER_LEN];
int shouldContinue = 1;
while (shouldContinue)
{
text[0] = '\0';
read(fd,buffer,BUFFER_LEN);
printf("Thread %d received: %s\n",threadNum,buffer);
sscanf(buffer,"%c %d \"%[^\"]\"",&command,&fileNum,text);
// YOUR CODE HERE
}
It read()s a line of text from the client into buffer[], and parses the line into a command character, fileNum integer, and quote-delineated text[] string. (The fileNum and text[] may or may not be given, depending upon the value of command.)
Then do the following operations based upon the value of command. Except for QUIT_CMD_CHAR I strongly recommend using a different function for each!
When the function ends just have it do:
printf("Thread %d quitting.\n",threadNum);
return(NULL);
command == DIR_CMD_CHAR (15 Points):
Open the current directory (named "."). If an error occurs then just send STD_ERROR_MSG back to the client.
Copy as many entries that will fit into a buffer of length BUFFER_LEN. Be sure to put a separating '\n' after each entry.
Close the directory.
Any help would be appreciated, if you need the full code I could send that to you if that would help.
EDIT: Here are two additional functions, one called getPortNum() and another called getServerFileDescriptor() which address receiving a port number and setting up sockets for the connection. Additionally I included the main() which utilizes these.
// PURPOSE: To decide a port number, either from the command line arguments
// 'argc' and 'argv[]', or by asking the user. Returns port number.
int getPortNum (int argc,
char* argv[]
)
{
// I. Application validity check:
// II. Get listening socket:
int portNum;
if (argc >= 2)
portNum = strtol(argv[1],NULL,0);
else
{
char buffer[BUFFER_LEN];
printf("Port number to monopolize? ");
fgets(buffer,BUFFER_LEN,stdin);
portNum = strtol(buffer,NULL,0);
}
// III. Finished:
return(portNum);
}
// PURPOSE: To attempt to create and return a file-descriptor for listening
// to the OS telling this server when a client process has connect()-ed
// to 'port'. Returns that file-descriptor, or 'ERROR_FD' on failure.
int getServerFileDescriptor
(int port
)
{
// I. Application validity check:
// II. Attempt to get socket file descriptor and bind it to 'port':
// II.A. Create a socket
int socketDescriptor = socket(AF_INET, // AF_INET domain
SOCK_STREAM, // Reliable TCP
0);
if (socketDescriptor < 0)
{
perror(THIS_PROGRAM_NAME);
return(ERROR_FD);
}
// II.B. Attempt to bind 'socketDescriptor' to 'port':
// II.B.1. We'll fill in this datastruct
struct sockaddr_in socketInfo;
// II.B.2. Fill socketInfo with 0's
memset(&socketInfo,'\0',sizeof(socketInfo));
// II.B.3. Use TCP/IP:
socketInfo.sin_family = AF_INET;
// II.B.4. Tell port in network endian with htons()
socketInfo.sin_port = htons(port);
// II.B.5. Allow machine to connect to this service
socketInfo.sin_addr.s_addr = INADDR_ANY;
// II.B.6. Try to bind socket with port and other specifications
int status = bind(socketDescriptor, // from socket()
(struct sockaddr*)&socketInfo,
sizeof(socketInfo)
);
if (status < 0)
{
perror(THIS_PROGRAM_NAME);
return(ERROR_FD);
}
// II.B.6. Set OS queue length:
listen(socketDescriptor,5);
// III. Finished:
return(socketDescriptor);
}
int main (int argc,
char* argv[]
)
{
// I. Application validity check:
// II. Do server:
int port = getPortNum(argc,argv);
int listenFd = getServerFileDescriptor(port);
int status = EXIT_FAILURE;
if (listenFd >= 0)
{
doServer(listenFd);
close(listenFd);
status = EXIT_SUCCESS;
}
// III. Finished:
return(status);
I am using multiple threads to access various random files using threads. However, I get an error [Thread 0x7ffff7007700 (LWP 16256) exited]. Also, info threads shows that only 2 threads are created. However, I am trying to create 100 of them. Also, do I have to use the pthread_join() function in my case? The code:
#define NTHREADS 100
void *encrypt(void *args)
{
int count = *((int*) args);
AES_KEY enc_key;
AES_set_encrypt_key(key, 128, &enc_key);
int i;
for(i=1;i<=count;i++){
char *ifile;
char *ofile;
long length;
size_t result;
char * buffer;
sprintf(ifile,"random_files/random_%d.txt",i);
FILE *ifp = fopen(ifile,"rb");
if (ifp==NULL) {fputs ("File error",stderr); exit (1);}
fseek(ifp,0, SEEK_END);
length = ftell(ifp);
fseek (ifp,0, SEEK_SET);
buffer = (char*) malloc (sizeof(char)*length);
if (buffer == NULL) {fputs ("Memory error",stderr); exit (2);}
result = fread (buffer, 1, length, ifp);
if (result != length) {fputs ("Reading error",stderr); exit (3);}
printf("%s",buffer);
fclose (ifp);
free(buffer);
}
}
int main(){
int i,j,count =0;
pthread_t threads[NTHREADS];
for (i=0; i<NTHREADS; i++){
count = count +20;
int *count_ptr = &count;
if(pthread_create(&threads[i], NULL, encrypt, count_ptr)!=0){
fprintf(stderr, "error: Cannot create thread # %d\n", i);
break;
}
}
printf("After Thread\n");
exit(0);
}
Yes, you should join your threads, as none were created detached (and you're probably not deep enough in learning pthreads to deal with that anyway).
That said, you have a significant logic problem in your thread parameters. They're all getting the address of the same count variable in main. Probably the fastest way to change that with no modifications to your thread-proc is simply stand up a side-by-side array of int counts[NTHREADS] matching your threads, using each element as the corresponding thread's data param:
int main()
{
pthread_t threads[NTHREADS];
int counts[NTHREADS]; // HERE
int i,j,count =0;
for (i=0; i<NTHREADS; i++)
{
counts[i] = (count += 20);
if(pthread_create(&threads[i], NULL, encrypt, counts+i)!=0) // LAST PARAM
{
fprintf(stderr, "error: Cannot create thread # %d\n", i);
break;
}
}
for (i=0; i<NTHREADS; ++i)
pthread_join(threads[i], NULL));
return EXIT_SUCCESS;
}
Alternatively, you could do some dynamic allocation hoops, or send the value via intptr_t, cast to void*, but the method shown above has the advantage of requiring no changes on your thread-proc, a target I was aiming for.
I've left any issues in your thread proc to you to solve, but that should get you up and running on your thread stack, at least.
There are missing things in the code, first, in your encrypt function you need to close the threads at the end and also in the main function BUT, before you do so, after your for loop you need to join them all, then they are going to be processed and closed correctly.
The flow will go like this:
(void) pthread_join(th1, NULL);
(void) pthread_join(th2, NULL);
You can put them all in a for to join them by tid.
Another update: There is not really anything "random" going on, I would just start open the files according to its tid, and then add other big implementations, your as is will try to open files like crazy, it should work anyways but, just saying.
I'm currently programming my Bachelor Project what consists an RFB-Client and a Shared Memory. The initialisation of the RFB-Client is done, the Shared Memory is created. My teacher told me to decouple the code and I wrote several functions and used a global variable for the shared memory.
But now a segmentation fault occurs while trying to read the content of the global variable. I debugged the code and found out: The content of the global variabel "my_shm" is always "0x00" :-/
Could you please help me?
These are the portions of code where the problem occurs:
(I know, this is a long code, but sending just parts of it would be useless...)
char *my_shm; --> //global variable
int SHM_init (int shmid, char* shm, key_t key, long int size) {
/* Create a new (System V) shared memory segment of the specified size */
shmid = shmget(key, SHM_SIZE, IPC_CREAT|0777);
/* Check if SHM creation was successful */
if (shmid < 0) {
/* DBG: Debug message to show which point of the program has been passed */
DBG_PRINT("C\n");
/* Check if creation failed because of already existing SHM */
if (EEXIST == errno) {
/* DBG: Debug message to show which point of the program has been passed */
DBG_PRINT("CC\n");
/* Delete already existing SHM with shmctl */
shmctl(shmid, IPC_RMID, NULL);
} else {
/* DBG: Debug message to show which point of the program has been passed */
DBG_PRINT("CCC\n");
}
/* Creation and initialization of SHM failed */
return -1;
}
/* Attach the SHM data pointer to the previously created SHM segment */
shm = shmat(shmid, NULL, 0);
if(shm == (char *) -1) {
/* Attaching failed */
return -1;
}
DBG_PRINT("Shared Memory Initialization successful\n");
/* Creation and initialization of shared memory was successful */
return 0;
}
void RFB_update(rfbClient* client) {
DBG_PRINT("RFB_update called\n");
int i,j;
rfbPixelFormat* pformat=&client->format;
DBG_PRINT("A\n");
/*bytesPerPix: variable which stores Bytes per Pixel*/
int bytesPerPix = pformat->bitsPerPixel/8;
DBG_PRINT("B\n");
/*row= width of frame*bytes per Pixel*/
int row=client->width*bytesPerPix;
DBG_PRINT("C\n");
char byte_to_write;
//as long as j is smaller than 128*(width*bytesPerPix)
for(j=0;j<client->height*row;j+=row) {
//as long as i is smaller than 128 * bytesPerPix
for(i=0;i<client->width*bytesPerPix;i+=bytesPerPix) {
/*frameBuff: Pointer on FrameBuffer*/
unsigned char* frameBuff = client->frameBuffer+j+i;
unsigned int v;
if(bytesPerPix==4)
v=(unsigned int*)frameBuff;
byte_to_write = ((v>>pformat->redShift)*256/(pformat->redMax+1));
SHM_write_byte(my_shm,byte_to_write);
byte_to_write = ((v>>pformat->greenShift)*256/(pformat->greenMax+1));
SHM_write_byte(my_shm,byte_to_write);
byte_to_write = ((v>>pformat->blueShift)*256/(pformat->blueMax+1));
SHM_write_byte(my_shm,byte_to_write);
}
}
DBG_PRINT("RFB_update successful, Shared Memory is filled\n");
}
int SHM_write_byte (char** shm, char byte) {
/*Check if pointer to SHM is valid */
if (shm == (char **) -1) {
/* Pointer is invalid */
return -1;
}
shm = byte;
shm++;
return 0;
}
int main (int argc, char *argv[]) {
if (SHM_init(shmid, my_shm, SHM_KEY, SHM_SIZE) != 0) {
DBG_PRINT("Shared Memory initialized\n");
/* Couldn't initialize SHM,initializing failed */
return -1;
}
/* Initialize RFB Client */
if (RFB_client_init(rfb_client, (FinishedFrameBufferUpdateProc)RFB_update) != 0) {
DBG_PRINT("Couldn't initialize client\n");
/* Couldn't initialize Client,initializing failed */
return -1;
}
--> everywhere the variable "my_shm" is used: the content is: 0x00...
This seems to be a very common problem here on stackoverflow.com today, and the problem is that you pass arguments to function by value and not by reference.
What that means is that when you pass an argument to a function, its value is copied, and the function only work on the copy locally inside the function. As you should know, modifying a copy will of course not modify the original.
C does not have pass by reference, but it can be emulated by using pointers. In your case, since you have a pointer you need to pass a pointer to the pointer using the address-of operator, like
SHM_init(shmid, &my_shm, SHM_KEY, SHM_SIZE)
// ^
// |
// Note ampersand (address-of operator) here
You of course need to modify the function to actually accept a pointer to the pointer:
int SHM_init (int shmid, char** shm, key_t key, long int size)
And of course use the dereference operator * when using the variable:
*shm = shmat(shmid, NULL, 0);
I wanted to write a program that test if two files are duplicates (have exactly the same content). First I test if the files have the same sizes, and if they have i start to compare their contents.
My first idea, was to "split" the files into fixed size blocks, then start a thread for every block, fseek to startup character of every block and continue the comparisons in parallel. When a comparison from a thread fails, the other working threads are canceled, and the program exits out of the thread spawning loop.
The code looks like this:
dupf.h
#ifndef __NM__DUPF__H__
#define __NM__DUPF__H__
#define NUM_THREADS 15
#define BLOCK_SIZE 8192
/* Thread argument structure */
struct thread_arg_s {
const char *name_f1; /* First file name */
const char *name_f2; /* Second file name */
int cursor; /* Where to seek in the file */
};
typedef struct thread_arg_s thread_arg;
/**
* 'arg' is of type thread_arg.
* Checks if the specified file blocks are
* duplicates.
*/
void *check_block_dup(void *arg);
/**
* Checks if two files are duplicates
*/
int check_dup(const char *name_f1, const char *name_f2);
/**
* Returns a valid pointer to a file.
* If the file (given by the path/name 'fname') cannot be opened
* in 'mode', the program is interrupted an error message is shown.
**/
FILE *safe_fopen(const char *name, const char *mode);
#endif
dupf.c
#include <errno.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include "dupf.h"
FILE *safe_fopen(const char *fname, const char *mode)
{
FILE *f = NULL;
f = fopen(fname, mode);
if (f == NULL) {
char emsg[255];
sprintf(emsg, "FOPEN() %s\t", fname);
perror(emsg);
exit(-1);
}
return (f);
}
void *check_block_dup(void *arg)
{
const char *name_f1 = NULL, *name_f2 = NULL; /* File names */
FILE *f1 = NULL, *f2 = NULL; /* Streams */
int cursor = 0; /* Reading cursor */
char buff_f1[BLOCK_SIZE], buff_f2[BLOCK_SIZE]; /* Character buffers */
int rchars_1, rchars_2; /* Readed characters */
/* Initializing variables from 'arg' */
name_f1 = ((thread_arg*)arg)->name_f1;
name_f2 = ((thread_arg*)arg)->name_f2;
cursor = ((thread_arg*)arg)->cursor;
/* Opening files */
f1 = safe_fopen(name_f1, "r");
f2 = safe_fopen(name_f2, "r");
/* Setup cursor in files */
fseek(f1, cursor, SEEK_SET);
fseek(f2, cursor, SEEK_SET);
/* Initialize buffers */
rchars_1 = fread(buff_f1, 1, BLOCK_SIZE, f1);
rchars_2 = fread(buff_f2, 1, BLOCK_SIZE, f2);
if (rchars_1 != rchars_2) {
/* fread failed to read the same portion.
* program cannot continue */
perror("ERROR WHEN READING BLOCK");
exit(-1);
}
while (rchars_1-->0) {
if (buff_f1[rchars_1] != buff_f2[rchars_1]) {
/* Different characters */
fclose(f1);
fclose(f2);
pthread_exit("notdup");
}
}
/* Close streams */
fclose(f1);
fclose(f2);
pthread_exit("dup");
}
int check_dup(const char *name_f1, const char *name_f2)
{
int num_blocks = 0; /* Number of 'blocks' to check */
int num_tsp = 0; /* Number of threads spawns */
int tsp_iter = 0; /* Iterator for threads spawns */
pthread_t *tsp_threads = NULL;
thread_arg *tsp_threads_args = NULL;
int tsp_threads_iter = 0;
int thread_c_res = 0; /* Thread creation result */
int thread_j_res = 0; /* Thread join res */
int loop_res = 0; /* Function result */
int cursor;
struct stat buf_f1;
struct stat buf_f2;
if (name_f1 == NULL || name_f2 == NULL) {
/* Invalid input parameters */
perror("INVALID FNAMES\t");
return (-1);
}
if (stat(name_f1, &buf_f1) != 0 || stat(name_f2, &buf_f2) != 0) {
/* Stat fails */
char emsg[255];
sprintf(emsg, "STAT() ERROR: %s %s\t", name_f1, name_f2);
perror(emsg);
return (-1);
}
if (buf_f1.st_size != buf_f2.st_size) {
/* File have different sizes */
return (1);
}
/* Files have the same size, function exec. is continued */
num_blocks = (buf_f1.st_size / BLOCK_SIZE) + 1;
num_tsp = (num_blocks / NUM_THREADS) + 1;
cursor = 0;
for (tsp_iter = 0; tsp_iter < num_tsp; tsp_iter++) {
loop_res = 0;
/* Create threads array for this spawn */
tsp_threads = malloc(NUM_THREADS * sizeof(*tsp_threads));
if (tsp_threads == NULL) {
perror("TSP_THREADS ALLOC FAILURE\t");
return (-1);
}
/* Create arguments for every thread in the current spawn */
tsp_threads_args = malloc(NUM_THREADS * sizeof(*tsp_threads_args));
if (tsp_threads_args == NULL) {
perror("TSP THREADS ARGS ALLOCA FAILURE\t");
return (-1);
}
/* Initialize arguments and create threads */
for (tsp_threads_iter = 0; tsp_threads_iter < NUM_THREADS;
tsp_threads_iter++) {
if (cursor >= buf_f1.st_size) {
break;
}
tsp_threads_args[tsp_threads_iter].name_f1 = name_f1;
tsp_threads_args[tsp_threads_iter].name_f2 = name_f2;
tsp_threads_args[tsp_threads_iter].cursor = cursor;
thread_c_res = pthread_create(
&tsp_threads[tsp_threads_iter],
NULL,
check_block_dup,
(void*)&tsp_threads_args[tsp_threads_iter]);
if (thread_c_res != 0) {
perror("THREAD CREATION FAILURE");
return (-1);
}
cursor+=BLOCK_SIZE;
}
/* Join last threads and get their status */
while (tsp_threads_iter-->0) {
void *thread_res = NULL;
thread_j_res = pthread_join(tsp_threads[tsp_threads_iter],
&thread_res);
if (thread_j_res != 0) {
perror("THREAD JOIN FAILURE");
return (-1);
}
if (strcmp((char*)thread_res, "notdup")==0) {
loop_res++;
/* Closing other threads and exiting by condition
* from loop. */
while (tsp_threads_iter-->0) {
pthread_cancel(tsp_threads[tsp_threads_iter]);
}
}
}
free(tsp_threads);
free(tsp_threads_args);
if (loop_res > 0) {
break;
}
}
return (loop_res > 0) ? 1 : 0;
}
The function works fine (at least for what I've tested). Still, some guys from #C (freenode) suggested that the solution is overly complicated, and it may perform poorly because of parallel reading on hddisk.
What I want to know:
Is the threaded approach flawed by default ?
Is fseek() so slow ?
Is there a way to somehow map the files to memory and then compare them ?
LATED EDIT:
Today I had some time, and I've followed your advices. You were right, this threaded version actually performs worse than a single threaded version, and all because of the parallel readings on hard disk.
Another thing is that I've written a function that uses mmap(), and until now is the optimal one. Still the biggest drawback of that function is that it fails, when the files are getting really big.
Here is the new implementation (a very brute and direct code):
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include "dupf.h"
/**
* Safely assures that a file is opened.
* If cannot open file, the flow of the program is interrupted.
* The error code returned is -1.
**/
FILE *safe_fopen(const char *fname, const char *mode)
{
FILE *f = NULL;
f = fopen(fname, mode);
if (f == NULL) {
char emsg[1024];
sprintf(emsg, "Cannot open file: %s\t", fname);
perror(emsg);
exit(-1);
}
return (f);
}
/**
* Check if two files have the same size.
* Returns:
* -1 Error.
* 0 If they have the same size.
* 1 If the don't have the same size.
**/
int check_same_size(const char *f1_name, const char *f2_name, off_t *f1_size, off_t *f2_size)
{
struct stat f1_stat, f2_stat;
if((f1_name == NULL) || (f2_name == NULL)){
fprintf(stderr, "Invalid filename passed to function [check_same_size].\n");
return (-1);
}
if((stat(f1_name, &f1_stat) != 0) || (stat(f2_name, &f2_stat) !=0)){
fprintf(stderr, "Cannot apply stat. [check_same_size].\n");
return (-1);
}
if(f1_size != NULL){
*f1_size = f1_stat.st_size;
}
if(f2_size != NULL){
*f2_size = f2_stat.st_size;
}
return (f1_stat.st_size == f2_stat.st_size) ? 0 : 1;
}
/**
* Test if two files are duplicates.
* Returns:
* -1 Error.
* 0 If they are duplicates.
* 1 If they are not duplicates.
**/
int check_dup_plain(char *f1_name, char *f2_name, int block_size)
{
if ((f1_name == NULL) || (f2_name == NULL)){
fprintf(stderr, "Invalid filename passed to function [check_dup_plain].\n");
return (-1);
}
FILE *f1 = NULL, *f2 = NULL;
char f1_buff[block_size], f2_buff[block_size];
size_t rch1, rch2;
if(check_same_size(f1_name, f2_name, NULL, NULL) == 1){
return (1);
}
f1 = safe_fopen(f1_name, "r");
f2 = safe_fopen(f2_name, "r");
while(!feof(f1) && !feof(f2)){
rch1 = fread(f1_buff, 1, block_size, f1);
rch2 = fread(f2_buff, 1, block_size, f2);
if(rch1 != rch2){
fprintf(stderr, "Invalid reading from file. Cannot continue. [check_dup_plain].\n");
return (-1);
}
while(rch1-->0){
if(f1_buff[rch1] != f2_buff[rch1]){
return (1);
}
}
}
fclose(f1);
fclose(f2);
return (0);
}
/**
* Test if two files are duplicates.
* Returns:
* -1 Error.
* 0 If they are duplicates.
* 1 If they are not duplicates.
**/
int check_dup_memmap(char *f1_name, char *f2_name)
{
struct stat f1_stat, f2_stat;
char *f1_array = NULL, *f2_array = NULL;
off_t f1_size, f2_size;
int f1_des, f2_des, cont, res;
if((f1_name == NULL) || (f2_name == NULL)){
fprintf(stderr, "Invalid filename passed to function [check_dup_memmap].\n");
return (-1);
}
if(check_same_size(f1_name, f2_name, &f1_size, &f2_size) == 1){
return (1);
}
f1_des = open(f1_name, O_RDONLY);
f2_des = open(f2_name, O_RDONLY);
if((f1_des == -1) || (f2_des == -1)){
perror("Cannot open file");
exit(-1);
}
f1_array = mmap(0, f1_size * sizeof(*f1_array), PROT_READ, MAP_SHARED, f1_des, 0);
if(f1_array == NULL){
fprintf(stderr, "Cannot map file to memory [check_dup_memmap].\n");
return (-1);
}
f2_array = mmap(0, f2_size * sizeof(*f2_array), PROT_READ, MAP_SHARED, f2_des, 0);
if(f2_array == NULL){
fprintf(stderr, "Cannot map file to memory [check_dup_memmap].\n");
return (-1);
}
cont = f1_size;
res = 0;
while(cont-->0){
if(f1_array[cont]!=f2_array[cont]){
res = 1;
break;
}
}
munmap((void*) f1_array, f1_size * sizeof(*f1_array));
munmap((void*) f2_array, f2_size * sizeof(*f2_array));
return res;
}
int main(int argc, char *argv[])
{
printf("result: %d\n",check_dup_memmap("f2","f1"));
return (0);
}
I am planning now to extend this code, by re-adding the threaded functionality, but this time the reading will be on memory.
Thanks for your answers.
The limiting factor will be disk reads, which (assuming that both files are on the same disk) will be serialized anyway, so I don't think threading will help much at all.
You could probably simplify your code greatly by using hashes, instead of doing a byte-by-byte comparison. Assuming you're not doing anything important, like deleting, an md5 or similar hash function should be plenty. Boost provides quite a few, and they're usually pretty fast.
if fileA.size == fileB.size
if fileA.hash() == fileB.hash()
flag(fileA, fileB, same);
I wouldn't delete files after that comparison, but it's plenty safe to move them to a temporary directory for further review or just build a list of possible duplicates.
It's hard to guess about performance without a real system to test against (for example if you're using a solid state drive, there's no head seek time and the cost of reading different sectors from different threads is almost zero).
If this is running against a reasonably standard computer with regular (spinning platter) hard drives, having multiple threads contend for the part of the disk they want to read from will possibly slow things down (depending, again, on the hardware and also the size of the chunks).
If the time it takes to compute the "sameness" of a chunk is fast compared to the time it takes to read that chunk from disk, having a separate thread will not help much since the second (or third...) thread would spend most of it's time waiting for IO to complete anyway.
Another factor is the cache size of the CPU. If all of the memory you're processing at one time fits in the CPU cache, things will be much faster than if different threads cause different chunks of memory to be loaded into cache as they execute instructions.
If you have more threads than you have CPU cores, you will just slow things down by making unnecessary context switches (since a thread needs a core to run on).
After reading all of that, if you still think multithreading is going to help for your target system, consider one thread that does IO only, places the data in a queue, and has two or more worker threads taking data off of the queue to process. That way, you optimize disk IO and can take advantage of multiple cores to crunch the numbers.
Steve suggested you can memory map you files on Unix. That will speed up access to the underlying data a bit by leveraging low level OS functionality (the same kind used to manage swap files). That will give you some performance improvement as the OS will handle loading the parts of the file you are working on into memory efficiently, as long as the file fits into available address space. FYI you can do the same thing on Windows.
Before even considering the performance effects of parallel disk reads and thread overhead and such...
Is there any reason to believe that scanning the files in chunks will find the differences any quicker than straight through? Is the data contained in the files predominantly in a certain format, and if so, is the splitting scheme tailored to it? If not, I don't see how scanning the files by skipping over every n bytes (which is all the multithreaded splitting is effectively doing) could offer any improvement over reading the bytes in the order they are on disk.
Think of the two limiting cases -- "splitting" the file into one block, and splitting the file into as many one-byte "blocks" as there are bytes in the file. Will either of those cases be more efficient than the other, or some in-between value? If there is no in-between value that you know you should optimize to, then you know nothing about how the data is stored in the files, so it should make no difference how you scan them.
Even if you set the split to optimize to the disk's performance like block size, you're still going to have to go back to read the next byte, which will likely be at an extremely non-optimal position. And in the end you're going to have to read every single byte in the file, no matter how you split it.
Because you're using pthreads, I assume you're working in a Unix environment -- in which case you could mmap(2) both files into memory and compare the memory arrays directly.
Well, there is the standard memory mapping mmap() function that maps a file to memory. You should be able to do something like
int fd1;
int fd2;
int size1;
int size2;
fd1 = open(name1, O_RDONLY);
size1 = lseek(fd1, 0, SEEK_END);
fd2 = open(name2, O_RDONLY);
size2 = lseek(fd2, 0, SEEK_END);
if ( size1 == size2 )
{
char * data1 = mmap(0, size1, PROT_READ, MAP_SHARED, fd1, 0);
char * data2 = mmap(0, size1, PROT_READ, MAP_SHARED, fd2, 0);
int i;
/* ...and this is, obviously, where you'd do something more clever */
for ( i = 0; i < size1 && *data1 == *data2; i++, data1++, data2++ );
if ( i == size1 )
printf("Equal\n");
}
close(fd1);
close(fd2);
Other than that, yes, your solution looks overly complicated ;-) The threaded approach is not necessarily flawed, but you might not see that parallel access improves performance. For SAN drives or ramdisks it might improve performance, for normal spinning platter drives it might impede it. But simpler is usually better, unless you really have a performance issue.
Regarding fseek() vs other methods, it depends on the operating system you use. Google is you friend here, you can easily find articles at least for Solaris and Linux.
Even if disk access was not the limiting factor (it will be), unless you have a multi-core processor that could hand off different threads to different cores, you would not see a speed-up from going multi-threaded. Basically, you have to compare all N bytes of the file one way or another, and even if you use threads, if they execute in the same core, it will take the same amount of time as without using threads.
There are some environments that could spread the workload across cores, but even so, the CPU will be able to process so much faster than the data can be pulled in from disk that the disk I/O system will be the limiting factor.
I have been trying to debug my code whenever I had free-time for the past day and a half and I don't know what is wrong with my code. When I add the close() function to a recursive call, the program gives me an invalid pointer. But when I remove the close() function call the program runs fine, except it does not do what it is supposed to do, which is:
add up all the file sizes in a user
input directory
open sub-directories, if any, and add
up all the files inside the
sub-directory
Instead, it adds up all the file sizes in the input directory and is able to open the last sub-directory and add the files within that directory to the total file size count.
I am trying to do this with threads. The main() function creates one main thread from the user input directory and runs opendirectory() off the bat.
/*
* Iterates through given directory
*/
void *opendirectory(void *t)
{
pthread_mutex_lock(&dirlock);
DIR *dpntr;
struct dirent *dentry;
char new_directory[512], dir = t;
printf("OPENING DIRECTORY ... %s\n", t);
/* Checks if given directory can be opened */
if((dpntr = opendir(t)) == NULL) {
printf("DIRECTORY FAILED ...%s\n",t);
perror("ERROR -- COULD NOT OPEN DIR");
pthread_exit(NULL);
}
printf("DIRECTORY OPENED: %s\n", t);
/* Read each file in current directory */
while ((dentry = readdir(dpntr)) != NULL ) {
/* Ignore special directories */
if(strcmp(dentry -> d_name, ".") == 0 || strcmp(dentry -> d_name, "..") == 0) {
continue;
} else {
compilelist( t, dentry->d_name );
}
}
pthread_mutex_unlock(&dirlock);
/* Checks if directory can be closed */
if(closedir(dpntr) < 0)
printf("ERROR CLOSING %s.\n", t);
}
This is the function that will determine if a new thread should be created and is supposed to run recursively.
/*
* Determines if current file is a directory
* Creates a new thread if true
*/
void compilelist (const char* dirname, const char *filename)
{
pthread_mutex_lock(&filelock);
struct stat statdata;
char *filepathname, *dpntr;
/* Allocate memory for filepathname */
if((filepathname = (char *) malloc(sizeof(char) * strlen(dirname))) == NULL)
{
printf("CANNOT ALLOCATE MEMORY FOR FILE PATH NAME.");
pthread_exit(NULL);
}
/* Concats directory name with file name */
if(dirname[strlen(dirname) -1] == '/')
{
pthread_mutex_lock(&pathlock);
sprintf(filepathname, "%s%s", dirname, filename);
pthread_mutex_unlock(&pathlock);
}else
{
pthread_mutex_lock(&pathlock);
sprintf(filepathname, "%s/%s", dirname, filename);
pthread_mutex_unlock(&pathlock);
}
lstat(filepathname, &statdata);
/* Calls print_statdata() if current item is a file */
if(!(S_ISDIR(statdata.st_mode)))
{
printf("FILE: %s\n", filepathname);
if(!stat( filepathname, &statdata))
{
print_statdata( filename, &statdata );
}
else {
fprintf (stderr, "GETTING STAT FOR %s", filepathname);
perror( "ERROR IN STATDATA WHILE GETTING STAT");
}
}
/* Recursive call to opendirectory() */
else {
pthread_mutex_lock(&dircountlock);
dirCount++;
pthread_mutex_unlock(&dircountlock);
dpntr = filepathname;
free(filepathname);
printf("SUB-DIRECTORY THREAD: %s\nTHREAD ID NUMBER: %d\n", dpntr, dirCount);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
pthread_create(&threads[dirCount-1], &attr, opendirectory, (void *)dpntr);
}
pthread_mutex_unlock(&filelock);
}
Here is the main()
/*
* Main function prompts user for a directory
*/
int main(int argc, char *argv[])
{
int i;
char *dPtr;
// pthread_attr_t attr;
printf("ENTER A DIRECTORY:\n\t");
scanf("%s", directory);
dPtr = directory;
/* Initialize mutex and condition variable objects */
pthread_mutex_init(&mutex, NULL);
pthread_mutex_init(&filelock, NULL);
pthread_mutex_init(&dirlock, NULL);
pthread_mutex_init(&dircountlock, NULL);
pthread_cond_init (&count_threshold_cv, NULL);
/* For portability, explicitly create threads in a joinable state */
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
pthread_create(&threads[0], &attr, opendirectory, (void *)dPtr);
/* Wait for all threads to complete */
for (i = 0; i < dirCount; i++) {
pthread_join(threads[i], NULL);
}
printf("TOTAL DIRECTORY SIZE: %d\n", dirSize);
/* Clean up and exit */
pthread_attr_destroy(&attr);
pthread_mutex_destroy(&mutex);
pthread_mutex_destroy(&filelock);
pthread_mutex_destroy(&dirlock);
pthread_mutex_destroy(&dircountlock);
pthread_cond_destroy(&count_threshold_cv);
pthread_exit (NULL);
}
And the global variables ...
pthread_mutex_t mutex;
pthread_mutex_t dirlock;
pthread_mutex_t filelock;
pthread_mutex_t dircountlock;
pthread_mutex_t threadlock;
pthread_cond_t count_threshold_cv;
pthread_attr_t attr;
pthread_t threads[128]; // handles up to 128 threads (i.e. 128 directories, change accordingly)
char directory[512];
int dirSize = 0;
int dirCount = 1; // user's input directory
I feel that the pthread_create() called at the bottom of the compilelist() function is not working properly. The threads[] refers to a global array of threads that has a default size of 20, assuming that there will be no more than 20 total directories. dirCount starts off at 1 because of the user's input directory and increases as new directories are encountered.
Your code:
dpntr = opendir(t)
...
if(closedir(t) < 0)
should be:
if(closedir(dpntr) < 0)
Here I found 2 problems of your code:
As wrang-wrang metioned, closedir(t) leads segfault.
"char filepathname[512];" of compilelist() is a local memory buffer, but you pass it to your thread (opendirectory) and use it continuously. You should use copying or dynamic-allocation instead.
Effo Upd#2009nov17:
After fixing above 2 points, it works fine on my FC9 x86_64 so far. Btw: threads number 20 is really not enough.
First problem:
whenever I had free-time for the past day and a half
Don't do that, your brain isn't built for it. Allocate a time, tell your workmates/wife-and-kids that, if they bother you during this time, there will be gunshots and police involvement :-)
Other problems: no idea (hence the community wiki).