I have the following code:
for(i = 0 ; i < max_thread; i++)
{
struct arg_struct args;
args.arg1 = file;
args.arg2 = word;
args.arg3 = repl;
if(pthread_create(&thread_id[i],NULL,&do_process,&args) != 0)
{
i--;
fprintf(stderr,RED "\nError in creating thread\n" NONE);
}
}
for(i = 0 ; i < max_thread; i++)
if(pthread_join(thread_id[i],NULL) != 0)
{
fprintf(stderr,RED "\nError in joining thread\n" NONE);
}
int do_process(void *arguments)
{
//code missing
}
*How can I transform (void *)do_process into (int) do_process ?*
That function returns very important info and without those returns I don't know how to read the replies
I get the following error: warning: passing arg 3 of `pthread_create' makes pointer from integer without a cast
The thread function returns a pointer. At minimum, you can allocate an integer dynamically and return it.
void * do_process (void *arg) {
/* ... */
int *result = malloc(sizeof(int));
*result = the_result_code;
return result;
}
Then, you can recover this pointer from the thread_join() call;
void *join_result;
if(pthread_join(thread_id[i],&join_result) != 0)
{
fprintf(stderr,RED "\nError in joining thread\n" NONE);
} else {
int result = *(int *)join_result;
free(join_result);
/* ... */
}
Just write a helper function that is of the correct type, but all it does is take the void * input parameter, get all the right parameters out of it, call your function, take the return of that, and package it up as a void * for pthread_join to get.
To your specific question, you can't/shouldn't. Just do what I outlined above and you'll be golden.
The pthread_join() is a simple way to communicate between the two threads. It has two limitations. First, it can pass only one value from the pointer (you can make it a pointer and store multiple values). Second, you can return it only when the thread is all done -- after returning this value, the thread goes in terminated state. So, if you want the threads to communicate in a more granular fashion, you will be better served in using a common shared data. Of course, at teh very least, you would to use Pthread mutex to synchronize access to the common data. And, if you want the threads to communicate with each other, then you would also need to use Pthread condvars.
Related
I want to use condition variables to launch at most N thread to process all files one one huge directory (1M files).
The code seems to work but after some times, it blocks in main thread. Below the frustrating code:
void* run(void* ctx)
{
clientCtx* client = (clientCtx*)ctx;
printf("New file from thread %d: %s\n", client->num, client->filename);
free(client->filename);
pthread_mutex_lock(&clientFreeMutex);
client->state = IDLE_STATE;
pthread_cond_signal(&clientFreeCond);
printf("Thread %d is free\n", client->num);
pthread_mutex_unlock(&clientFreeMutex);
return NULL;
}
int main(int argc, char** argv)
{
pthread_t client[MAX_CLIENT] = {0};
clientCtx ctx[MAX_CLIENT] = {0};
DIR* directory = NULL;
struct dirent* element = NULL;
/* Initialize condition variable for max clients */
pthread_mutex_init(&clientFreeMutex, NULL);
pthread_cond_init(&clientFreeCond, NULL);
/* Initialize contexts for clients */
for (int cnt = 0; cnt < MAX_CLIENT; cnt ++)
{
ctx[cnt].state = IDLE_STATE;
ctx[cnt].num = cnt;
}
directory = opendir(argv[1]);
while((element = readdir(directory)) != NULL)
{
pthread_mutex_lock(&clientFreeMutex);
int cnt;
for (cnt = 0; cnt < MAX_CLIENT; cnt++)
{
if(ctx[cnt].state == IDLE_STATE)
{
ctx[cnt].filename = strdup(element->d_name);
ctx[cnt].state = BUSY_STATE;
pthread_create(&client[cnt], NULL, run, &(ctx[cnt]));
break;
}
}
/* No free client */
if (cnt == MAX_CLIENT)
{
printf("No free thread. Waiting.\n");
pthread_cond_wait(&clientFreeCond, &clientFreeMutex);
}
pthread_mutex_unlock(&clientFreeMutex);
}
closedir(directory);
exit(EXIT_SUCCESS);
}
What is the problem? thanks for your help :)
Warning you use the value of readdir in separate threads without any protection against the multi-threading, so when you (try to) printf client->file->d_name may be you are doing at the same time readdir in the main thread modifying the saved result, this has an undefined behavior.
You need for example to save a strdup of element->file->d_name in main and save that string in the clientCtx rather than the struct dirent *, and of course to free it in run
Note also a closedir is missing at the end of main even in this case it is not a real problem (just do to remember for your other programs).
I finally found the problem: launched threads were not joined and pthread_create finally returned an error code with errno message set to "Could not allocate memory". The signal was never sent and the main thread was then blocking.
I fixed this creating a new state for already launched threads and adding a join in main loop.
I am working on an assignment which requires me to use threads to process and synchronize fetching data from a file. My professor told me that I can change my data to a void pointer to pass it to my function and then cast it back. I am trying to do this with file IO.
pthread_create(&th1, NULL, processing, (void *)&fp);
In my processing function I am trying to cast it back to a FILE pointer with this:
FILE driveOne = (FILE *)file;
This clearly doesn't work, so can someone explain this to me?
Here's a more complete example.
Let's say your worker function needs a file handle. For simplicity, let's say it reads each char from it, and returns the number of chars read, cast to a pointer:
void *worker(void *data)
{
FILE *handle = (FILE *)data;
uintptr_t count = 0;
if (handle && !ferror(handle)) {
/* handle is a valid file handle */
while (getc(handle) != EOF)
count++;
}
return (void *)count;
}
If count were of some other type than intptr_t or uintptr_t (declared in <stdint.h>, which is typically included by including <inttypes.h>), you'd need to cast it first to that type, and then to void pointer, i.e. (void *)(uintptr_t)count.
Because such worker threads don't need much stack (almost none, to be precise), and default thread stack sizes are huge (megabytes), we can save some memory (and allow much more threads if needed, especially on 32-bit architectures) by creating a pthread attribute that instructs pthread_create() to use a smaller stack. This attribute is not "consumed" by the call; it is more like an configuration block.
Let's say you have three streams, FILE *in[3];, and you wish to use three threads to check their lengths. Using a pthread attribute to use a smaller stack (2*PTHREAD_STACK_MIN, as defined in <limits.h>, is a good, safe value for worker threads that don't use alloca() or local arrays.):
pthread_t worker_id[3];
uintptr_t length[3];
pthread_attr_t attrs;
void *retptr;
int i, result;
/* Create a pthread attribute set, defining smaller stack size. */
pthread_attr_init(&attrs);
pthread_attr_setstacksize(&attrs, 2*PTHREAD_STACK_MIN);
/* Create the three worker threads. */
for (i = 0; i < 3; i++) {
result = pthread_create(&(worker_id[i]), &attrs, worker, (void *)in[i]);
if (result) {
fprintf(stderr, "Cannot create thread: %s.\n", strerror(result));
exit(EXIT_FAILURE);
}
}
/* pthread attributes are no longer needed. */
pthread_attr_destroy(&attrs);
/*
... This thread can do something else here ...
*/
/* Reap the threads, and collect their return values. */
for (i = 0; i < 3; i++) {
result = pthread_join(worker_id[i], &retptr);
if (result) {
fprintf(stderr, "Cannot reap thread: %s.\n", strerror(result));
exit(EXIT_FAILURE);
}
length[i] = (uintptr_t)retptr;
}
for (i = 0; i < 3; i++)
printf("in[%d] contained %llu chars.\n", i, (unsigned long long)length[i]);
The same pattern can be used when you want to pass multiple parameters to the thread function. You first construct a structure to hold those parameters, and create them. You can allocate them dynamically, declare them as global variables, or declare them as local variables in main() -- any scope that exists for the full duration when the worker thread exists, works.
For example, let's say your worker function calculates a histogram of each unsigned char value it reads from the stream:
struct work {
pthread_t id; /* Thread identifier */
FILE *in; /* File handle to read from */
size_t count[UCHAR_MAX + 1]; /* Histogram */
};
void *worker(void *data) {
struct work *const work = (struct worker_data *)data;
int c;
if (!work || !work->in) {
/* Invalid data, or invalid file handle. */
return (void *)(intptr_t)(EINVAL);
}
if (ferror(work->in)) {
/* Stream is in error state. */
return (void *)(intptr_t)(EIO);
}
/* Read the stream. */
while ((c = getc(work->in)) != EOF) {
/* Update histogram. */
work->count[(unsigned char)c]++;
}
/* Did the reading stop due to an I/O error? */
if (ferror(work->in))
return (void *)(intptr_t)(EIO);
/* No errors, all done. */
return (void *)0;
}
Note that struct work *const work = ... initializes a constant pointer work, not a pointer to constant. The const there is just an optimization that tells the C compiler that we won't try to modify work pointer itself. The data it points to, is modifiable.
(To read pointer declarations, read them from right to left, replacing each * with "is a pointer to", to get the proper sense of it.)
The code to create these workers is very similar, except that we allocate the work dynamically:
struct work *work[3];
pthread_attr_t attrs;
void *retptr;
int i, result;
/* Create and initialize the three pointers. */
for (i = 0; i < 3; i++) {
/* Allocate a work structure. */
work[i] = malloc(sizeof *(work[i]));
if (!work[i]) {
fprintf(stderr, "Out of memory.\n");
exit(EXIT_FAILURE);
}
/* Copy the handle to read from, */
work[i]->in = in[i];
/* and clear the histogram part. */
memset(work[i]->count, 0, sizeof work[i]->count);
}
/* Create a pthread attribute set, defining smaller stack size. */
pthread_attr_init(&attrs);
pthread_attr_setstacksize(&attrs, 2*PTHREAD_STACK_MIN);
/* Create the three worker threads. */
for (i = 0; i < 3; i++) {
result = pthread_create(&(work[i]->id), &attrs, worker, (void *)work[i]);
if (result) {
fprintf(stderr, "Cannot create thread: %s.\n", strerror(result));
exit(EXIT_FAILURE);
}
}
/* pthread attributes are no longer needed. */
pthread_attr_destroy(&attrs);
/*
... This thread can do something else here ...
*/
/* Reap the threads, and collect their return values. */
for (i = 0; i < 3; i++) {
result = pthread_join(work[i]->id, &retptr);
if (result) {
fprintf(stderr, "Cannot reap thread: %s.\n", strerror(result));
exit(EXIT_FAILURE);
}
/* If the thread reported a failure, print the corresponding
error message (but do not exit). */
if (retptr)
fprintf(stderr, "Thread %d of 3: %s.\n", i+1, strerror((intptr_t)retptr));
/* ... print the histogram here? ... */
}
/* Free the work structures. */
for (i = 0; i < 3; i++)
free(work[i]);
If you don't want to abort the program when an error occurs, it is useful to note that free(NULL) is safe and does nothing; and that struct work *pointerarray[SIZE] = {0}; declares an array of SIZE pointers to struct work, and initializes them all to zero. For example, if an allocation or thread creation fails at some point, you can just free() each pointer, whether or not its allocation was successful.
That is, if you want to allocate three different types of structures (struct atype *a;, struct btype *b;, and struct ctype *c;), you can do
a = malloc(sizeof *a);
b = malloc(sizeof *b);
c = malloc(sizeof *c);
if (!a || !b || !c) {
free(c);
free(b);
free(a);
return ALLOCATION_FAILED;
}
/* Allocation was successful */
instead of allocating each one and testing for failure separately.
You need to declare driveOne to be FILE *, not FILE.
FILE *driveOne = (FILE *)file;
In addition, assuming that fp was initially declared as FILE *, your call to pthread_create should not have & before fp, like so:
pthread_create(&th1, NULL, processing, (void *)fp);
I am writing a unit test that involves running multiple threads and I ran into a memory access issue that I can't seem to understand.
Here is the original (pseudo) code:
void thread_func(void * err) {
/*foo will return an allocated error_type if error occurs else NULL*/
err = (void *) foo(...)
pthread_exit(NULL);
}
void unit_test() {
int i = 0;
err_type *err_array[10];
pthread_t threads[10];
for (i = 0; i < 10; i++) {
pthread_create(&(threads[i]), NULL, thread_func, (void *) err_array[i]);
}
for(i = 0; i < 10; i++) {
pthread_join(threads[i], NULL);
ASSERT_NULL(err_array[i]);
}
}
What I am confused about is that all the threads will return NULL (checked with a gdb), however err_array[1] and err_array[5] will be NOT NULL. And instead of a valid err_type, they will contain garbage. In the case of err_array[1] will contain a string of the unit_test file path, and err_array[5] will contain a bunch of access out of bounds addresses.
A work-around I've found is to use a global err_array, and pass in the index for each element to the thread. As well as initializing all the elements of the array to NULL.
My question is why does the above 2 methods work, and not the original code?
The err variable is local to thread_func. It goes out of scope when thread_func returns. You need to pass the thread a pointer to the thing you want it to modify, not the current value of the thing you want it to modify.
So:
void thread_func(void ** err) {
/*foo will return an allocated error_type if error occurs else NULL*/
*err = (void *) foo(...)
pthread_exit(NULL);
}
And:
pthread_create(&(threads[i]), NULL, thread_func, (void **) &err_array[i]);
I'm building a user defined shell. I have an array of pointers to functions -- that is, an array full of shared library functions that can be invoked at any point.
I typedef here
typedef void (*func_ptr)(char **);
func_ptr function;
void *pointers_to_functions[64];
I have debugged and confirmed that my initializations of placing the pointers into the array is working properly, but here's the code for safe measures...
void initialize_built_in(){
void *handle;
char *error;
int i;
for (i = 0; i < 5; i++){
handle = dlopen(builtin_files[i], RTLD_LOCAL | RTLD_LAZY);
if (!handle) {
fprintf(stderr, "%s\n", dlerror());
exit(1);
}
pointers_to_functions[i] = dlsym(handle, builtin_functions[i]);
if ((error = dlerror()) != NULL) {
fprintf(stderr, "%c\n", *error);
exit(1);
}
if (dlclose(handle) < 0) {
fprintf(stderr, "%c\n", *dlerror());
exit(1);
}
}
}
Here's where the seg fault occurs -- when I invoke the function
int execute_built_in(char **argv){
int i;
//scan through the builtin_functions strings to find the correct index of pointers_to_functions - that is, they have the same ordering
for (i = 0; i < sizeof(builtin_functions); i++){
if (!strcmp(argv[0], builtin_functions[i])){
//the index has been found
function = pointers_to_functions[i];
function(argv); //execute function
return 0;
}
}
return -1;
}
My shared library does indeed take argv as a parameter -- so I don't believe this is the problem.
As I said, debugging I see that the array of pointers is filled with addresses. I suppose it could be an incorrect address somehow, but I'm at a brick wall here.
Any ideas?
So I tested the pointer to function call be defining my own function type (void *) in the same file.
func_ptr function;
void *testfunction(char **);
void *pointers_to_functions[64] = {testfunction};
where the function just prints something out to the shell
then I added a condition in the execute_function function to force execution...
for (i = 0; i < sizeof(builtin_functions); i++){
if (i == 0){
function = pointers_to_functions[1];
char *bleh[] = {"bleh"};
function(bleh);
}
if (!strcmp(argv[0], builtin_functions[i])){
//the index has been found
function = pointers_to_functions[i];
function(argv); //execute function
return 0;
}
}
and it works WOOHOO!
So I either have a problem with my dynamic linking, which I can't see. Or with my shared library -- which is unlikely because I've already built successful libraries that work with another shell code of the same project.
So, what's wrong with my dynamic linking?
Here's an example shared library
#include <stdio.h>
#include <unistd.h>
struct NewBuiltIn{
char *CommandName[64];
char *FunctionName[64];
char *AnalyzerName[64];
};
struct NewBuiltIn pluggin_method = {{"cd", "cd", ""}};
void cd(char *argv[]) {
if(chdir(argv[1]) < 0){
printf("There was an error in changing your directory. Please check the path name and retry.\n");
}
}
Your init function is wrong:
you perform a dlclose(handle) which causes all the lib you loaded to be unmapped from memory, and obviously, the functions you expect to call are flying away.
You must keep the lib mapped to memory until you completely finished using the function pointers you stored.
You can probably check this by observing that the address that segfaults actually lies in the library mapped segment (in /proc/your_app_pid/maps)
I'm writing a variant of the producer-consumer problem with multi threading. I'm trying to use a queue to store the "produced" items until they get "consumed" later on. My problem is that when the consumer thread runs, it only processes the most recent item added to the queue (rather than the oldest item on the queue). Further, it processes that item repeatedly (up to the number of items on the queue itself).
I think that my problem might be that I need to allocate some memory when I push an item onto the queue (not sure about this, though). But then, I need a way to refer to this memory when that item is about to be consumed.
Anyway, here is a paired down version of my program. I realize that what I am posting here is incomplete (this is an infinite loop), but I'm trying just show the part that is relevant to this issue. The functions queue_push() and and queue_pop() are well tested, so I don't think that the problem lies there. I'll post more if needed.
Can anyone see why my consumer thread only processes the newest queue item? Thank you!
sem_t mutex;
queue q;
FILE* inputFPtr[10];
char host_in[BUFFERSIZE];
char host_out[BUFFERSIZE];
void* p(void* inputFile) {
while (fscanf(inputFile, INPUTFS, host_in) > 0)
{
sem_wait(&mutex);
queue_push(&q, host_in); //this function pushes the hostname onto the back of the queue
fprintf(stdout, "Produced: %d) %s\n", i, host_in);
sem_post(&mutex);
}
fclose (inputFile);
}
void* c() {
while (TRUE)
{
sem_wait(&mutex);
sprintf(hostname_out, "%s", (char *) queue_pop(&q));
printf("%s\n", host_out);
sem_post(&mutex);
}
}
int main (int argc, char* argv[]) {
int i;
pthread_t *th_in[argc-2];
pthread_t *th_out[2];
for (i = 0; i < (argc-2); i++) {
th_in[i] = (pthread_t *) malloc(sizeof(pthread_t));
inputFPtr[i] = fopen(argv[i+1], "r");
pthread_create (th_in[i], NULL, p, inputFPtr[i]);
}
for (i = 0; i < 2; i++) {
th_out[i] = (pthread_t *) malloc(sizeof(pthread_t));
pthread_create (th_out[i], NULL, c, null);
}
for (i = 0; i < (argc - 2); i++) {
pthread_join(*th_in[i], 0);
free(th_in[i]);
}
for (i = 0; i < (2); i++) {
pthread_join(*th_out[i], 0);
free(th_out[i]);
}
return EXIT_SUCCESS;
}
You forgot to post you code. However from your description, it seems like all the queue members point to the same memory block. This is why all your pops result with the same item.
The answer to you question is YES. You need to allocate memory for each one of the items and free it after it was "consumed".
Try to post some code for more specific answers...