I have a simple apache web server with a hook function and a handler function.
int globalVar1;
int globalVar2;
static void register_hooks(apr_pool_t *pool)
{
globalVar1 = 9; /* or any other value... */
/* Create a hook in the request handler, so we get called when a request arrives */
ap_hook_handler(example_handler, NULL, NULL, APR_HOOK_LAST);
}
static int example_handler(request_rec *r)
{
printf("globalVar1=%d", globalVar1); /* print 9 */
globalVar2 = 9; /* or any other value... */
printf("globalVar2=%d", globalVar2); /* sometimes print 9 and sometimes not - race condition. */
/* do something... */
return OK;
}
I noticed that when I initialize globalVar1 in the hook, the variable has the same value as I initialized in the hook,
although the hook and the handler are been called on different processes.
1. what is the reason for that kind of behavior?
As a result, I decided to move the variable initialization to the handler function (globalVar2).
The problem I noticed is happening when the handler gets 2 requests at the same time and therefore the variable is not being initialized correctly.
So if I want to avoid race condition, I need to use lock on the variable initialization, but if I want to use lock, I need to initialize the lock before
and again I have the problem of initialization in multi threaded system.
2. How can I use lock in this kind of situation?
By mentioning variable initialization, I mean any initialization, even calling another function to initialize a global struct.
It could be much easier if I could share memory between the two processes (hook and handler), but from the investigation I did - it is impossible.
To make sure the initialization is done only once in a multithreaded situation, use a function CallOnce, that internally makes sure it is called exactly once.
For example: C11 threads.h call_once, or pthread_once.
Related
I have a data structure which I personally implemented that now needs to be used across multiple threads.
typedef struct
{
void** array_of_elements;
size_t size;
} myStruct;
For simplicity, let's say my data structure has these functions:
// Gets a data element from the structure.
void* get(myStruct *x);
// Prints out all the data elements.
void print(myStruct *x);
// Adds an element into the structure.
void add(myStruct *x, void *to_be_added);
It's not a problem whatsoever to call get while another thread is calling print since they are both accessors. However, get and print cannot work while add is currently being called. Vice versa, add cannot work if get and print are currently in-progress.
So I changed myStruct to look like the following:
typedef struct
{
void** array_of_elements;
size_t size;
// True when a mutator is editing this struct.
bool mutating;
// The number of threads currently accessing this struct.
int accessors;
} myStruct;
Now my functions look like the following:
void* get(myStruct *x)
{
// Wait for mutating to end.
while (x->mutating);
// Indicate that another accessor is now using this struct.
x->accessors++;
// get algorithm goes here
// Declare we are finished reading.
x->accessors--;
return ...
}
// Same as above...
void print(myStruct *x)
...
void add(myStruct *x)
{
// Wait for any accessors or mutators to finish.
while (x->mutating || x->accessors > 0);
x->mutating = true;
// add algorithm here
x->mutating = false;
}
BUT, I think there are a lot of problems with this approach and I can't find a way to solve them:
One of my classmates told me using while loops like this slows the thread down immensely.
It has no sense of a queue. The first method that begins waiting for the myStruct to finish being used isn't necessarily the one that goes next.
Even IF I had a queue data structure for which thread goes next, that data structure would also need to be synchronized, which in itself is an infinite loop of needing a synchronized data structure to synchronize itself.
I think it's possible that in the same nano second one thread changes the accessors counter from 0 to 1 (meaning they want to start reading), it's possible for a mutator thread to see it's value is 0 and start mutating. Then, both a mutator thread and an accessor thread would be going at the same time.
I'm pretty sure this logic can cause grid-lock (threads waiting on each other infinitely).
I don't know how to make certain threads sleep and wake up right when they need to for this task, besides having it stuck in a while loop.
You have the right idea, just the wrong approach. I'm not sure what OS you're programming on, but you want to look at the concepts of mutex or semaphore to do what you want to do.
On Linux/Unix that is POSIX compliant, you can look at pthreads:
http://www.cs.wm.edu/wmpthreads.html
On Windows, you can look at Critical Sections for something close to a mutex concept:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682530(v=vs.85).aspx
Or WaitForMultipleObjects for something close to a semaphore:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms687025(v=vs.85).aspx
And, yes, using while loops are a bad idea. In this case, you are using what is known as a busy loop. More reading on it here:
What is a busy loop?
Using mutex or semaphore, no while loop is required. Good luck!
I have a question about a program I am writing, one which requires some multi-threading. The thread function requires a couple of parameters, which I am passing by doing the following:
Having a struct defined for the parameters (lets call it ThreadDataStruct)
Initializing this struct and filling the values (call our ThreadDataStruct instance THREAD_DATA)
Passing the values on to my thread during CreateThread by setting lpParameter to a pointer to the struct (&THREAD_DATA)
Re-cast the LPVOID to ThreadDataStruct in the thread function
The issue is that my function which creates the threads returns directly after the call to CreateThread. My question is: can returning so soon after creating the thread cause the thread to not get its parameters? If the struct was created in the function and then the function returns, won't the parameters for the thread function be destroyed if it can't get them fast enough?
Here is some POC to show what I mean:
Definition of our struct:
typedef struct ThreadDataStruct
{
int Number;
};
Our function we are running inside:
void Function()
{
ThreadDataStruct THREAD_DATA;
THREAD_DATA.Number = 1;
CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)ThreadFunction, &THREAD_DATA, 0, NULL);
return;
}
Our thread function:
void ThreadFunction(LPVOID lpParam)
{
ThreadDataStruct THREAD_DATA = *(ThreadDataStruct *)lpParam;
int NumberFromStruct = THREAD_DATA.Number;
return;
}
Do I need to find another way to pass the parameters safely?
Transferring comments into an answer.
Yes, you have a race condition; and yes, you need to have a different way of passing the parameters to the function safely. The variable passed to the thread function is (normally) passed by address, so it needs to last as long as the thread needs to access it. In your code, the function returns and that means the variable is no longer valid, so all hell breaks loose.
Thank you for clearing that up. What methods are there to pass the parameters safely? I guess I could put a Sleep(5000) after CreateThread(), but that obviously is not the best way to do this.
One option is to have Function() wait for the the thread to complete before it returns. Another option is to have Function() wait on a semaphore, and the ThreadFunction() would signal on the semaphore when it has finished reading its control data.
Another option, probably the best, is to have the function that calls Function() pass in a ThreadDataStruct * that can be used, and for it to keep that around until all the threads have terminated. So, if it is main() that calls Function(), then main() might have an array of ThreadDataStruct thread_params[32]; and it could pass &thread_params[i] to the ith call to Function() (probably within a loop).
void Function(ThreadDataStruct *THREAD_DATA)
{
THREAD_DATA->Number = 1;
CreateThread(NULL, 0, ThreadFunction, THREAD_DATA, 0, NULL);
}
int main(void)
{
ThreadDataStruct thread_params[32];
for (int i = 0; i < 32; i++)
Function(&thread_params[i]);
…code to wait for threads to exit…
}
Changes in Function() include:
Add argument.
Initialize argument via pointer.
Pass argument directly, not &THREAD_DATA.
Don't cast the thread function pointer type; make sure ThreadFunction() is the correct type to be called by CreateThread().
There's no virtue in a return; at the end of a void function.
No doubt there are other equivalent techniques available too; these come to mind immediately. Normally, though, the controlling code has some sort of array of the data structures available, and passes one of the elements of the array to each invocation of the thread-launching function.
In a signal handler, I'm changing the value of a variable. However, the program never notices the update, even though I've declared the variable leader_barrier as sig_atomic_t.
void timer_action(int signum)
{
static int count = 0;
if ( !(*pbarrier_in_proc) && !(leader_barrier) && !(*pno_more) )
leader_barrier = 1;
}
And its confirmed that timer_action does execute and leader_barrier does become 1 inside it, as I have seen it by printing its values inside the signal handler.
You should declare leader_barrier as volatile sig_atomic_t, not just sig_atomic_t. Otherwise the compiler is free to assume the variable does not change asynchronously. That is, it may read it once at the start of the function (say) and, assuming no other functions that could change leaderboard get called, it need not read it again.
Maybe the two variables are not the same variable. Try printing their addresses in both the signal handler and the other code in question.
This question is based on:
When is it safe to destroy a pthread barrier?
and the recent glibc bug report:
http://sourceware.org/bugzilla/show_bug.cgi?id=12674
I'm not sure about the semaphores issue reported in glibc, but presumably it's supposed to be valid to destroy a barrier as soon as pthread_barrier_wait returns, as per the above linked question. (Normally, the thread that got PTHREAD_BARRIER_SERIAL_THREAD, or a "special" thread that already considered itself "responsible" for the barrier object, would be the one to destroy it.) The main use case I can think of is when a barrier is used to synchronize a new thread's use of data on the creating thread's stack, preventing the creating thread from returning until the new thread gets to use the data; other barriers probably have a lifetime equal to that of the whole program, or controlled by some other synchronization object.
In any case, how can an implementation ensure that destruction of the barrier (and possibly even unmapping of the memory it resides in) is safe as soon as pthread_barrier_wait returns in any thread? It seems the other threads that have not yet returned would need to examine at least some part of the barrier object to finish their work and return, much like how, in the glibc bug report cited above, sem_post has to examine the waiters count after having adjusted the semaphore value.
I'm going to take another crack at this with an example implementation of pthread_barrier_wait() that uses mutex and condition variable functionality as might be provided by a pthreads implementation. Note that this example doesn't try to deal with performance considerations (specifically, when the waiting threads are unblocked, they are all re-serialized when exiting the wait). I think that using something like Linux Futex objects could help with the performance issues, but Futexes are still pretty much out of my experience.
Also, I doubt that this example handles signals or errors correctly (if at all in the case of signals). But I think proper support for those things can be added as an exercise for the reader.
My main fear is that the example may have a race condition or deadlock (the mutex handling is more complex than I like). Also note that it is an example that hasn't even been compiled. Treat it as pseudo-code. Also keep in mind that my experience is mainly in Windows - I'm tackling this more as an educational opportunity than anything else. So the quality of the pseudo-code may well be pretty low.
However, disclaimers aside, I think it may give an idea of how the problem asked in the question could be handled (ie., how can the pthread_barrier_wait() function allow the pthread_barrier_t object it uses to be destroyed by any of the released threads without danger of using the barrier object by one or more threads on their way out).
Here goes:
/*
* Since this is a part of the implementation of the pthread API, it uses
* reserved names that start with "__" for internal structures and functions
*
* Functions such as __mutex_lock() and __cond_wait() perform the same function
* as the corresponding pthread API.
*/
// struct __barrier_wait data is intended to hold all the data
// that `pthread_barrier_wait()` will need after releasing
// waiting threads. This will allow the function to avoid
// touching the passed in pthread_barrier_t object after
// the wait is satisfied (since any of the released threads
// can destroy it)
struct __barrier_waitdata {
struct __mutex cond_mutex;
struct __cond cond;
unsigned waiter_count;
int wait_complete;
};
struct __barrier {
unsigned count;
struct __mutex waitdata_mutex;
struct __barrier_waitdata* pwaitdata;
};
typedef struct __barrier pthread_barrier_t;
int __barrier_waitdata_init( struct __barrier_waitdata* pwaitdata)
{
waitdata.waiter_count = 0;
waitdata.wait_complete = 0;
rc = __mutex_init( &waitdata.cond_mutex, NULL);
if (!rc) {
return rc;
}
rc = __cond_init( &waitdata.cond, NULL);
if (!rc) {
__mutex_destroy( &pwaitdata->waitdata_mutex);
return rc;
}
return 0;
}
int pthread_barrier_init(pthread_barrier_t *barrier, const pthread_barrierattr_t *attr, unsigned int count)
{
int rc;
rc = __mutex_init( &barrier->waitdata_mutex, NULL);
if (!rc) return rc;
barrier->pwaitdata = NULL;
barrier->count = count;
//TODO: deal with attr
}
int pthread_barrier_wait(pthread_barrier_t *barrier)
{
int rc;
struct __barrier_waitdata* pwaitdata;
unsigned target_count;
// potential waitdata block (only one thread's will actually be used)
struct __barrier_waitdata waitdata;
// nothing to do if we only need to wait for one thread...
if (barrier->count == 1) return PTHREAD_BARRIER_SERIAL_THREAD;
rc = __mutex_lock( &barrier->waitdata_mutex);
if (!rc) return rc;
if (!barrier->pwaitdata) {
// no other thread has claimed the waitdata block yet -
// we'll use this thread's
rc = __barrier_waitdata_init( &waitdata);
if (!rc) {
__mutex_unlock( &barrier->waitdata_mutex);
return rc;
}
barrier->pwaitdata = &waitdata;
}
pwaitdata = barrier->pwaitdata;
target_count = barrier->count;
// all data necessary for handling the return from a wait is pointed to
// by `pwaitdata`, and `pwaitdata` points to a block of data on the stack of
// one of the waiting threads. We have to make sure that the thread that owns
// that block waits until all others have finished with the information
// pointed to by `pwaitdata` before it returns. However, after the 'big' wait
// is completed, the `pthread_barrier_t` object that's passed into this
// function isn't used. The last operation done to `*barrier` is to set
// `barrier->pwaitdata = NULL` to satisfy the requirement that this function
// leaves `*barrier` in a state as if `pthread_barrier_init()` had been called - and
// that operation is done by the thread that signals the wait condition
// completion before the completion is signaled.
// note: we're still holding `barrier->waitdata_mutex`;
rc = __mutex_lock( &pwaitdata->cond_mutex);
pwaitdata->waiter_count += 1;
if (pwaitdata->waiter_count < target_count) {
// need to wait for other threads
__mutex_unlock( &barrier->waitdata_mutex);
do {
// TODO: handle the return code from `__cond_wait()` to break out of this
// if a signal makes that necessary
__cond_wait( &pwaitdata->cond, &pwaitdata->cond_mutex);
} while (!pwaitdata->wait_complete);
}
else {
// this thread satisfies the wait - unblock all the other waiters
pwaitdata->wait_complete = 1;
// 'release' our use of the passed in pthread_barrier_t object
barrier->pwaitdata = NULL;
// unlock the barrier's waitdata_mutex - the barrier is
// ready for use by another set of threads
__mutex_unlock( barrier->waitdata_mutex);
// finally, unblock the waiting threads
__cond_broadcast( &pwaitdata->cond);
}
// at this point, barrier->waitdata_mutex is unlocked, the
// barrier->pwaitdata pointer has been cleared, and no further
// use of `*barrier` is permitted...
// however, each thread still has a valid `pwaitdata` pointer - the
// thread that owns that block needs to wait until all others have
// dropped the pwaitdata->waiter_count
// also, at this point the `pwaitdata->cond_mutex` is locked, so
// we're in a critical section
rc = 0;
pwaitdata->waiter_count--;
if (pwaitdata == &waitdata) {
// this thread owns the waitdata block - it needs to hang around until
// all other threads are done
// as a convenience, this thread will be the one that returns
// PTHREAD_BARRIER_SERIAL_THREAD
rc = PTHREAD_BARRIER_SERIAL_THREAD;
while (pwaitdata->waiter_count!= 0) {
__cond_wait( &pwaitdata->cond, &pwaitdata->cond_mutex);
};
__mutex_unlock( &pwaitdata->cond_mutex);
__cond_destroy( &pwaitdata->cond);
__mutex_destroy( &pwaitdata_cond_mutex);
}
else if (pwaitdata->waiter_count == 0) {
__cond_signal( &pwaitdata->cond);
__mutex_unlock( &pwaitdata->cond_mutex);
}
return rc;
}
17 July 20111: Update in response to a comment/question about process-shared barriers
I forgot completely about the situation with barriers that are shared between processes. And as you mention, the idea I outlined will fail horribly in that case. I don't really have experience with POSIX shared memory use, so any suggestions I make should be tempered with scepticism.
To summarize (for my benefit, if no one else's):
When any of the threads gets control after pthread_barrier_wait() returns, the barrier object needs to be in the 'init' state (however, the most recent pthread_barrier_init() on that object set it). Also implied by the API is that once any of the threads return, one or more of the the following things could occur:
another call to pthread_barrier_wait() to start a new round of synchronization of threads
pthread_barrier_destroy() on the barrier object
the memory allocated for the barrier object could be freed or unshared if it's in a shared memory region.
These things mean that before the pthread_barrier_wait() call allows any thread to return, it pretty much needs to ensure that all waiting threads are no longer using the barrier object in the context of that call. My first answer addressed this by creating a 'local' set of synchronization objects (a mutex and an associated condition variable) outside of the barrier object that would block all the threads. These local synchronization objects were allocated on the stack of the thread that happened to call pthread_barrier_wait() first.
I think that something similar would need to be done for barriers that are process-shared. However, in that case simply allocating those sync objects on a thread's stack isn't adequate (since the other processes would have no access). For a process-shared barrier, those objects would have to be allocated in process-shared memory. I think the technique I listed above could be applied similarly:
the waitdata_mutex that controls the 'allocation' of the local sync variables (the waitdata block) would be in process-shared memory already by virtue of it being in the barrier struct. Of course, when the barrier is set to THEAD_PROCESS_SHARED, that attribute would also need to be applied to the waitdata_mutex
when __barrier_waitdata_init() is called to initialize the local mutex & condition variable, it would have to allocate those objects in shared memory instead of simply using the stack-based waitdata variable.
when the 'cleanup' thread destroys the mutex and the condition variable in the waitdata block, it would also need to clean up the process-shared memory allocation for the block.
in the case where shared memory is used, there needs to be some mechanism to ensured that the shared memory object is opened at least once in each process, and closed the correct number of times in each process (but not closed entirely before every thread in the process is finished using it). I haven't thought through exactly how that would be done...
I think these changes would allow the scheme to operate with process-shared barriers. the last bullet point above is a key item to figure out. Another is how to construct a name for the shared memory object that will hold the 'local' process-shared waitdata. There are certain attributes you'd want for that name:
you'd want the storage for the name to reside in the struct pthread_barrier_t structure so all process have access to it; that means a known limit to the length of the name
you'd want the name to be unique to each 'instance' of a set of calls to pthread_barrier_wait() because it might be possible for a second round of waiting to start before all threads have gotten all the way out of the first round waiting (so the process-shared memory block set up for the waitdata might not have been freed yet). So the name probably has to be based on things like process id, thread id, address of the barrier object, and an atomic counter.
I don't know whether or not there are security implications to having the name be 'guessable'. if so, some randomization needs to be added - no idea how much. Maybe you'd also need to hash the data mentioned above along with the random bits. Like I said, I really have no idea if this is important or not.
As far as I can see there is no need for pthread_barrier_destroy to be an immediate operation. You could have it wait until all threads that are still in their wakeup phase are woken up.
E.g you could have an atomic counter awakening that initially set to the number of threads that are woken up. Then it would be decremented as last action before pthread_barrier_wait returns. pthread_barrier_destroy then just could be spinning until that counter falls to 0.
I have been writing C code for many years, but I recently came accross a feature that I have never used: a static variable inside a function. Therefore, I was wondering what are some ways that you have used this feature and it was the right design decision.
E.g.
int count(){
static int n;
n = n + 1;
return n;
}
is a BAD design decision. why? because later you might want to decrement the count which would involve changing the function parameters, changing all calling code, ...
Hopefully this is clear enough,
thanks!
void first_call()
{
static int n = 0;
if(!n) {
/* do stuff here only on first call */
n++;
}
/* other stuff here */
}
I have used static variables in test code for lazy initialization of state. Using static local variables in production code is fraught with peril and can lead to subtle bugs. It seems (at least in the code that I generally work on) that nearly any bit of code that starts out as a single-threaded only chunk of code has a nasty habit of eventually ending up working in a concurrent situation. And using a static variable in a concurrent environment can result in difficult issues to debug. The reason for this is because the resulting state change is essentially a hidden side effect.
I have used static variables as a way to control the execution of another thread.
For instance, thread #1 (the main thread) first declares and initializes a control variable such as:
/* on thread #1 */
static bool run_thread = true;
// then initialize the worker thread
and then it starts the execution of thread #2, which is going to do some work until thread #1 decides to stop it:
/* thread #2 */
while (run_thread)
{
// work until thread #1 stops me
}
There is one prominent example that you very much need to be static for protecting critical sections, namely a mutex. As an example for POSIX threads:
static pthread_mutex_t mut = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_lock(&mut);
/* critical code comes here */
pthread_mutex_unlock(&mut);
This wouldn't work with an auto variable.
POSIX has such static initialzers for mutexes, conditions and once variables.