I'm using semaphore.h and would like to acquire a semaphore if n instead of just one slot is available. Posix does not provide this natively. How can I work around that? I'm bound to using semaphores, no other means of synchronization are possible.
I'm pondering using a binary semaphore with a separate counter variable, but that would, in my opinion, kind of defeat its purpose.
Since you have multiple threads contending for slots of the semaphore (else you wouldn't need semaphores at all), you need to protect against deadlock. For example, if your semaphore has four slots, and each of two threads is trying to acquire three, then they will deadlock if each manages to acquire two. It follows that you must protect access to the process of acquiring semaphore slots.
A binary semaphore protecting a counter is not sufficient to prevent the deadlock scenario described above. Moreover, if not enough slots are available at any given time then you must have some synchronous means to wait for more slots to become available. You can do the job with two semaphores, though, one to protect access to the semaphore acquisition process, and another carrying the actual slots being acquired. Something like this, for example:
#define DO_OR_RETURN(x) do { int _r; if ((_r = (x))) return _r; } while (0)
typedef struct multi_sem {
sem_t sem_acquire_sem;
sem_t multislot_sem;
} multisem;
int multisem_init(multisem *ms, unsigned int slots) {
DO_OR_RETURN(sem_init(&ms->sem_acquire_sem, 0, 1));
return sem_init(&ms->multislot_sem, 0, slots);
}
int multisem_wait(multisem *ms, unsigned int slots_to_acquire) {
int result;
DO_OR_RETURN(sem_wait(&ms->sem_acquire_sem));
while (slots_to_acquire) {
result = sem_wait(&ms->multislot_sem);
switch (result) {
case 0:
slots_to_acquire -= 1;
break;
case EINTR:
/* interrupted by a signal; try again */
break;
default:
/* undocumented error - should never happen */
/* insert appropriate apocalypse response here */
slots_to_acquire = 0; /* bail out */
break;
}
}
if (sem_post(&ms->sem_acquire_sem)) {
/* big oops - no recovery possible - should never happen */
/* insert appropriate apocalypse response here */
}
return result;
}
int multisem_post(multisem *ms, unsigned int slots_to_post) {
while (slots_to_post) {
DO_OR_RETURN(sem_post(&ms->multislot_sem));
slots_to_post -= 1;
}
return 0;
}
Do note that that is still susceptible to deadlock in the event that a thread tries to acquire slots of the multisem when it already holds at least one (among other ways). I think that risk is inherent in the problem.
Related
I have a problem where I have multiple threads(Read threads) where shared data is to be read, no issue so far.
This shared data is to be updated in another single thread(Write thread).
During this write no reads should be done to prevent corrupt data.
My initial thought was to use a semaphore(Posix semaphore, ) after doing some reading.
Using sem_wait() and sem_post() works fine for blocking, decrementing and incrementing the semaphore inside the read threads. As long as the semaphore is not 0, sem_wait() does not block and allow the thread to read the data.
My issue starts here.
Now I would like to have the condition for the write thread to be opposite of what sem_wait() actually does. I would like the write thread to be blocked as long as the semaphore is larger than zero as this means there is a read thread in progress.
Is there any way for it to block while semaphore is larger than 0 or is there any different approach that could solve this issue?
Thank you!
One option is to use POSIX read-write lock, where multiple readers can acquire the lock unless there is a writer.
Another option is a lower-level seqlock but it may be trickier to implement and use correctly because it requires a good grasp of the memory model data races and memory order.
If you must use semaphores, and are in a unix-y kind of environment, I think you are stuck with something like this:
typedef struct rwl RWL;
struct rwl {
sem_t lock;
sem_t wread;
sem_t wwrite;
int nreaders;
int nwriters;
};
void RWL_REnter(RWL *l) {
while (1) {
sem_wait(&l->lock);
l->nreaders++;
if (l->nwriters) {
sem_post(&l->lock);
sem_wait(&l->wread);
} else {
sem_post(&l->lock);
break;
}
}
}
void RWL_WEnter(RWL *l) {
while (1) {
sem_wait(&l->lock);
l->nwriters++;
if (l->nreaders || l->nwriters > 1) {
sem_post(&l->lock);
sem_wait(&l->wwrite);
} else {
sem_post(&l->lock);
break;
}
}
}
void RWL_WExit(RWL *l) {
sem_wait(&l->lock);
if (--(l->nwriters)) {
sem_post(&l->wwrite);
} else while (l->nreaders--) {
sem_post(&l->wread);
}
sem_post(&l->lock);
}
void RWL_RExit(RWL *l) {
sem_wait(&l->lock);
if (--(l->nreaders)) {
sem_post(&l->wread);
} else if (l->nwriters--) {
sem_post(&l->wwrite);
}
sem_post(&l->lock);
}
Functionally, lock is a mutex for the structure, and wwrite, wread serve as condition variables for the writers and readers respectively. nreaders is required to remember how many concurrent readers there are (since you can't wait on a +tive semaphore ) and nwriters for how many writers are waiting [ you don't want to post blindly in either condition ].
I'm studying on condition variables of Pthread. When I'm reading the explanation of pthread_cond_signal, I see the following.
The pthread_cond_signal() function shall unblock at least one of
the
threads that are blocked on the specified condition variable cond (if
any threads are blocked on cond).
Till now I knew pthread_cond_signal() would make only one thread to wake up at a time. But, the quoted explanation says at least one. What does it mean? Can it make more than one thread wake up? If yes, why is there pthread_cond_broadcast()?
En passant, I wish the following code taken from UNIX Systems Programming book of Robbins is also related to my question. Is there any reason the author's pthread_cond_broadcast() usage instead of pthread_cond_signal() in waitbarrier function? As a minor point, why is !berror checking needed too as a part of the predicate? When I try both of them by changing, I cannot see any difference.
/*
The program implements a thread-safe barrier by using condition variables. The limit
variable specifies how many threads must arrive at the barrier (execute the
waitbarrier) before the threads are released from the barrier.
The count variable specifies how many threads are currently waiting at the barrier.
Both variables are declared with the static attribute to force access through
initbarrier and waitbarrier. If successful, the initbarrier and waitbarrier
functions return 0. If unsuccessful, these functions return a nonzero error code.
*/
#include <errno.h>
#include <pthread.h>
#include <stdio.h>
static pthread_cond_t bcond = PTHREAD_COND_INITIALIZER;
static pthread_mutex_t bmutex = PTHREAD_MUTEX_INITIALIZER;
static int count = 0;
static int limit = 0;
int initbarrier(int n) { /* initialize the barrier to be size n */
int error;
if (error = pthread_mutex_lock(&bmutex)) /* couldn't lock, give up */
return error;
if (limit != 0) { /* barrier can only be initialized once */
pthread_mutex_unlock(&bmutex);
return EINVAL;
}
limit = n;
return pthread_mutex_unlock(&bmutex);
}
int waitbarrier(void) { /* wait at the barrier until all n threads arrive */
int berror = 0;
int error;
if (error = pthread_mutex_lock(&bmutex)) /* couldn't lock, give up */
return error;
if (limit <= 0) { /* make sure barrier initialized */
pthread_mutex_unlock(&bmutex);
return EINVAL;
}
count++;
while ((count < limit) && !berror)
berror = pthread_cond_wait(&bcond, &bmutex);
if (!berror) {
fprintf(stderr,"soner %d\n",
(int)pthread_self());
berror = pthread_cond_broadcast(&bcond); /* wake up everyone */
}
error = pthread_mutex_unlock(&bmutex);
if (berror)
return berror;
return error;
}
/* ARGSUSED */
static void *printthread(void *arg) {
fprintf(stderr,"This is the first print of thread %d\n",
(int)pthread_self());
waitbarrier();
fprintf(stderr,"This is the second print of thread %d\n",
(int)pthread_self());
return NULL;
}
int main(void) {
pthread_t t0,t1,t2;
if (initbarrier(3)) {
fprintf(stderr,"Error initilizing barrier\n");
return 1;
}
if (pthread_create(&t0,NULL,printthread,NULL))
fprintf(stderr,"Error creating thread 0.\n");
if (pthread_create(&t1,NULL,printthread,NULL))
fprintf(stderr,"Error creating thread 1.\n");
if (pthread_create(&t2,NULL,printthread,NULL))
fprintf(stderr,"Error creating thread 2.\n");
if (pthread_join(t0,NULL))
fprintf(stderr,"Error joining thread 0.\n");
if (pthread_join(t1,NULL))
fprintf(stderr,"Error joining thread 1.\n");
if (pthread_join(t2,NULL))
fprintf(stderr,"Error joining thread 2.\n");
fprintf(stderr,"All threads complete.\n");
return 0;
}
Due to spurious wake-ups pthread_cond_signal could wake up more than one thread.
Look for word "spurious" in pthread_cond_wait.c from glibc.
In waitbarrier it must wake up all threads when they all have arrived to that point, hence it uses pthread_cond_broadcast.
Can [pthread_cond_signal()] make more than one thread wake up?
That's not guaranteed. On some operating system, on some hardware platform, under some circumstances it could wake more than one thread. It is allowed to wake more than one thread because that gives the implementer more freedom to make it work in the most efficient way possible for any given hardware and OS.
It must wake at least one waiting thread, because otherwise, what would be the point of calling it?
But, if your applicaton needs a signal that is guaranteed to wake all of the waiting threads, then that is what pthread_cond_broadcast() is for.
Making efficient use of a multi-processor system is hard. https://www.e-reading.club/bookreader.php/134637/Herlihy,Shavit-_The_art_of_multiprocessor_programming.pdf
Most programming language and library standards allow similar freedoms in the behavior of multi-threaded programs, for the same reason: To allow programs to achieve high performance on a variety of different platforms.
For example I want to create 5 threads and print them. How do I make the fourth one execute before the second one? I tried locking it with a mutex, but I don't know how to make only the second one locked, so it gives me segmentation fault.
Normally, you define the order of operations, not the threads that do those operations. It may sound like a trivial distinction, but when you start implementing it, you'll see it makes for a major difference. It is also more efficient approach, because you don't think of the number of threads you need, but the number of operations or tasks to be done, and how many of them can be done in parallel, and how they might need to be ordered or sequenced.
For learning purposes, however, it might make sense to look at ordering threads instead.
The OP passes a pointer to a string for each worker thread function. That works, but is slightly odd; typically you pass an integer identifier instead:
#include <stdlib.h>
#include <inttypes.h>
#include <pthread.h>
#define ID_TO_POINTER(id) ((void *)((intptr_t)(id)))
#define POINTER_TO_ID(ptr) ((intptr_t)(ptr))
The conversion of the ID type -- which I assume to be a signed integer above, typically either an int or a long -- to a pointer is done via two casts. The first cast is to intptr_t type defined in <stdint.h> (which gets automatically included when you include <inttypes.h>), which is a signed integer type that can hold the value of any void pointer; the second cast is to a void pointer. The intermediate cast avoids a warning in case your ID is of an integer type that cannot be converted to/from a void pointer without potential loss of information (usually described in the warning as "of different size").
The simplest method of ordering POSIX threads, that is not that dissimilar to ordering operations or tasks or jobs, is to use a single mutex as a lock to protect the ID of the thread that should run next, and a related condition variable for threads to wait on, until their ID appears.
The one problem left, is to how to define the order. Typically, you'd simply increment or decrement the ID value -- decrementing means the threads would run in descending order of ID value, but the ID value of -1 (assuming you number your threads from 0 onwards) would always mean "all done", regardless of the number of threads used:
static pthread_mutex_t worker_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t worker_wait = PTHREAD_COND_INITIALIZER;
static int worker_id = /* number of threads - 1 */;
void *worker(void *dataptr)
{
const int id = POINTER_TO_ID(dataptr);
pthread_mutex_lock(&worker_lock);
while (worker_id >= 0) {
if (worker_id == id) {
/* Do the work! */
printf("Worker %d running.\n", id);
fflush(stdout);
/* Choose next worker */
worker_id--;
pthread_cond_broadcast(&worker_wait);
}
/* Wait for someone else to broadcast on the condition. */
pthread_cond_wait(&worker_wait, &worker_lock);
}
/* All done; worker_id became negative.
We still hold the mutex; release it. */
pthread_mutex_unlock(&worker_lock);
return NULL;
}
Note that I didn't let the worker exit immediately after its task is done; this is because I wanted to expand the example a bit: let's say you want to define the order of operations in an array:
static pthread_mutex_t worker_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t worker_wait = PTHREAD_COND_INITIALIZER;
static int worker_order[] = { 0, 1, 2, 3, 4, 2, 3, 1, 4, -1 };
static int *worker_idptr = worker_order;
void *worker(void *dataptr)
{
const int id = POINTER_TO_ID(dataptr);
pthread_mutex_lock(&worker_lock);
while (*worker_idptr >= 0) {
if (*worker_idptr == id) {
/* Do the work! */
printf("Worker %d running.\n", id);
fflush(stdout);
/* Choose next worker */
worker_idptr++;
pthread_cond_broadcast(&worker_wait);
}
/* Wait for someone else to broadcast on the condition. */
pthread_cond_wait(&worker_wait, &worker_lock);
}
/* All done; worker_id became negative.
We still hold the mutex; release it. */
pthread_mutex_unlock(&worker_lock);
return NULL;
}
See how little changed?
Let's consider a third case: a separate thread, say the main thread, decides which thread will run next. In this case, we need two condition variables: one for the workers to wait on, and the other for the main thread to wait on.
static pthread_mutex_t worker_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t worker_wait = PTHREAD_COND_INITIALIZER;
static pthread_cond_t worker_done = PTHREAD_COND_INITIALIZER;
static int worker_id = 0;
void *worker(void *dataptr)
{
const int id = POINTER_TO_ID(dataptr);
pthread_mutex_lock(&worker_lock);
while (worker_id >= 0) {
if (worker_id == id) {
/* Do the work! */
printf("Worker %d running.\n", id);
fflush(stdout);
/* Notify we are done. Since there is only
one thread waiting on the _done condition,
we can use _signal instead of _broadcast. */
pthread_cond_signal(&worker_done);
}
/* Wait for a change in the worker_id. */
pthread_cond_wait(&worker_wait, &worker_lock);
}
/* All done; worker_id became negative.
We still hold the mutex; release it. */
pthread_mutex_unlock(&worker_lock);
return NULL;
}
The thread that decides which worker should run first should hold the worker_lock mutex when the worker threads are created, then wait on the worker_done condition variable. When the first worker completes its task, it will signal on the worker_cone condition variable, and wait on the worker_wait condition variable. The decider thread should then change the worker_id to the next ID that should run, and broadcast on the worker_wait condition variable. This continues, until the decider thread sets worker_id to a negative value. For example:
int threads; /* number of threads to create */
pthread_t *ptids; /* already allocated for that many */
pthread_attr_t attrs;
int i, result;
/* Simple POSIX threads will work with 65536 bytes of stack
on all architectures -- actually, even half that. */
pthread_attr_init(&attrs);
pthread_attr_setstacksize(&attrs, 65536);
/* Hold the worker_lock. */
pthread_mutex_lock(&worker_lock);
/* Create 'threads' threads. */
for (i = 0; i < threads; i++) {
result = pthread_create(&(ptids[i]), &attrs, worker, ID_TO_POINTER(i));
if (result) {
fprintf(stderr, "Cannot create worker threads: %s.\n", strerror(result));
exit(EXIT_FAILURE);
}
}
/* Thread attributes are no longer needed. */
pthread_attr_destroy(&attrs);
while (1) {
/*
TODO: Set worker_id to a new value, or
break when done.
*/
/* Wake that worker */
pthread_cond_broadcast(&worker_wait);
/* Wait for that worker to complete */
pthread_cond_wait(&worker_done, &worker_lock);
}
/* Tell workers to exit */
worker_id = -1;
pthread_cond_broadcast(&worker_wait);
/* and reap the workers */
for (i = 0; i < threads; i++)
pthread_join(ptids[i], NULL);
There is a very important detail in all of the above examples, that may be hard to understand without a lot of practice: the way how mutexes and condition variables interact (if paired via pthread_cond_wait()).
When a thread calls pthread_cond_wait(), it will atomically release the specified mutex, and wait for new signals/broadcasts on the condition variable. "Atomic" means that there is no time inbetween the two; nothing can occur in between. The call returns when a signal or broadcast is received -- the difference is that a signal goes to only one, a random waiter; whereas a broadcast reaches all threads waiting on the condition variable --, and the thread acquires the lock. You can think of this as if the signal/broadcast first wakes up the thread, but the pthread_cond_wait() will only return when it re-acquires the mutex.
This behaviour is implicitly used in all of the examples above. In particular, you'll notice that the pthread_cond_signal()/pthread_cond_broadcast() is always done while holding the worker_lock mutex; this ensures that the other thread or threads wake up and get to act only after the worker_lock mutex is unlocked -- either explicitly, or by the holding thread waiting on a condition variable.
I thought I might draw a directed graph (using Graphviz) about the order of events and actions, but this "answer" is already too long. I do suggest you do it yourself -- perhaps on paper? -- as that kind of visualization has been very useful for myself when I was learning about all this stuff.
I do feel quite uncomfortable about the above scheme, I must admit. At any one time, only one thread is running, and that is basically wrong: any job where tasks should be done in a specific order, should only require one thread.
However, I showed the above examples in order for you (not just OP, but any C programmer interested in POSIX threads) to get more comfortable about how to use mutexes and condition variables.
I have some thread to write resource and some to read it.But pthread_rwlock cause a lot of context switch. So I imagine a way to avoid it. But I'm not sure it is safe or not.
This is the code:
sig_atomic_t slot = 0;
struct resource {
sig_atomic_t in_use; /*Counter,if in_use, not zero*/
.....
} xxx[2];
int read_thread()
{
i = slot; /*avoid slot changes in process */
xxx[i].in_use++;
read(xxx[i]);
xxx[i].in_use--;
}
int write_thread()
{
mutex_lock; /*mutex between write threads */
if (slot == 0) {
while(xxx[1].in_use != 0); /*wait last read thread in slot 1*/
clear(xxx[1]);
write(xxx[1]);
slot = 1;
} else if (slot == 1) {
while(xxx[0].in_use != 0);
clear(xxx[0]);
write(xxx[0]);
slot = 0;
}
mutex_unlock;
}
Will that works? The cost is 2 times storage and 3 atomic variable.
Thanks a lot!
Your algorithm is not lock-free; the writers use a spin lock.
Is it really necessary to do double-buffering and spin locks? Could you instead use (slot ^ 1) as the writing slot and slot as the reading slot? After writing, the writer would atomically change the value of slot, thus "publishing" its write. You may read the same slot many times consecutively this way, but if that's not the semantics you want then you should be using a queue.
By the way, a sig_atomic_t does not provide the type of atomicity you need for multiple threads. At a minimum, you should declare slot as volatile sig_atomic_t, and use memory barriers when reading and writing.
Your strategy is to have writers write to a different slot than what the readers are reading from. And you are switching the reading slot number after a write is completed. However, you will have a race.
slot reader writer1 writer2
---- ------ ------- -------
0 mutex_lock
i = 0
... slot=1
1 mutex_unlock mutex_lock
... clear(xxx[0])
xxx[0].in_use++
read(xxx[0]) write(xxx[0])
In general, though, this strategy could lead to starvation of writers (that is a writer may spin forever).
However, if you are willing to tolerate that, it would be safer to let xxx[] be an array of 2 pointers to resource. Let the reader always read from xxx[0], and let the writers contend for updates on xxx[1]. When a writer is finished updating xxx[1], it uses CAS on xxx[0] and xxx[1].
struct resource {
sig_atomic_t in_use; /*Counter,if in_use, not zero*/
sig_atomic_t writer;
.....
} *xxx[2];
void read_thread()
{
resource *p = xxx[0];
p->in_use++;
while (p->writer) {
p->in_use--;
p = xxx[0];
p->in_use++;
}
read(*p);
p->in_use--;
}
void write_thread()
{
resource *p;
mutex_lock; /*mutex between write threads */
xxx[1]->writer = 1;
while(xxx[1]->in_use != 0); /*wait last read thread in slot 1*/
clear(xxx[1]);
write(xxx[1]);
xxx[1] = CAS(&xxx[0], p = xxx[0], xxx[1]);
assert(p == xxx[1]);
xxx[0]->writer = 0;
mutex_unlock;
}
If you want to avoid writer starvation, but you want the performance of spinlocks, you are looking at implementing your own reader/writer locks using spinlocks instead of mutex locks. A google search for "read write spinlock implementation" pointed to this page which I found to be an interesting read.
I've got some code that currently looks like this (simplified)
/* instance in global var *mystruct, count initialized to 0 */
typedef struct {
volatile unsigned int count;
} mystruct_t;
pthread_mutex_t mymutex; // is initialized
/* one thread, goal: block while mystruct->count == 0 */
void x(void *n) {
while(1) {
pthread_mutex_lock(&mymutex);
if(mystruct->count != 0) break;
pthread_mutex_unlock(&mymutex);
}
pthread_mutex_unlock(&mymutex);
printf("count no longer equals zero");
pthread_exit((void*) 0)
}
/* another thread */
void y(void *n) {
sleep(10);
pthread_mutex_lock(&mymutex);
mystruct->count = 10;
pthread_mutex_unlock(&mymutex);
}
This seems inefficient and wrong to me--but I don't know a better way of doing it. Is there a better way, and if so, what is it?
Condition variables allow you to wait for a certain event, and have a different thread signal that condition variable.
You could have a thread that does this:
for (;;)
{
if (avail() > 0)
do_work();
else
pthread_cond_wait();
}
and a different thread that does this:
for (;;)
{
put_work();
pthread_cond_signal();
}
Very simplified of course. :) You'll need to look up how to use it properly, there are some difficulties working with condition variables due to race conditions.
However, if you are certain that the thread will block for a very short time (in order of µs) and rarely, using a spin loop like that is probably more efficient.
A general solution is to use a POSIX semaphore. These are not part of the pthread library but work with pthreads just the same.
Since semaphores are provided in most other multi-threading APIs, it is a general technique that may be applied perhaps more portably; however perhaps more appropriate in this instance is a condition variable, which allows a thread to pend on the conditional value of a variable without polling, which seems to be exactly what you want.
Condition variables are the solution to this problem. Your code can be fairly easily modified to use them:
pthread_mutex_t mymutex = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t mycond = PTHREAD_COND_INITIALIZER;
/* one thread, goal: block while mystruct->count == 0 */
void x(void *n) {
pthread_mutex_lock(&mymutex);
while (mystruct->count == 0)
pthread_cond_wait(&mycond, &mymutex);
printf("count no longer equals zero");
pthread_mutex_unlock(&mymutex);
pthread_exit((void*) 0)
}
/* another thread */
void y(void *n) {
sleep(10);
pthread_mutex_lock(&mymutex);
mystruct->count = 10;
pthread_cond_signal(&mycond);
pthread_mutex_unlock(&mymutex);
}
Note that you should generally keep the mutex locked while you act on the result - "consume" the event or similar. That's why I moved the pthread_mutex_unlock() to a point after the printf(), even though it doesn't really matter in this toy case.
(Also, in real code it might well make sense to put the mutex and condition variable inside mystruct).
You could use barriers.
You may also use semaphores to synchronize threads.