I have a linux kernel module, I had to implement a TCP server and multiple clients to send some traffic. I had a question,
Now at the server end, I have a main-kthread which spawns another kthread once it sees a TCP connection waiting to be Accepted. Now after the main kthread spawns say a couple of kthreads how can I put the main-kthread in wait state to make sure the other kthreads exit before the main-kthread can exit.
I am looking something similar to phtread-join semantics, but i need this in the Linux kernel space.
I could not find kthread_join as such, I am looking at alternatives. I did read about waitqueues. Hence was looking at interruptible_sleep_on() and wake_up().
But am unable to figure out how to use these API's.
For example,
In the main-thread, I do a interruptible_sleep_on, then who should wake him up? All the other threads, or the last thread? how do i figure who is the last thread exiting? Also how to make sure, the other threads have exited when this main-kthread is woken up.
Wait queues are the right tools. There is a good description in Linux Device Drivers (LDD) chapter 6. Here's a sketch of how your code might look like:
Maintain a counter of server threads.
Increment the counter just before spawning a server thread.
When the server thread exists, decrease the counter.
In the main thread, when exiting, wait on a wait queue until the counter value is 0.
In the server threads, just after decreasing the counter, notify the main thread by signaling on the wait queue.
Warning: untested code, this is just a skeleton.
struct global_data {
wait_queue_head_t wq;
atomic_t thread_count;
};
main_thread () {
global_data gd = {0};
init_wait_queue_head(&gd.wq);
while (serving) {
…
atomic_inc(&server_threads);
if (IS_ERR_OR_NULL(kthread_create(server_thread, &gd)) {
atomic_dec(&server_threads);
}
}
wait_event_interruptible(&gd.wq, atomic_read(gd.thread_count) == 0);
/* final cleanup */
}
server_thread (global_data *gd) {
…
atomic_dec(&gd->server_threads);
wake_up(&gd->wq);
}
Related
Hi I'm stuck with this problem and I can't figure out which is the best solution for that:
I have a process with some threads. All the threads have signal masked and only one (SignalHandler) that check if there are pending signals.
while(!shouldEnd){
if(!sigwait(&set, &signum)){
if(signum == SIGINT){
fprintf(stderr, "Ricevuto SIGINT\n");
}
else{
//TODO
}
}
}
Now I would like to terminate even other threads when it receive a SIGINT (for instance). How can I do it properly ? The main problem is that the others thread might be waiting on a condition variable or waiting in a accept() (for socket connections). I think would be a good thing put a variable like "shouldEnd" and set it to 1 when threads should stop working. Is that a good idea ? And what about the waiting one ? Mostly for the one waiting for a socket connection ?
So, first and foremost, we don't know what you're using for threads. You might consider using libBoost for this, as it will make your life easier with RAII style locks and whatnot.
Anyhow, only the elected thread of your process (typically main(), in most examples), will catch the signal, unless you've enabled signal masking. When you want the threads to shut down cleanly, you just need to:
Wakeup the threads waiting on accept() by modifying your FD set to include a pipe that can also wake up the blocking call.
Simply signal the condvars the other threads are waiting on, and set some sort of mutex-protected boolean/flag to notify the thread that it should exit early (ie: mutexLock()l; bool bExitEarly = true; signal(condVar); mutexUnlock();).
Assuming you spawned the threads as joinable (ie: non-detached), just make sure you have a copy of the pointer to each thread object, and call thread_join() on each of them after you've signaled them to stop. This will ensure that the threads are fully stopped before main() exits. If you don't do this, main() could exit before the threads are done, and just forcibly kill off the threads while they are in the middle of their shutdown logic, which is messy, and could cause your program to crash or worse.
I am writing a simple multi-client server communication program using POSIX threads in C. I am creating a thread every time a new client is connected, i.e. after the accept(...) routine in main().
I have put the accept(...) and the pthread_create(...) inside a while(1) loop, so that server continues to accept clients forever. Now, where should I write the pthread_join(...) routine after a thread exits.
More Info: Inside the thread's "start routine", I have used poll() & then recv() functions, again inside a while(1) loop to continuously poll for availability of client and receive the data from client, respectively. The thread exits in following cases:
1) Either poll() returns some error event or client hangs up.
2) recv() returns a value <= 0.
Language: C
Platform: Suse Linux Enterprise Server 10.3 (x86_64)
First up starting a new thread for each client is probably wasteful and surely won't scale very far. You should try a design where a thread handles more than one client (i.e. calls poll on more than one socket). Indeed, that's what poll(2), epoll etc were designed for.
That being said, in this design you likely needn't join the threads at all. You're not mentioning any reason why the main thread would need information from a thread that finished. Put another way, there's no need for joining.
Just set them as "detached" (pthread_detach or pthread_attr_setdetachstate) and they will be cleaned up automatically when their function returns.
The problem is that pthread_join blocks the calling thread until the thread exits. This means you can't really call it and hope the thread have exited as then the main thread will not be able to do anything else until the thread have exited.
One solution is that each child thread have a flag that is polled by the main thread, and the child thread set that flag just before exiting. When the main thread notices the flag being set, it can join the child thread.
Another possible solution, is if you have e.g. pthread_tryjoin_np (which you should have since you're on a Linux system). Then the main thread in its loop can simply try to join all the child threads in a non-blocking way.
Yet another solution may be to detach the child threads. Then they will run by themselves and do not need to be joined.
Ah, the ol' clean shutdown problem.
Assuming that you may want to cleanly disconnect the server from all clients under some circumstance or other, your main thread will have to tell the client threads that they're to disconnect. So how could this be done?
One way would be to have a pipe (one per client thread) between the main thread and client thread. The client thread includes the file descriptor for that pipe in its call to poll(). That way the main thread can easily send a command to the client thread, telling it to terminate. The client thread reads the command when poll() tells it that the pipe has become ready for reading.
So your main thread can then send some sort of command through the pipe to the client thread and then call pthread_join() waiting for the client thread to tidy itself up and terminate.
Similarly another pipe (again one per client thread) can be used by the client thread to send information to the main thread. Instead of being stuck in a call to accept(), the main thread can be using poll() to wait for a new client connection and for messages from the existing client threads. A timeout on poll() also allows the main thread to do something periodically.
Welcome to the world of the actor model of concurrent programming.
Of course, if you don't need a clean shut down then you can just let threads terminate as and when they want to, and just ctrl c the program to close it...
As other people have said getting the balance of work per thread is important for efficient scaling.
I'm trying to start a thread as soon as an interrupt occurs. However, I have realized that I can't start a thread from within an interrupt handler (or any function that is directly or indirectly being called by the interrupt handler). So, what I have decided to do is have the handler assert a flag. Then, a separate thread continously monitors that flag and if it's asserted it will in turn create (and start) a thread. Here's a pseudocode:
int interrupt_flag = 0;
interrupt_handler(void)
{
interrupt_flag = 1
}
monitoring_thread(void) //this thread is started at the start of the program
{
while(1)
{
if(interrupt_flag)
{
interrupt_flag = 0;
//start the thread here
sleep(/*some amount of time*/);
}
}
}
I'm not really happy with having a dedicated while loop constantly monitoring a flag. The problem with this is that it significantly reduces the speed of the other threads in my program. For this reason, I'm calling the sleep function to increase the speed of the other threads in my program.
Question: Is there a way I can truly start a thread upon interrupt, without having a dedicated while loop? Is there a workaround for starting a thread from within an interrupt handler?
If it makes any difference, I'm using the POSIX library.
Thanks,
PS. This question is somewhat related to an earlier question posted here:
Sharing data between master thread and slave thread in interrupt driven environment in C
Instead of having your monitoring thread spin on a flag, it could wait until the interrupt handler provides notification that a thread should be spawned. One way to do this is with a semaphore:
sem_t interrupt_sem;
void interrupt_handler(void)
{
sem_post(&interrupt_sem);
}
void monitoring_thread(void)
{
while(1)
{
sem_wait(&interrupt_sem);
//start the thread here
}
}
Previously, I had a solution based on a condition variable, but it is unlikely your system would operate correctly if the interrupt handler makes blocking calls. It could cause a deadlock or other undefined behaviors, as the variables in the system may not have consistent values at the time the interrupt takes place.
As pointed out in comments by myself and others, your operating system should provide some kind of interface to explicitly wake up a waiting task. In the code above, we are assuming the monitoring thread is always active in the background.
you can use POSIX semaphore too
you can wait a semaphore that initial value is 0 by a thread that will be blocked by wait
and post this semaphore in your signal handle function
then , thread above will be waked up and do things you want(create thread)
I have an application that waits for clients to connect. Each time a client connects, a new frame gets created (with the new socket file descriptor). I know how many clients will connect, after I reach that number I just run pthread_join in a for loop.
My problem is that I would like the main thread to control all the other threads. My goal is to have each thread send the same message back to the client, at the same time, and only once. There are multiple messages a thread can send.
My current thinking is to define a list of command, as follows:
char *commands[] = {
(char*) "TERMINATE\0",
.... };
And then specify a command number that represents which command to use in that char* array. All threads will do something like
write(sockfd, buffer[commandNumber], length[commandNumber]);
I thought about waiting on a condition variable, but I see two problems:
1) I want to make sure that each thread, although synchronized, execute the command only once.
2) The main thread that initiates the command has to know when all those threads is done executing the command.
Only way I see to execute 2) is to keep track of a counter (with mutexes), and when each thread executes the command, it can increase that counter. I am not sure I will be able to avoid a thread from running the command twice.
What is the best possible way please to coordinate multiple threads to execute a single action at once; and also be able to know when that action has finished executing for every thread please?
You might use a barrier to gate the operation.
Synchronizing the send
The main thread initializes a barrier named "Ready" to N+1. Then it begins accept()ing N client connections, spawning a worker thread for each. The new worker threads immediately wait on barrier "Ready".
After spawning the Nth (and last) worker, the main thread sets the desired command (perhaps using a global commandNumber). Then the main thread waits on barrier "Ready". As soon as all workers and the main thread have arrived (reaching the barrier's limit of N+1), all threads are released, knowing that they are ready to issue their command immediately.
(A common alternate approach is to use a predicate and condition variable rather than a barrier. For example, the main thread might spawn the Nth worker and then cond_broadcast() that it has set a flag ready = 1. This approach is flawed. The main thread cannot know that the Nth worker — or, indeed, any of the workers — are yet waiting on that condition. The barrier solves this problem.)
Indicating completion
Another N+1 barrier, "AllDone", could be used to indicate that the workers are all done. A semaphore initialized to -N and posted by workers would do the same. Having the workers close() their connections and the main thread select()ing or poll()ing connections would convey the same information, too.
I have three threads, one thread is the main and the other two are worker threads. The first thread, when there is work to be done wakes up one of the two threads. Each thread when awakened perform some computation and while doing this if it finds more work to do can wake up the other working thread or simply decide to do the job by itself (By adding work to a local queue, for example).
While the worker threads have work to do, the main thread must wait for the work to be done. I have implemented this with condition variables as follows (the code reported here hides a lot of details, please ask if there's something non understandable):
MAIN THREAD (pseudocode):
//this function can be called from the main several time. It blocks the main thread till the work is done.
void new_work(){
//signaling to worker threads if work is available
//Now, the threads have been awakened, it's time to sleep till they have finished.
pthread_mutex_lock(&main_lock);
while (work > 0) //work is a shared atomic integer, incremented each time there's work to do and decremented when finished executing some work unit
pthread_cond_wait(&main_cond);
pthread_mutex_unlock(&main_lock);
}
WORKER THREADS:
while (1){
pthread_mutex_lock(&main_lock);
if (work == 0)
pthread_cond_signal(&main_cond);
pthread_mutex_unlock(&main_lock);
//code to let the worker thread wait again -- PROBLEM!
while (I have work to do, in my queue){
do_work()
}
}
Here is the problem: when a worker thread wakes up the main thread I'm not sure that the worker thread calls a wait to put itself in a waiting state for new work. Even if I implement this wait with another condition variable, it can happen that the main thread is awake, does some work until reaches a point in which he has to wake up the thread that has not called a wait yet... and this can lead to bad results. I've tried several ways to solve this issue but I couldn't find a solution, maybe there is an obvious way to solve it but I'm missing it.
Can you provide a scheme to solve this kind of problem? I'm using the C language and I can use whatever synchronization mechanism you think can be suited, like pthreads or posix semaphores.
Thanks
The usual way to handle this is to have a single work queue and protect it from overflow and underflow. Something like this (where I have left off the "pthread_" prefixes):
mutex queue_mutex;
cond_t queue_not_full, queue_not_empty;
void enqueue_work(Work w) {
mutex_lock(&queue_mutex);
while (queue_full())
cond_wait(&queue_not_full, &queue_mutex);
add_work_to_queue(w);
cond_signal(&queue_not_empty);
mutex_unlock(&queue_mutex);
}
Work dequeue_work() {
mutex_lock(&queue_mutex);
while (queue_empty())
cond_wait(&queue_not_empty, &queue_mutex);
Work w = remove_work_from_queue();
cond_signal(&queue_not_full);
mutex_unlock(&queue_mutex);
}
Note the symmetry between these functions: enqueue <-> dequeue, empty <-> full, not_empty <-> not full.
This provides a thread-safe bounded-size queue for any number of threads producing work and any number of threads consuming work. (Actually, it is sort of the canonical example for the use of condition variables.) If your solution does not look exactly like this, it should probably be pretty close...
If you want the main thread to distribute work to the other two, then wait until both threads have completed their work before moving on, you might be able to accomplish this with a barrier.
A barrier is a synchronization construct that you can use to make threads wait at a certain point in your code until a set number of threads are all ready to move on. Essentially, you initialize a pthread barrier, saying that x number of threads must wait on it before any are allowed to continue. As each thread finishes its work and is ready to go on, it will wait on the barrier, and once x number of threads have reached the barrier, they are all allowed to continue.
In your case, you might be able to do something like:
pthread_barrier_t barrier;
pthread_barrier_init(&barrier, 3);
master()
{
while (work_to_do) {
put_work_on_worker_queues();
pthread_barrier_wait(&barrier);
}
}
worker()
{
while(1) {
while (work_on_my_queue()) {
do_work();
}
pthread_barrier_wait(&barrier);
}
}
This should make your main thread give out work, then wait both worker threads to complete the work they were given (if any) before moving on.
Could you have "new job" queue, which is managed by the main thread? The main thread could dish out 1 job at a time to each worker thread. The main thread would also listen for completed jobs by the workers. If a worker thread finds a new job that needs doing just add it to the "new job" queue and the main thread will distribute it.
Pseudocode:
JobQueue NewJobs;
Job JobForWorker[NUM_WORKERS];
workerthread()
{
while(wait for new job)
{
do job (this may include adding new jobs to NewJobs queue)
signal job complete to main thread
}
}
main thread()
{
while(whatever)
{
wait for job completion on any worker thread
now a worker thread is free put a new job on it
}
}
I believe that what you have here is a variation on the producer-consumer problem. What you are doing is writing up an ad-hoc implementation of a counting semaphore (one that is used to provide more than just mutual exclusion).
If I've read your question right, what you are trying to do is have the worker threads block until there is a unit of work available and then perform a unit of work once it becomes available. Your issue is with the case where there is too much work available and the main thread tries to unblock a worker that is already working. I would structure your code as follows.
sem_t main_sem;
sem_init(&main_sem, 0, 0);
void new_work() {
sem_post(&main_sem);
pthread_cond_wait(&main_cond);
}
void do_work() {
while (1) {
sem_wait(&main_sem);
// do stuff
// do more stuff
pthread_cond_signal(&main_sem);
}
}
Now, if the worker threads generate more work then they can simply sem_post to the semaphore and simply defer the pthread_cond_signal till all the work is done.
Note however, if you actually need the main thread to always block when the worker is working, it's not useful to push the work to another thread when you could just call a function that does the work.