Without keeping a list of current threads, I'm trying to see that a realtime signal gets delivered to all threads in my process. My idea is to go about it like this:
Initially the signal handler is installed and the signal is unblocked in all threads.
When one thread wants to send the 'broadcast' signal, it acquires a mutex and sets a global flag that the broadcast is taking place.
The sender blocks the signal (using pthread_sigmask) for itself, and enters a loop repeatedly calling raise(sig) until sigpending indicates that the signal is pending (there were no threads remaining with the signal blocked).
As threads receive the signal, they act on it but wait in the signal handler for the broadcast flag to be cleared, so that the signal will remain masked.
The sender finishes the loop by unblocking the signal (in order to get its own delivery).
When the sender handles its own signal, it clears the global flag so that all the other threads can continue with their business.
The problem I'm running into is that pthread_sigmask is not being respected. Everything works right if I run the test program under strace (presumably due to different scheduling timing), but as soon as I run it alone, the sender receives its own signal (despite having blocked it..?) and none of the other threads ever get scheduled.
Any ideas what might be wrong? I've tried using sigqueue instead of raise, probing the signal mask, adding sleep all over the place to make sure the threads are patiently waiting for their signals, etc. and now I'm at a loss.
Edit: Thanks to psmears' answer, I think I understand the problem. Here's a potential solution. Feedback would be great:
At any given time, I can know the number of threads running, and I can prevent all thread creation and exiting during the broadcast signal if I need to.
The thread that wants to do the broadcast signal acquires a lock (so no other thread can do it at the same time), then blocks the signal for itself, and sends num_threads signals to the process, then unblocks the signal for itself.
The signal handler atomically increments a counter, and each instance of the signal handler waits until that counter is equal to num_threads to return.
The thread that did the broadcast also waits for the counter to reach num_threads, then it releases the lock.
One possible concern is that the signals will not get queued if the kernel is out of memory (Linux seems to have that issue). Do you know if sigqueue reliably informs the caller when it's unable to queue the signal (in which case I would loop until it succeeds), or could signals possibly be silently lost?
Edit 2: It seems to be working now. According to the documentation for sigqueue, it returns EAGAIN if it fails to queue the signal. But for robustness, I decided to just keep calling sigqueue until num_threads-1 signal handlers are running, interleaving calls to sched_yield after I've sent num_threads-1 signals.
There was a race condition at thread creation time, counting new threads, but I solved it with a strange (ab)use of read-write locks. Thread creation is "reading" and the broadcast signal is "writing", so unless there's a thread trying to broadcast, it doesn't create any contention at thread-creation.
raise() sends the signal to the current thread (only), so other threads won't receive it. I suspect that the fact that strace makes things work is a bug in strace (due to the way it works it ends up intercepting all signals sent to the process and re-raising them, so it may be re-raising them in the wrong way...).
You can probably get round that using kill(getpid(), <signal>) to send the signal to the current process as a whole.
However, another potential issue you might see is that sigpending() can indicate that the signal is pending on the process before all threads have received it - all that means is that there is at least one such signal pending for the process, and no CPU has yet become available to run a thread to deliver it...
Can you describe more details of what you're aiming to achieve? And how portable you want it to be? There's almost certainly a better way of doing it (signals are almost always a major headache, especially when mixed with threads...)
In multithreaded program raise(sig) is equivalent to pthread_kill(pthread_self(), sig).
Try kill(getpid(), sig)
Given that you can apparently lock thread creation and destruction, could you not just have the "broadcasting" thread post the required updates to thread-local-state in a per-thread queue, which each thread checks whenever it goes to use the thread-local-state? If there's outstanding update(s), it first applies them.
You are trying to synchronize a set of threads.
From a design pattern point of view the pthread native solution for your problem would be a pthread barrier.
Related
Platform is Linux/POSIX.
The signal is sent to a whole process, not a specific thread.
No signal is set to blocked, all default.
The process is multi-thread process.
From what I've googled, a signal may be handled by a random thread.
And when that signal's handler is executing, it's temporarily blocked until handler returns.
QUESTION: Multiple signals of different types reached simultaneously. Do their handler execute simultaneously on multiple thread or all of them go to one randomly picked thread (SUB-QUESTION: in this case a handler could interrupt another handler's execution started previously, so there could be a interrupt stack?) ? Or mixed? For instance there are 3 type of signals received but only 2 thread free (this is actually the first case).
EXAMPLE: SIGHUP, SIGINT, SIGTERM reached almost simultaneously. The program has two available thread to dispatch signal handler execution.
SIDE-QUESTION: If signal handlers run in parallel, I'll have to use mutex to synchronize them properly. Otherwise 'volatile sig_atomic_t' would be enough, right?
Expected: all signals go to one thread (randomly picked) despite of their different signal types, I haven't seen an example of using mutexes and atoms to synchronize signal handlers.
Your understanding is correct - unless a signal was directed
to a specific thread, there's no guarantee which thread will handle a signal.
See POSIX's Signal Generation and Delivery and pthreads(7):
POSIX.1 distinguishes the notions of signals that are directed
to the process as a whole and signals that are directed to
individual threads. According to POSIX.1, a process-directed
signal (sent using kill(2), for example) should be handled by
a single, arbitrarily selected thread within the process.
So it may be delivered & handled by the same thread that's currently handling another signal (in that case, the previous handler may be interrupted by the new signal). Or may be delivered to another signal.
You can block other signals while one is being handled using sa_mask field
of sigaction to avoid a signal handler being interrupted.
SIDE-QUESTION: If signal handlers run in parallel, I'll have to use mutex to synchronize them properly. Otherwise 'volatile sig_atomic_t' would be enough, right?
You almost certainly don't want to use mutex in a signal handler. There are only few functions that can be safely called from a signal handler (you can only call the functions that are async-signal-safe).
See signal-safty for more information.
If you can use volatile sig_atomic_t for whatever the purpose (do you need to co-ordinate execution of different signal handlers?), it should be preferred.
Expected: all signals go to one thread (randomly picked) despite of their different signal types, I haven't seen an example of using mutexes and atoms to synchronize signal handlers.
This is commonly done by blocking signals that you're interested in from main and fetching/handling them in a specific thread. See pthread_sigmask which also has an example on how to implement this.
This request is about PThreads and using conditions or signals to pause/resume a continuous cycle worker thread.
A while ago, I came into this:
https://stackoverflow.com/a/23945651/6421961
Basically, user johnnycrash uses sigwait() to get a thread into a pause state (waiting for external wakening) and pthread_kill(thread_id, USR1) to signal the thread into waking up. He claims it to be faster than using the mutex+condition construct and it appears to be less complex. I am developing a piece of software that would indeed require a thread to sleep until signaled and return to sleep after doing work in an infinite cycle (the eater of a feeder-eater paradigm).
I am using this to have a separate thread waiting for the conclusion of worker threads. In my current implementation, worker threads add their handles to a list protected by a mutex, signal the waiting thread with pthread_kill and finish with pthread_join.
My questions are all related:
How valid is it to actually use pthread_kill()+sigwait() instead of mutex+condition?
In case it is an acceptable solution, what pitfalls/race conditions
should one be aware of?
Would it be better to use pthread_sigqueue() instead of pthread_kill()? Would it actually be able catch signals sent while sigwait() is not running and immediately process them as soon as sigwait() is called?
Last question, derived from some contradicting information I found: Will different threads both paused with sigwait() expecting USR1 be able to be signaled independently, or will only one of them be able to actually catch the signal regardless of which one was signaled?
I will try to answer points 1 and 4.
pthread_kill() + sigqueue() and mutex+condition they both have their own purposes. When you're working with data (i.e. global variable) which is used by multiple threads in that case mutex are more appropriate. But, when you're waiting for an external event (like. network packet) and want to signal your thread based on that event pthread_kill() is more appropriate.
It depends how the signal (USR1) was sent. If it was sent using pthread_kill() or pthread_sigqueue() you can specify which thread you're sending that signal to, the only difference is with pthread_sigqueue() - you can send an additional information with the signal. You can also send signal to specific pid or group-wise signal sending using kill(). So, it largely depends on your need.
In my application ,I have multiple thread and one of these threads is responsible for catching signals and handling them. My main problem is since the OS picks up one of the threads in the process randomly and deliver to it the signal to handle it. So the thread which is picked up might be not the one which is responsible for handling the signal.
I have to block the signals in the main thread and any new thread will inherit the mask of the main thread, so they won't be able to handle the signals and only the corresponding thread will do that.
So can anyone provide me with a sample code in C, how to do that?
I have already wrote the part related to the keep the thread sleeping and wake it up upon the receipt of a signal, you can find the following stackoverflow - Thread blocked waiting for a signal.
Can someone explain why we should not call non async functions from signal handlers ? Like the exact sequence of steps that corrupt the programs while calling with such functions.
And, does signals always run on separate stack ? if so is it a separate context or it runs on the context of the signaled thread ?
Finally, in case of a multi-threaded system what happens when signal handler is executed and some other thread is signaled and calls the same signal handler ?
(I am trying to develop deep understanding of signals and its applications)
When a process receives a signal, it is handled in the context of the process. You should only use aync-safe functions or re-entrant functions from inside a signal handler. For instance, you cannot call a malloc() or a printf() within a signal handler. The reason being:
*) Lets assume your process was executing in malloc when you received the signal. So the global heap data structures are in an inconsistent state. Now if you acquire the heap lock from inside your signal handler and make changes you will further render the heap inconsistent.
*) Another possibility is if the heap lock has been acquired by your process when it received the signal, and then you call malloc() from your signal handler, it sees that lock is held and it waits infinitely to acquire the lock (infinitely because the thread that can release the lock will not run till the signal is completely handled).
2) Signals run in the context of the process. As for the signal stack you can look at this SO answer -> Do signal handers have a separate stack?
3) As for getting multiple instances of the same signal you can look at this link -> Signal Handling in UNIX where Rumple Stiltskin answers it well.
I know some Solaris. So I'm using that for details. LWP==Solaris for "thread" as in pthreads.
trap signals like SIGILL, are delivered to the thread that caused the trap. Asynchronous signals are delivered to the first active thread (LWP), or process that is not blocking that signal. A kernel module called aslwp() traverses the process-header table (has associated LWP's) looking for the first likely candidate to receive the asynch signal.
A signal stack lives in the kernel. I'm not sure what/how to answer your signal stack question.
One process may have several pending signals. Is that what you mean?
Each signal destined for a process is held there until the process switches context (or is forced) into the active state. This in part because you generally cannot incur a trap when the process context has been swapped out and the process does nothing cpu-wise. You certainly can incur asynch signals. But the process cannot "do anything" with any signal if it cannot run. So, at this point the kernel swaps the context back to active, and the signal is delivered via aslwp().
Realtime signals behave differently, and I'm letting it stay with that.
Try reading this:
developers.sun.com/solaris/articles/signalprimer.html
I am trying to create a wrapper on Linux which controls how many concurrent executions of something are allowed at once. To do so, I am using a system wide counting semaphore. I create the semaphore, do a sem_wait(), launch the child process and then do a sem_post() when the child terminates. That is fine.
The problem is how to safely handle signals sent to this wrapper. If it doesn't catch signals, the command might terminate without doing a sem_post(), causing the semaphore count to permanently decrease by one. So, I created a signal handler which does the sem_post(). But still, there is a problem.
If the handler is attached before the sem_wait() is performed, a signal could arrive before the sem_wait() completes, causing a sem_post() to occur without a sem_wait(). The reverse is possible if I do the sem_wait() before setting up the signal handler.
The obvious next step was to block signals during the setup of the handler and the sem_wait(). This is pseudocode of what I have now:
void handler(int sig)
{
sem_post(sem);
exit(1);
}
...
sigprocmask(...); /* Block signals */
sigaction(...); /* Set signal handler */
sem_wait(sem);
sigprocmask(...); /* Unblock signals */
RunChild();
sem_post(sem);
exit(0);
The problem now is that the sem_wait() can block and during that time, signals are blocked. A user attempting to kill the process may end up resorting to "kill -9" which is behaviour I don't want to encourage since I cannot handle that case no matter what. I could use sem_trywait() for a small time and test sigpending() but that impacts fairness because there is no longer a guarantee that the process waiting on the semaphore the longest will get to run next.
Is there a truly safe solution here which allows me to handle signals during semaphore acquisition? I am considering resorting to a "Do I have the semaphore" global and removing the signal blocking but that is not 100% safe since acquiring the semaphore and setting the global isn't atomic but might be better than blocking signals while waiting.
Are you sure sem_wait() causes signals to be blocked? I don't think this is the case. The man page for sem_wait() says that the EINTR error code is returned from sem_wait() if it is interrupted by a signal.
You should be able to handle this error code and then your signals will be received. Have you run into a case where signals have not been received?
I would make sure you handle the error codes that sem_wait() can return. Although it may be rare, if you want to be 100% sure you want to cover 100% of your bases.
Are you sure you are approaching the problem correctly? If you want to wait for a child terminating, you may want to use the waitpid() system call. As you observed, it is not reliable to expect the child to do the sem_post() if it may receive signals.
I know this is old, but for the benefit of those still reading this courtesy of Google...
The simplest (and only?) robust solution to this problem is to use a System V semaphore, which allows the client to acquire the semaphore resource in a way which is automatically returned by the kernel NO MATTER HOW THE PROCESS EXITS.