Some background:
I have an application that relies on third party hardware and a closed source driver. The driver currently has a bug in it that causes the device to stop responding after a random period of time. This is caused by an apparent deadlock within the driver and interrupts proper functioning of my application, which is in an always-on 24/7 highly visible environment.
What I have found is that attaching GDB to the process, and immediately detaching GDB from the process results in the device resuming functionality. This was my first indication that there was a thread locking issue within the driver itself. There is some kind of race condition that leads to a deadlock. Attaching GDB was obviously causing some reshuffling of threads and probably pushing them out of their wait state, causing them to re-evaluate their conditions and thus breaking the deadlock.
The question:
My question is simply this: is there a clean wait for an application to trigger all threads within the program to interrupt their wait state? One thing that definitely works (at least on my implementation) is to send a SIGSTOP followed immediately by a SIGCONT from another process (i.e. from bash):
kill -19 `cat /var/run/mypidfile` ; kill -18 `cat /var/run/mypidfile`
This triggers a spurious wake-up within the process and everything comes back to life.
I'm hoping there is an intelligent method to trigger a spurious wake-up of all threads within my process. Think pthread_cond_broadcast(...) but without having access to the actual condition variable being waited on.
Is this possible, or is relying on a program like kill my only approach?
The way you're doing it right now is probably the most correct and simplest. There is no "wake all waiting futexes in a given process" operation in the kernel, which is what you would need to achieve this more directly.
Note that if the failure-to-wake "deadlock" is in pthread_cond_wait but interrupting it with a signal breaks out of the deadlock, the bug cannot be in the application; it must actually be in the implementation of pthread condition variables. glibc has known unfixed bugs in its condition variable implementation; see http://sourceware.org/bugzilla/show_bug.cgi?id=13165 and related bug reports. However, you might have found a new one, since I don't think the existing known ones can be fixed by breaking out of the futex wait with a signal. If you can report this bug to the glibc bug tracker, it would be very helpful.
Related
Is there any (dirty)method to provoke context switching to specific process after specific ISR?
In normal situation, after an ISR, the process which was interrupted will keep running, and I have to wait the scheduler to pick that specific process. I want to switch to the specific process right away after the ISR.
Any advice will be great. Thanks!
Construct your driver so that the process has a thread blocking on suitable syscall (read(), ioctl()) with the ISR waking up that thread (because at least one byte became available for read()).
Then, make sure that thread has the highest priority possible, and preferably uses a realtime scheduler (SCHED_FIFO or SCHED_RR). In practice, if your process does not run with root privileges, you'll need to start the service with root privileges, setup the thread, then drop privileges; or give the binary executable CAP_SYS_NICE capability via e.g. setcap pe=CAP_SYS_NICE binary.
It is technically possible for the driver to also mess with the scheduling, but I would not do that. Anything that is so time-critical, should be done in the kernel ISR instead.
If you want to do it in userspace, because you don't want your code to be a derivative of the kernel and therefore GPL-licensed, you're on your own.
Let's consider a case where a user program calls a system call that has some synchronization measures. The simplest example would be
rwlock_t lock; // let's assume it's initialized properly.
write_lock(&lock);
// do something...
write_unlock(&lock);
Now, what happens when the user program terminates after locking lock but before releasing it is that the lock becomes perpetually locked, which we do not want it to happen. What we want the kernel to do is smartly detect any hanging locks and release them accordingly. But detecting such tasks can incur too much overhead, as the system needs to periodically record and check every task for every synchronizing action.
Or perhaps we can centralize the code into another kernel thread and do synchronization job there. But invoking on another thread still requires some form of synchronization, so I don't think it is possible to completely remove synchronizing code from the user program.
I have put a lot of thought into this and tried to google for some information but I couldn't see any light on this. Any help would be very much appreciated. Thank you.
I have an system running embedded linux and it is critical that it runs continuously. Basically it is a process for communicating to sensors and relaying that data to database and web client.
If a crash occurs, how do I restart the application automatically?
Also, there are several threads doing polling(eg sockets & uart communications). How do I ensure none of the threads get hung up or exit unexpectedly? Is there an easy to use watchdog that is threading friendly?
You can seamlessly restart your process as it dies with fork and waitpid as described in this answer. It does not cost any significant resources, since the OS will share the memory pages.
Which leaves only the problem of detecting a hung process. You can use any of the solutions pointed out by Michael Aaron Safyan for this, but a yet easier solution would be to use the alarm syscall repeatedly, having the signal terminate the process (use sigaction accordingly). As long as you keep calling alarm (i.e. as long as your program is running) it will keep running. Once you don't, the signal will fire.
That way, no extra programs needed, and only portable POSIX stuff used.
The gist of it is:
You need to detect if the program is still running and not hung.
You need to (re)start the program if the program is not running or is hung.
There are a number of different ways to do #1, but two that come to mind are:
Listening on a UNIX domain socket, to handle status requests. An external application can then inquire as to whether the application is still ok. If it gets no response within some timeout period, then it can be assumed that the application being queried has deadlocked or is dead.
Periodically touching a file with a preselected path. An external application can look a the timestamp for the file, and if it is stale, then it can assume that the appliation is dead or deadlocked.
With respect to #2, killing the previous PID and using fork+exec to launch a new process is typical. You might also consider making your application that runs "continuously", into an application that runs once, but then use "cron" or some other application to continuously rerun that single-run application.
Unfortunately, watchdog timers and getting out of deadlock are non-trivial issues. I don't know of any generic way to do it, and the few that I've seen are pretty ugly and not 100% bug-free. However, tsan can help detect potential deadlock scenarios and other threading issues with static analysis.
You could create a CRON job to check if the process is running with start-stop-daemon from time to time.
use this script for running your application
#!/bin/bash
while ! /path/to/program #This will wait for the program to exit successfully.
do
echo “restarting” # Else it will restart.
done
you can also put this script on your /etc/init.d/ in other to start as daemon
I'm trying to port a code that relies on SIGCONT to stop certain threads of an application. With current linux nptl implementation seems one can't rely on that in 2.6.x kernels. I'm trying to devise a method to stop other threads. Currently I can only think on mutexes and condition variables. Any hints is appreciated.
If you are relying on stopping and resuming other threads, then your application will eventually fail.
That is because, you cannot guarantee that you're not going to stop a thread while it has a mutex taken which protects a shared resource. This would result in deadlock, as any other threads (possibly including the one which stopped the first thread) which then need to wait for the mutex, will wait forever.
I'm sure it is possible, but also, you're doing it wrong.
NB: such mutexes probably exist in parts of the C library, even if you have none in your own code. If you have none in your own code and it is nontrivial, I'd be surprised.
How are you sending the signals to the target thread? If you use pthread_kill() to send SIGSTOP / SIGCONT to a single thread, it should work.
Is there any way to perform POSIX shared synchronization objects cleanup especially on process crash? Locked POSIX semaphores unblock is most desired thing but automatically 'collected' queues / shared memory region would be nice too. Another thing to keep eye on is we can't in general use signal handlers because of SIGKILL which cannot be caught.
I see only one alternative: some external daemon which accepts subscriptions and 'keep-alive' requests working as watchdog so not having notifications about some object it could close / unlock object in accordance to registered policy.
Has anyone better alternative / proposition? I never worked seriously with POSIX shared objects before (sockets were enough for all my needs and are much more useful by my opinion) and I did not found any applicable article. I'd gladly use sockets here but can't because of historical reasons.
Rather than using semaphores you could use file locking to co-oridinate your processes. The big advanatge of file locks being that they are released if the process terminates. You can map each semaphore onto a lock for a byte in a shared file and know that locks will get released on exit; in mosts version of unix the bytes you lock don't even have to exist. There is code for this in Marc Rochkind's book Advanced Unix Programming 1st edition, don't know if it's in the latest 2nd edition though.
I know this question is old, but another great solution is POSIX robust mutexes. They automatically unlock and enter an "inconsistent flag" state when the owner dies, and the next thread to attempt locking the mutex gets an EOWNERDEAD error but succeeds in becoming the new owner of the mutex. It's then able to clean up whatever state the mutex was protecting (which could be in a very bad inconsistent state due to asynchronous termination of the previous owner!) and mark the mutex as consistent again before unlocking it.
See the documentation on robust mutexes here:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_mutex_lock.html
The usual way is to work with signal handlers. Just catch the signals and call the cleanup functions.
But your watchdog daemon has some merits, too. It would surely make the system more simple to understand and manage. To make it more simple to administrate, your application should start the daemon when it's not running and the daemon should be able to clean up any residue from the last crash.