The ASIO documentation for basic_deadline_timer::cancel() has the following remarks section:
If the timer has already expired when cancel() is called, then the handlers for asynchronous wait operations will:
have already been invoked; or
have been queued for invocation in the near future.
These handlers can no longer be cancelled, and therefore are passed an error code that indicates the successful completion of the wait operation.
The emphasis has been added by me. Normally when you call cancel() on a timer, the callback is run with an error code of "operation cancelled by user". But this says there is a small chance it could actually be called with a success error code. I think it is trying to say that the following could happen:
Thread A calls async_wait(myTimerHandler) on a timer, where myTimerHandler() is a user callback function.
Thread B calls io_context::post(cancelMyTimer) where cancelMyTimer() is a user callback function. This is now queued up to be called in thread A.
The timer deadline expires, so ASIO queues up the timer callback handler, with a success error code. It doesn't call it yet, but it is queued up to be called in thread A.
ASIO gets round to calling cancelMyTimer() in thread A, which calls cancel() on the timer. But the timer already fired, and ASIO doesn't check that the handler is still queued up and not executed, so this does nothing.
ASIO now calls myTimerHandler, and doesn't check that cancel() was called in the meantime, and so it still passes success as the error code.
Bear in mind this example only has a single thread calling io_context::run(), deadline_timer::async_wait or deadline_timer::cancel(). The only thing that happened in another thread was a call to post(), which happened in an attempt to avoid any race conditions. Is this sequence of events possible? Or is it referring to some multithreading scenario (that seems unlikely given that timers are not thread safe)?
Context: If you have a timer that you wish to repeat periodically, then the obvious thing to do is check the error code in the callback, and set the timer again if the code is success. If the above race is possible, then it would be necessary to have a separate variable saying whether you cancelled the timer, which you update in addition to calling cancel().
You don't even need a second thread to run into a situation where basic_waitable_timer::cancel() is invoked too late (because the timer's (completion) handler is already queued).
It's sufficient that your program executes some other asynchronous operations concurrently to the not yet resumed basic_waitable_timer::async_wait(). If you then only rely on basic_waitable_timer::cancel() for cancellation then the cancel() call from another asynchronous (completion) handler races with an already scheduled async_wait() handler:
If the timer has already expired when cancel() is called, then the handlers for asynchronous wait operations will:
have already been invoked; or
have been queued for invocation in the near future.
These handlers can no longer be cancelled, and therefore are passed an error code that indicates the successful completion of the wait operation.
(basic_waitable_timer::cancel(), emphasis mine, i.e. the race condition is due to the second case)
A real-world example that is single-threaded (i.e. the program doesn't explicitly start any threads and only invokes io_server.run() once) and contains the described race:
void Fetch_Timer::resume()
{
timer_.expires_from_now(std::chrono::seconds(1));
timer_.async_wait([this](const boost::system::error_code &ec)
{
BOOST_LOG_FUNCTION();
if (ec) {
if (ec.value() == boost::asio::error::operation_aborted)
return;
THROW_ERROR(ec);
} else {
print();
resume();
}
});
}
void Fetch_Timer::stop()
{
print();
timer_.cancel();
}
(Source: imapdl/copy/fetch_timer.cc)
In this example, the obvious fix (i.e. also querying a boolean flag) doesn't even need to use any synchronization primitives (such as atomics), because the program is single-threaded. That means it executes (asynchronous) operations concurrently but not in parallel.
(FWIW, in the above example, the bug manifested itself only every 2 years or so, even under daily usage)
Everything you stated is correct. So in your situation you could need a separate variable to indicate you don’t want to continue the loop. I normally used a atomic_bool and I don’t bother posting a cancel routine, I just set the bool & call cancel from whatever thread I am on.
UPDATE:
The source for my answer is mainly experience in using ASIO for years and for understanding the asio codebase enough to fix problems and extend parts of it when required.
Yes the documentation says that the it's not thread safe between shared instances of the deadline_timer, but the documentation is not the best (what documentation is...). If you look at the source for how the "cancel" works we can see:
Boost Asio version 1.69: boost\asio\detail\impl\win_iocp_io_context.hpp
template <typename Time_Traits>
std::size_t win_iocp_io_context::cancel_timer(timer_queue<Time_Traits>& queue,
typename timer_queue<Time_Traits>::per_timer_data& timer,
std::size_t max_cancelled)
{
// If the service has been shut down we silently ignore the cancellation.
if (::InterlockedExchangeAdd(&shutdown_, 0) != 0)
return 0;
mutex::scoped_lock lock(dispatch_mutex_);
op_queue<win_iocp_operation> ops;
std::size_t n = queue.cancel_timer(timer, ops, max_cancelled);
post_deferred_completions(ops);
return n;
}
You can see that the cancel operation is guarded by a mutex lock so the "cancel" operation is thread safe.
Calling most of the other operations on deadline timer is not (in regards to calling them at the same time from multiple threads).
Also I think you are correct about the restarting of timers in quick order. I don't normally have a use case for stopping and starting timers in that sort of fashion, so I've never needed to do that.
Related
Quoting the documentation (emphasis theirs)
Timer callback functions execute in the context of the timer service task. It is therefore essential that timer callback functions never attempt to block. For example, a timer callback function must not call vTaskDelay(), vTaskDelayUntil(), or specify a non zero block time when accessing a queue or a semaphore.
The FreeRTOS reference book elaborates a little more, abut again without a clear explanation
It is ok to call functions such as xQueueReceive(), but
only if the function’s xTicksToWait parameter (which specifies the function’s block time) is set
to 0. It is not ok to call functions such as vTaskDelay(), as calling vTaskDelay() will always
place the calling task into the Blocked state.
My question is: why is this such a problem? I'm waiting for a semaphore, that's set by an interrupt, in the timer callback, and it has worked fine so far. (It's used for sending long packets using a USB bulk endpoint.)
Is the only issue possibly delaying other waiting timers?
The statement:
Timer callback functions execute in the context of the timer service task.
is the key. If your callback, blocks, you are blocking the timer service task, which if it were allowed to happen would delay other timer actions, and the RTOS scheduling guarantees could not be fulfilled.
The timer service task will perform the timer actions for all timers that have expired in that tick in a loop. If your timer callback were to perform a delay or blocking action, that would delay all timer actions not yet invoked but which were scheduled for the same tick, and would delay all actions in subsequent ticks if they came due while the service task were blocked.
If the action to be performed requires blocking (or even just takes a significant amount of time), the correct action would be to have your callback signal an independent task to perform the action. Timer callbacks should be treated like interrupt service routines - run to completion as quickly and as deterministically as possible, without blocking. In fact some RTOS actually invoke timer callbacks in the interrupt context rather then a special service task in any case, so it is a good guide to follow regardless of what RTOS you are using.
This really is two questions, but I suppose it's better they be combined.
We're working on a client that uses asynchronous TCP connection. The idea is that the program will block until certain message is received from the server, which will invoke a SIGPOLL handler. We are using a busy waiting loop, basically:
var = 1
while (var) usleep(100);
//...and somewhere else
void sigpoll_handler(int signum){
......
var = 0;
......
}
We would like to use something more reliable instead, like a semaphore. The thing is, when a thread is blocked on a semaphore, will the signal get through still? Especially considering that signals get delivered when it switches back to user level; if the process is off the runqueue, how will it happen?
Side question (just out of curiosity):
Without the "usleep(100)" the program never progresses past the while loop, although I can verify the variable was set in the handler. Why is that? Printing changes its behaviour too.
Cheers!
[too long for a comment]
Accessing var from inside the signal handler invokes undefined behaviour (at least for a POSIX conforming system).
From the related POSIX specification:
[...] if the process is single-threaded and a signal handler is executed [...] the behavior is undefined if the signal handler refers to any object [...] with static storage duration other than by assigning a value to an object declared as volatile sig_atomic_t [...]
So var shall be defined:
volatile sig_atomic_t var;
The busy waiting while-loop, can be replaced by a single call to a blocking pause(), as it will return on reception of the signal.
From the related POSIX specification:
The pause() function shall suspend the calling thread until delivery of a signal whose action is either to execute a signal-catching function or to terminate the process.
Using pause(), btw, will make the use of any global flag like var redundant, to not say needless.
Short answer: yes, the signal will get through fine with a good implementation.
If you're going to be using a semaphore to control the flow of the program, you'll want to have the listening be on one child with the actual data processing be on another. This will then put the concurrency fairness in the hands of the OS which will make sure your signal listening thread gets a chance to check for a signal with some regularity. It shouldn't ever be really "off the runqueue," but cycling through positions on the runqueue instead.
If it helps you to think about it, what you have right now seems to basically be a a very rough implementation of a semaphore on its own -- a shared variable whose value will stop one block of code from executing until another code block clears it. There isn't anything inherently paralyzing about a semaphore on a system level.
I kind of wonder why whatever function you're using to listen for the SIGPOLL isn't doing its own blocking, though. Most of those utilities that I've seen will stop their calling thread until they return a value. Basically they handle the concurrency for you and you can code as if you were dealing with a normal synchronous program.
With regards to the usleep loop: I'd have to look at what the optimizer's doing, but I think there are basically two possibilities. I think it's unlikely, but it could be that the no-body loop is compiling into something that isn't actually checking for a value change and is instead just looping. More likely to me would be that the lack of any body steps is messing up the underlying concurrency handling, and the loop is executing so quickly that nothing else is getting a chance to run -- the queue is being flooded by loop iterations and your signal processsing can't get a word in edgewise. You could try just watching it for a few hours to see if anything changes; theoretically if it's just a concurrency problem then the random factor involved could clear the block on its own with a few billion chances.
I need to use a non single shot Timer (such as QTimer from the Qt library) in a C code for embedded Linux (by no single shot Timer I mean one that triggers once ever x seconds indefinitely till a "stop timer" is called, not one that triggers only once or one that blocks the code while it's counting).
Some libraries that are available to me do implement such a timer using signal handlers, but I would like to avoid using such a system (I learned that is not the best solution). I know I can emulate what I want with single shot Timers by restarting the timer (calling it again) once it's finished, and that is an acceptable solution (actually the libraries I talked about work that way), but I don't know how to implement that without blocking the running code till the timer is triggered.
And one more thing: I need to be capable of implementing more then just one of them (here is where signal handlers stop being a viable solution AFAIK).
So how could I do such a solution? The closes to what Qt's QTimer has to offer, the better!
If you do need an unspecified number of triggers at varying intervals/times, a dedicated timer thread (as described by nneonneo in another answer) has the least number of pitfalls in my experience.
Timers are a limited resource (the number of timers available is configurable, and varies from system to system, so you cannot make any sweeping statements like "I'm sure there's enough for my purposes").
Signals interrupt blocking syscalls unless SA_RESTART flag is used; even then there are a few exceptions (see man 7 signal, Interruption of system calls and library functions by signal handlers chapter for details).
A dedicated timer thread is built around two components:
A queue, list, tree, or heap holding all timer events
A typical implementation only needs to know when the next event occurs, so a min-heap or a priority queue works quite well. I've found a min-heap to be simple and robust to implement, and efficient enough (O(log N) time complexity for both inserts and deletes); using absolute times (using CLOCK_MONOTONIC in Linux) for the events as keys.
Note that if you use the timer events as timeouts, you'll also need to make sure cancelling an event is efficient. In normal operation, timeouts are rare, so something like a web server is likely to cancel just about all the timeouts it sets, without any of them actually ever triggering.
A thread that waits for either the next event, or another thread inserting a new timer event
Personally, I use an array to hold the min-heap of events, protected by a pthread_mutex_t, with a pthread_cond_t for other threads to signal on after adding a new event. Then, it's a simple matter to use pthread_cond_timedwait() to wait/sleep for either the specified time, or until a thread notifies of a new event, whichever happens sooner.
When the next event occurs -- note that due to scheduling, you might find more than one separate event to occur, so you might not wish to sleep at all (but you might still check if new events were added) --, you perform the event. If the event is periodic, you reinsert it into the heap/queue, too, primed for the next time.
Choosing how events are performed is very important, and really, the only truly tricky bit. You can use flags -- switching from zero to nonzero is safe in practice, even if the change is not atomic, as long as you don't rely on any specific nonzero value --; you can cause a condition variable to be signaled or broadcast on; you can post a semaphore; you can raise a specific signal in a specific thread (even an empty signal handler will cause blocking I/O calls to interrupt, if the handler is installed without SA_RESTART flag; I've used this as an I/O timeout quite successfully); you can even use __atomic or __sync to modify a value atomically if using GCC (or Intel CC, Pathscale, or Portland Group C compilers); and so on.
If you need a specific function to be called, I recommend using a separate thread (or, if most of the work in the application/program/game is done in these timer events, a thread pool) to execute the events. This keeps the timer thread simple and precise, while keeping all resource use easily controlled. The worker thread or thread pool should simply have a FIFO queue of events protected by a mutex and a condition variable, so that the timer thread can add each event to the queue and then signal on the condition variable to notify the (next) worker thread that work is available.
Indeed, in the couple of instances I used other event action models, I now believe the function worker model would have been easier. Especially if you make the worker functions to take a pointer (to a structure), defined by the caller, so that they all have the same signature, this interface becomes quite straightforward to implement, but extremely powerful and versatile.
There is one downside to the timer-thread plus worker-thread(s) approach, and that is the (minimal) added latency. The worker thread will not get the work at the appointed time, but a short while afterwards. However, if you have the worker thread get the current time, compare to the (un-adjusted) target time, and use that as a statistic to trigger the events correspondingly prior to the target time, you can typically take care of this issue. (I have not verified, but I do believe both Qt and GTK+ toolkits do continuously estimate this latency in a similar if not same manner.)
Questions?
You have several options, none of which require any libraries beyond the standard C and POSIX libraries.
POSIX timers API, e.g. timer_create and friends. These have flexible notification scheme based on sigev, which allows you to specify how you want to be notified (signal to a specific thread, creation of a new thread, or arbitrary signal). By specifying that the signal goes to a specific thread, you can set that thread up to be ready for async signals, and use sig_atomic_t to signal work to be done by the thread. The most interesting notification option is to use the creation of a brand new thread, but note that this can get expensive if the timer fires frequently.
Linux timerfd API, e.g. timerfd_create. These create timers that you can poll with poll or epoll, enabling you to add the timers to a low-level event loop, as well as operate on them in a perfectly thread-safe and signal-safe way.
alarm. This uses the SIGALRM asynchronous signal, so again you'll want to use sig_atomic_t and a signal-processing thread to handle the timer.
select, poll, or nanosleep on a dedicated timer thread: This is what QTimer usually does under the covers. You simply create a dedicated timer thread and have the thread repeatedly sleep. To keep the timer on schedule, you adjust the sleep time based on the length of each processing cycle.
The last option is the most portable, but also basically the most work (since you're implementing the timer yourself). The upshot is that you get to customize the "timer" completely since you're implementing it on top of a sleep primitive.
I have created a timer using the simple "timer_create". The timer is created using SIGEV_THREAD. That is when the timer expires, there is a call to the timer thread function.
How timer_create works is, suppose assume: expiry=3 secs and timer interval is 1 ns, then the timer keeps ticking every 1 ns until expiry reaches. Once the timer expires, from that instance it keeps on hitting the timer thread function after every 1 ns (timer interval). And keeps on creating one thread per hit till the timer is deleted.
I don't want this to happen, i want once the timer expires, it should go and hit the thread function only once.
How can i achieve this? Can we put any option in timer_create? If not any other timer API?
Thanks a lot in advance
I think this is an implementation flaw in the glibc implementation of POSIX timers. There is certainly no way the timer_getoverrun function, which is critical for realtime usage, can work in the glibc implementation, since it returns from the kernel the overrun count for the "current" expiration, but when multiple expiration events are running in parallel, "current" makes no sense. There are also serious issues with resource exhaustion and dropped expiration events which make the implementation unusable for realtime purposes. For example, in nptl/sysdeps/unix/sysv/linux/timer_routines.c:
struct thread_start_data *td = malloc (sizeof (*td));
/* There is not much we can do if the allocation fails. */
...
In the Linux man page for sigevent, you see for SIGEV_THREAD:
Among the implementation possibilities here are that each timer notification could result in the creation of a new thread, or that a single thread is created to receive all notifications.
The latter is the only choice that could provide correct realtime semantics, but for some reason, glibc did not take this choice.
Here is a possible workaround:
Choose a realtime signal, block that signal before creating any threads, and setup your timer to use that signal with SIGEV_SIGNAL. Now, create a thread for handling your timer(s), and loop on sigwaitinfo, then call your handler function each time it returns. This is actually one possible implementation (and the most-correct implementation) of SIGEV_THREAD which glibc should be using.
Another possibility: there is exactly one synchronization-related, non-syscall-invoking, async-signal-safe function in POSIX: sem_post. Thus it may be possible to make a signal handler (as opposed to getting the signal from sigwaitinfo) synchronize with another thread for the purpose of delivering timer events. But I haven't worked out the details, and it seems like it may be difficult or impossible still.
Just set timer interval to 0 and expiry to whatever you want. Your timer will expire once (and thread created and run) and then stay disarmed.
gcc 4.4.2 c89
I have a function that has to run (config_relays). It make a call to a API function called set_relay, then the code has to wait before continuing until the event for set_relay event has completed. The set_relay is any Async call.
i.e.
void run_processes()
{
switch()
{
case EV_RELAY_SET:
break;
}
}
void config_relays()
{
set_relay();
/* Wait until EV_RELAY_SET has fired */
/* Cannot do init_relay until set_relay event has fired - has to block here */
init_relay();
}
I guess I could put the init_relay() in the switch. However, that event is used for other things and not just for initializing the relay. I would really like to handle everything in the config_relays function.
In C# you can do this by using autoreset. Does C have anything like that.
Many thanks for any advice,
As Anders wrote, conditional wait is the solution. In the POSIX Thread API you use pthread_cond_wait together with a mutex. It is quite easy, the following pattern works:
int ready_flag = 0;
pthread_mutex_t ready_mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t ready_cond = PTHREAD_COND_INITIALIZER;
void wait_ready()
{
pthread_mutex_lock(&ready_mutex);
while(!ready_flag) {
pthread_cond_wait(&ready_cond, &ready_mutex);
}
pthread_mutex_unlock(&ready_mutex);
}
void set_ready(int ready)
{
pthread_mutex_lock(&ready_mutex);
ready_flag = ready;
pthread_cond_signal(&ready_cond);
// or using pthread_cond_broadcast(&ready_cond);
pthread_mutex_unlock(&ready_mutex);
}
The difference between pthread_cond_signal and pthread_cond_broadcast is that if more than one thread waits for the flag to be set, pthread_cond_signal only releases one thread but broadcast releases all threads.
Not that the while loop created around your condition is up to you, you can test multiple conditions or do whatever you want. The code pattern ensures that your tests are performed on protected variables so that race conditions can never cause problems for example
while(resource_a_busy && resource_b_busy) ...
Is a typical problem where both resource states must be protected by a mutex.
The cond_wait can be removed from the loop, but then it would translate the wait_ready to a polling loop which consumes CPU, the pthread_wait_cond does not consume any CPU.
There are porting libraries that provides a Win32 like API on top of pthreads as well as libraries that gives a phread like API on top of Win32 event API, the later is called [Pthreads-w32] 1
1) You can use OS specific API's like createEvent or pthread condition variable to wait on and signal it when set_relay() is completed.
2) Polling approach. Poll periodically to see set_relay() is completed, else sleep for few seconds and retry.
It depends on what threading library you are using and how the asynchronous method is called. If you are on Windows, there are auto-reset events in the Windows API that you can use. See the CreateEvent API. If you are on unix/linux you can look into condition variables in pthreads. Unfortunately pthreads doesn't have auto-reset events, because they are very hard to use in a race-condition-free way.
When choosing your waiting strategy you also have to take into consideration how the asynchronous call is done. Is it on another thread? Is it done through signal handling? Some other asynch mechanism which "borrows" the main thread? If the call is done by "borrowing" the main thread, you have to make sure that your waiting strategy does not stop the thread from being used to perform the asynch call.