How does Timer component polls? - apache-camel

Let's say I have following time component:
from("timer://foo?period=1000").setBody(constant("select * from customer")).to("jdbc:testdb").to("beanRef:processResult");
How does timer component work here? Does it reads from database in every 1 sec or waits for bean to finish the processing?
If bean is still processing the earlier result and timer will keep polling the database then it will create a bottleneck. Is there any way to avoid it?

Okay, Update: looking at the source code, the timer component relies on the java TimerTask implementation. And your question is already answered here: Is Java's Timer task guarenteed not to run concurrently?
Short answer: one single thread executes the trigger and the routes connected to it, so there will be no concurrent execution.
That said, you might want to controll the execution a bit. It is recommended with Timer Tasks (and hence Camel timers) to have a margin between the period in the timer and the max task execution time.
You can use a SEDA component (with concurrentConsumers=[num threads]) in between to fine grain controll the execution with a work queue. The timer will finish it's task right away while the real route can continue to process.
from("timer://foo?period=1000")
.to("seda:startRoute");
from("seda:startRoute")
.setBody(constant("select * from customer"))
.to("jdbc:testdb").to("beanRef:processResult");
Each event will stack up non the less, so over time, you might want tune the route so that the period > avg route exec time.
You could add a shared boolean variable either in a singleton bean or in a static class:
public static synchronized boolean isRunning(){
return running;
}
public static synchronized void setRunning(boolean isRunning){
running = isRunning;
}
The variable should telling weather the route is running or not and filter timer events that occurs while the variable is true. Just hook up a few processors/bean-calls to handle this.

Related

Some questions related to Fraud detection demo from Flink DataStream API

The example is very useful at first,it illustrates how keyedProcessFunction is working in Flink
there is something worth noticing, it suddenly came to me...
It is from Fraud Detector v2: State + Time part
It is reasonable to set a timer here, regarding the real application requirement part
override def onTimer(
timestamp: Long,
ctx: KeyedProcessFunction[Long, Transaction, Alert]#OnTimerContext,
out: Collector[Alert]): Unit = {
// remove flag after 1 minute
timerState.clear()
flagState.clear()
}
Here is the problem:
The TimeCharacteristic IS ProcessingTime which is determined by the system clock of the running machine, according to ProcessingTime property, the watermark will NOT be changed overtime, so that means onTimer will never be called, unless the TimeCharacteristic changes to eventTime
According the flink website:
An hourly processing time window will include all records that arrived at a specific operator between the times when the system clock indicated the full hour. For example, if an application begins running at 9:15am, the first hourly processing time window will include events processed between 9:15am and 10:00am, the next window will include events processed between 10:00am and 11:00am, and so on.
If the watermark doesn't change over time, will the window function be triggered? because the condition for a window to be triggered is when the watermark enters the end time of a window
I'm wondering the condition where the window is triggered or not doesn't depend on watermark in priocessingTime, even though the official website doesn't mention that at all, it will be based on the processing time to trigger the window
Hope someone can spend a little time on this,many thx!
Let me try to clarify a few things:
Flink provides two kinds of timers: event time timers, and processing time timers. An event time timer is triggered by the arrival of a watermark equal to or greater than the timer's timestamp, and a processing time timer is triggered by the system clock reaching the timer's timestamp.
Watermarks are only relevant when doing event time processing, and only purpose they serve is to trigger event time timers. They play no role at all in applications like the one in this DataStream API Code Walkthrough that you have referred to. If this application used event time timers, either directly, or indirectly (by using event time windows, or through one of the higher level APIs like SQL or CEP), then it would need watermarks. But since it only uses processing time timers, it has no use for watermarks.
BTW, this fraud detection example isn't using Flink's Window API, because Flink's windowing mechanism isn't a good fit for this application's requirements. Here we are trying to a match a pattern to a sequence of events within a specific timeframe -- so we want a different kind of "window" that begins at the moment of a special triggering event (a small transaction, in this case), rather than a TimeWindow (like those provided by Flink's Window API) that is aligned to the clock (i.e., 10:00am to 10:01am).

Libev: how to schedule a callback to be called as soon as possible

I'm learning libev and I've stumbled upon this question. Assume that I want to process something as soon as possible but not now (i.e. not in the current executing function). For example I want to divide some big synchronous job into multiple pieces that will be queued so that other callbacks can fire in between. In other words I want to schedule a callback with timeout 0.
So the first idea is to use ev_timer with timeout 0. The first question is: is that efficient? Is libev capable of transforming 0 timeout timer into an efficient "call as soon as possible" job? I assume it is not.
I've been digging through libev's docs and I found other options as well:
it can artificially delay invoking the callback by using a prepare or idle watcher
So the idle watcher is probably not going to be good here because
Idle watchers trigger events when no other events of the same or higher priority are pending
Which probably is not what I want. Prepare watchers might work here. But why not check watcher? Is there any crutial difference in the context I'm talking about?
The other option these docs suggest is:
or more sneakily, by reusing an existing (stopped) watcher and pushing it into the pending queue:
ev_set_cb (watcher, callback);
ev_feed_event (EV_A_ watcher, 0);
But that would require to always have a stopped watcher. Also since I don't know a priori how many calls I want to schedule at the same time then I would have to have multiple watchers and additionally keep track of them via some kind of list and increase it when needed.
So am I on the right track? Are these all possibilities or am I missing something simple?
You may want to check out the ev_prepare watcher. That one is scheduled for execution as the last handler in the given event loop iteration. It can be used for 'Execute this task ASAP' implementations. You can create dedicated watcher for each task you want to execute, or you can implement a queue with a one prepare watcher that is started once queue contains at least one task.
Alternatively, you can implement similar mechanism using ev_idle watcher, but this time, it will be executed only if the application doesn't process any 'higher priority' watcher handlers.

Activiti / Camunda change boundary timer with variable

I got a special question regarding timer boundary events on a user task in Activiti/Camunda:
When starting the process I set the timer duration with a process variable and use expressions in the boundary definition to resolve the variable. The boundary event is defined on a user task.
<bpmn2:timerEventDefinition id="_TimerEventDefinition_11">
<bpmn2:timeDuration xsi:type="bpmn2:tFormalExpression">${hurry}</bpmn2:timeDuration>
</bpmn2:timerEventDefinition>
In some cases, when the timer is already running it can occur, that the deadline (dueDate) should be extended because the asignee has requested more time. For this purpose i want to change the value of the process variable defining the deadline.
As it happens, the variable is already resolved at the process-start and set to the boundary event.
Any further changes of the variable do not affect the dueDate of the boundary timer because it is stored in the database and is not updated when the value of the variable changes.
I know how to update the dueDate of the job element via the Java API, but i want to provide a generic approach like setting it with changing the value of the variable.
The most common use case for extending the deadline will be when the boundary timer is already running.
Any ideas how to cope with this problem?
Any tips are very apprechiated.
Cheers Chris
After some time of thinking I came up with a workaround like that:
I start the process with two variables. "hurry" is evaluated for the boundary timer. And "extendDeadline" is initialized with false. If the timer triggers and the process advances to the exclusive gateway, the value of "extendDeadline" is evaluated.
If a user has changed the value of "extendDeadline" to true during the the timer was running the process returns to the user task again where the boundary timer is set to the value of "hurry".
If the "extendDeadline" is still set to false, the process can proceed.
If the timer is running you can change dueDate of the timer by executing a signal. If a assginee requested more time, set new value of hurry and execute the signal. The old timer will be canceled and the new timer will be created with new due date.
runtimeService.setVariable(execution.getId(), "hurry", newDueDate);
runtimeService.signalEventReceived(signalName, execution.getId());
Solution is to have 2 out going sequence flow, one should be from boundary timer on task and another one should be from task it self, as shown in diagram added by #theFriedC. See following image also.
Then you can use some exclusive gateway on 2nd sequence flow and reroute that back to same task with a new timer value.

Problem with WPF Application and slow GUI response

I've been analyzing a WPF application which basically fetch data from a server and display the data in the GUI.
This code is not mine, and the application has a problem related with slow response from the GUI, I'm trying to find the reason for that problem.
I want to share with you my idea of which the problem could be and I'll like to hear what do you think about it, whether it makes any sense or not.
To get the data from the server, the application is using 7 threads (this is done in this way mostly because of the application logic, so don't pay too much attention to why 7 and not just one...), now, each thread is created by calling a method, called CreateThreadForTask()
public void StartAllThreads()
{
this.CreateThreadForTask(Tasks.Task1);
this.CreateThreadForTask(Tasks.Task2);
this.CreateThreadForTask(Tasks.Task3);
this.CreateThreadForTask(Tasks.Task4);
this.CreateThreadForTask(Tasks.Task5);
this.CreateThreadForTask(Tasks.Task6);
this.CreateThreadForTask(Tasks.Task7);
}
public void CreateThreadForTask(Tasks task)
{
... // this part of the code is not important
//! Initialize and start timer
timer = null;
timer = new DispatcherTimer();
timer.Tick += new EventHandler(RunMainSyncForTask);
timer.Start();
}
public void RunMainSyncForTask(object s, EventArgs e)
{
int sec = int.Parse(AppSettings.GetSetting("syncInterval"));
timer.Interval = new TimeSpan(0, 0, sec);
//threadCaller is a background worker
threadCaller = InitializeThread();
threadCaller.DoWork += DoWorkEventHandler(StartSync);
threadCaller.RunWorkerAsync();
}
When I was debugging the code I noticed that all the threads are created using a DispatcherTimer; what I think is that the application is creating 7 DispatcherTimer's and is linking the Tick event of the timers with the RunMainSyncForTask() method, which inside, create a background worker that fetch the data from the server and save that data to a local database.
Now, this was taken from the MSDN
The DispatcherTimer is reevaluated at the top of every Dispatcher loop.
Timers are not guaranteed to execute exactly when the time interval occurs, but they are guaranteed to not execute before the time interval occurs. This is because DispatcherTimer operations are placed on the Dispatcher queue like other operations. When the DispatcherTimer operation executes is dependent on the other jobs in the queue and their priorities.
So, based on that I believe that the application is spamming threads every time a timer does a tick event, and this is done 7 times simultaneously; and all these operations, because of the DispatcherTimer nature, are being added to the Dispatcher queue, which makes the GUI response slow, due to the Dispatcher being busy.
Also, another problem with the application is that, when it runs, it takes about 90-95% of the CPU, I think that if my hypothesis is right, this could be also the cause of for this problem.
So if you can share some insides about this I'll appreciate it.
Thanks.
You're getting the 90-95% CPU because you've instituted a form of busy waiting through a crazy web of threading calls.
If you're using this StartSync logic to post status notifications or get data back to the GUI, you're jumping through a lot of hoops. If you're on .Net 4.0 you should switch to the Task Parallel Library and let the framework handle all of this for you. It also supports graceful cancellation, etc.
If you don't wish to use TPL, I would suggest instead passing the Windows Dispatcher (use the usual suspects: Invoke or BeginInvoke) or SynchronizationContext (asynchronously with Post, synchronously with Send) to the individual tasks for use for these tasks which must be done in the GUI.

WPF Application thread usage going up and up

I have a multithreaded WPF application that is using > 600 threads after is has been running for more than 8 hours. All but approximately 10 of these threads have a stack trace that is very similar to this:
Stack trace 1:
ntkrnlpa.exe!NtInitialUserProcessBuffer+0x7b
ntkrnlpa.exe!MiAddWorkingSetPage+0x174
ntkrnlpa.exe!MiAddWsleHash+0x12a
ntkrnlpa.exe!PopSystemButtonHandler+0x141
ntkrnlpa.exe!KiInterruptTemplate+0x62
ntdll.dll!KiFastSystemCallRet
ntdll.dll!ZwWaitForMultipleObjects+0xc
KERNEL32.dll!WaitForMultipleObjectsEx+0x12c
mscorwks.dll!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
mscorwks.dll!Thread::DoAppropriateAptStateWait+0x3c
mscorwks.dll!Thread::DoAppropriateWaitWorker+0x13c
mscorwks.dll!Thread::DoAppropriateWait+0x40
mscorwks.dll!WaitHandleNative::CorWaitOneNative+0x156
mscorlib.ni.dll+0x1f68af
mscorlib.ni.dll+0x1caa17
WindowsBase.ni.dll+0x24ac34
WindowsBase.ni.dll+0x2aeb1e
WindowsBase.ni.dll+0x9445d
WindowsBase.ni.dll+0x9267f
mscorwks.dll!JITutil_IsInstanceOfAny+0x106
mscorlib.ni.dll+0x1e842f
mscorwks.dll!CallDescrWorker+0x33
mscorwks.dll!CallDescrWorkerWithHandler+0xa3
mscorwks.dll!MethodDesc::CallDescr+0x19c
mscorwks.dll!MethodDesc::CallTargetWorker+0x1f
mscorwks.dll!MethodDescCallSite::Call+0x1a
mscorwks.dll!ExecuteCodeWithGuaranteedCleanupHelper+0x9f
mscorwks.dll!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0x10f
mscorlib.ni.dll+0x235677
mscorlib.ni.dll+0x2202a5
mscorlib.ni.dll+0x1e839b
mscorwks.dll!CallDescrWorker+0x33
mscorwks.dll!CallDescrWorkerWithHandler+0xa3
mscorwks.dll!DispatchCallBody+0x1e
mscorwks.dll!DispatchCallDebuggerWrapper+0x3d
mscorwks.dll!DispatchCallNoEH+0x51
mscorwks.dll!AddTimerCallback_Worker+0x66
mscorwks.dll!Thread::DoADCallBack+0x32a
mscorwks.dll!Thread::ShouldChangeAbortToUnload+0xe3
mscorwks.dll!Thread::ShouldChangeAbortToUnload+0x30a
mscorwks.dll!Thread::ShouldChangeAbortToUnload+0x33e
mscorwks.dll!ManagedThreadBase::ThreadPool+0x13
mscorwks.dll!AddTimerCallbackEx+0x83
mscorwks.dll!AddTimerCallback+0x10
mscorwks.dll!ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
mscorwks.dll!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x9a
mscorwks.dll!ThreadpoolMgr::ExecuteWorkRequest+0xaf
mscorwks.dll!ThreadpoolMgr::WorkerThreadStart+0x20b
mscorwks.dll!Thread::intermediateThreadProc+0x49
KERNEL32.dll!BaseThreadStart+0x37
Stack Trace 2:
ntkrnlpa.exe!NtInitialUserProcessBuffer+0x7b
ntkrnlpa.exe!MiAddWorkingSetPage+0x174
ntkrnlpa.exe!MiAddWsleHash+0x12a
ntkrnlpa.exe!PopSystemButtonHandler+0x141
ntkrnlpa.exe!KiInterruptTemplate+0x62
ntdll.dll!KiFastSystemCallRet
ntdll.dll!ZwWaitForMultipleObjects+0xc
KERNEL32.dll!WaitForMultipleObjectsEx+0x12c
mscorwks.dll!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
mscorwks.dll!Thread::DoAppropriateAptStateWait+0x3c
mscorwks.dll!Thread::DoAppropriateWaitWorker+0x13c
mscorwks.dll!Thread::DoAppropriateWait+0x40
mscorwks.dll!WaitHandleNative::CorWaitOneNative+0x156
mscorlib.ni.dll+0x1f68af
mscorlib.ni.dll+0x1caa17
WindowsBase.ni.dll+0x24ac34
WindowsBase.ni.dll+0x2aeb1e
WindowsBase.ni.dll+0x9445d
WindowsBase.ni.dll+0x9267f
mscorwks.dll!JITutil_IsInstanceOfAny+0x106
mscorlib.ni.dll+0x1e842f
mscorwks.dll!CallDescrWorker+0x33
mscorwks.dll!CallDescrWorkerWithHandler+0xa3
mscorwks.dll!MethodDesc::CallDescr+0x19c
mscorwks.dll!MethodDesc::CallTargetWorker+0x1f
mscorwks.dll!MethodDescCallSite::Call+0x1a
mscorwks.dll!ExecuteCodeWithGuaranteedCleanupHelper+0x9f
mscorwks.dll!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0x10f
mscorlib.ni.dll+0x235677
mscorlib.ni.dll+0x2202a5
mscorlib.ni.dll+0x1e839b
mscorwks.dll!CallDescrWorker+0x33
mscorwks.dll!CallDescrWorkerWithHandler+0xa3
mscorwks.dll!DispatchCallBody+0x1e
mscorwks.dll!DispatchCallDebuggerWrapper+0x3d
mscorwks.dll!DispatchCallNoEH+0x51
mscorwks.dll!AddTimerCallback_Worker+0x66
mscorwks.dll!Thread::DoADCallBack+0x32a
mscorwks.dll!Thread::ShouldChangeAbortToUnload+0xe3
mscorwks.dll!Thread::ShouldChangeAbortToUnload+0x30a
mscorwks.dll!Thread::ShouldChangeAbortToUnload+0x33e
mscorwks.dll!ManagedThreadBase::ThreadPool+0x13
mscorwks.dll!AddTimerCallbackEx+0x83
mscorwks.dll!AddTimerCallback+0x10
mscorwks.dll!ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
mscorwks.dll!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x9a
mscorwks.dll!ThreadpoolMgr::ExecuteWorkRequest+0xaf
mscorwks.dll!ThreadpoolMgr::WorkerThreadStart+0x20b
mscorwks.dll!Thread::intermediateThreadProc+0x49
KERNEL32.dll!BaseThreadStart+0x37
Application uses System.Threading.Timer to periodically poll for data from several webservices using a WCF client proxy and at any giving time, could be making about 20 of these requests at the same time. Each call to a webservice instantiates a new proxy instance but the client is always closed when a response is received from the webservice.
Application also manipulates with bitmaps for a GIS and this is also done on a periodic interval. No where in code am I explicitly creating threads besides localized usage of the Timer class to poll for data periodically. The GIS does
use the BackgroundWorker but they do limit the thread count.
Anyone have an idea on what is spawning these new threads and why they are not being disposed?
TIA.
Yeah, looks like something you shouldn't ignore. They are the threadpool thread that the Timer class uses to make the callback. They are deadlocked, looks like they are waiting for a method call that's marshaled by COM to complete. There should be another thread in your program, one of the other 10 on which you created the GIS object. That thread is not pumping a message loop, a hard requirement for an STA thread that creates single apartment threaded COM components. Or it is stuck itself, not re-entering the message loop. Getting a managed stack trace ought to make it easier to see where the thread is stuck.
Trying to use threads on a COM object that explicitly doesn't support them (very few do) is pointless. Be sure to create the GIS object on your program's main UI thread. And use a DispatcherTimer. Creating your own STA thread that pumps a message loop can be a solution when the GIS component is taking too much of a hit on your user interface.
Problem was not GIS related but was due to System.Threading.Timer call backs queuing up as they were being called faster than they could be actioned. Problem was further compounded by the fact that timer call back was doing a Dispatcher.Invoke to the main user interface thread to refresh data grids. Dispatcher.Invoke was blocking when main user interface was busy, say when user was panning and zooming on map, and before timer call back got a chance to complete it was invoked again.
Fixed problem by temporarily stopping timer when it starts executing the call back and restarting it again after call back is complete. Also, instead of doing a Dispatcher.Invoke to get my grids updated, I changed it to a Dispatcher.BeginInvoke to avoid blocking.

Resources