My use case is to fire an event timer if the 2nd event did not come within 30 mins. Hence, I'll have 30 mins idless in between of the 2 events. My event timer triggers only if I send consequent events every atleast 1 sec, but does NOT if in between it's 30 mins gap, the onTimer never triggers. (Using PeriodicWatermarkAssigner)
How can I trigger if 30 mins gap is needed?
Event time timers are triggered by watermarks, which are only created from events. So if there are no events, there are no watermarks, and the timer will never fire. You'll need to use a processing time timer if the sources are completely idle.
Related
Let's say we have an EventTimeSlidingWindow with an EventTime trigger based on some watermark. If the watermark is generated very infrequently, say every five minutes, and the window sizes are say one minute, then will five window results get fired at the same time when the watermark progresses? i.e. in my output stream would I have the same timestamp for all of their output, which is the time when the watermark generator produced the watermark?
Not exactly. Flink registers a timer with a timestamp when the window will fire. That timestamp is the event which has caused the window to open. As the watermark advances it will cause all the timers to fire (more or less at once) with the timestamps registered previously. So you are right with the assumption that all the windows will fire but having timestamps when the window (timer) was registered.
But be aware that if you "only" create a watermark every 5 minutes the minimum delay of the window will be that 5 minutes. But still the timestamps will be correct.
There are a lot of late events in my Flink job so set allowedLateness() to 10mins (using TumblingEventTimeWindows and a complex AggregateFunction runs on every window)
Seems the aggregation happens on every late event but I'd like to fire less frequently.
Is there any trigger which fires only in every minute?
Do the triggers affect to late events?
Are there any triggers which effect only to the late events?
You could implement a custom Trigger with whatever behavior you desire.
If you look at the implementation of EventTimeTrigger, the default trigger for tumbling event time windows,
public TriggerResult onElement(Object element, long timestamp, TimeWindow window, TriggerContext ctx) throws Exception {
if (window.maxTimestamp() <= ctx.getCurrentWatermark()) {
// if the watermark is already past the window fire immediately
return TriggerResult.FIRE;
} else {
ctx.registerEventTimeTimer(window.maxTimestamp());
return TriggerResult.CONTINUE;
}
}
you'll see that whenever an event is assigned to the window after the stream's watermark has reached or surpassed the window's end, then the trigger returns FIRE. This is why every late event causes another firing.
An alternative would be to have no allowed lateness, but instead collect the late events into their own stream (using a side output), and then process the late events independently.
Just to clarify, the late events I mention below, are those late events still in the range of allowlatenes you set.
Is there any trigger which fires only in every minute ?
No. However you can customize your own Trigger, try using the event timer service to achieve that.
Do the triggers affect to late events ?
Yes. The late events will be referenced in trigger by calling onElement function.
Are there any triggers which effect only to the late events ?
You can filter the late events in custom trigger like this:
if (window.maxTimestamp() <= ctx.getCurrentWatermark()) {
return TriggerResult.FIRE;
In my opinion the best approach is to implement processes function with custom watermark.
When I use flink event time window, the window just doesn't trigger. How can I solve the problem, and are there any ways to debug?
As you are using the event time window, it is probably a watermark problem. The window only output when watermarks make a progress. There are some reasons why the event time has not been advanced:
There are no data from the source
One of the source parallelisms doesn't have data
The time field extracted from the record should be millisecond instead of second.
Data should cover a longer time span than the window size to advance the event time.
The window will output if we change event time to processing time. Furthermore, we can monitor event time by checking the watermarks in the web dashboard[1] or print-debug it with a ProcessFunction which can lookup the current watermark.
[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_event_time.html#monitoring-current-event-time
Be sure you're setting environment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime).
The watermark and late event handling is easy to understand, but how about early event? For example, if the original stream contains events happened from 3:00 to 4:00, but if I insert some events which happened from 6:00 to 7:00 into the stream, then how flink handles them? It would create separate window(s) for them and when the window expires, they get handled too?
Depending on the watermarking strategy, early events can advance the watermark and then cause subsequent "on time" events to be considered late.
Early events are not dropped but put into the corresponding window. The window is processed when the watermark passes the end timestamp of the window. So, Flink is able to maintain several windows at the same time.
I am trying to use a WPF combo box to change a timer.
Currently, I have in an observable collection
15 mins
30 mins
45 mins.
1 hour.
If the user clicks 15 minutes, then a timer will start counting down.
But my problem is, once the user clicks 15 minutes, the combo box won't let me pick 15 minutes again to reset it.
I have to click on another item such as 30 minutes, then I can click 15 minutes.
This is using XAML, I was wondering if there is any quick fix to this.
Thanks!!
you could also handle the ComboBox DropDownClosed event to start your timer.
what event are you hooking onto to initiate your timer? i am guessing selecteditemchanged? if so then that will be the cause of the problem. try using the click event and get the selected item in your handler. if you post your code / xaml i will be able to give a better answer. you should be able to do what you are asking in xaml.