When I use flink event time window, the window just doesn't trigger. How can I solve the problem, and are there any ways to debug?
As you are using the event time window, it is probably a watermark problem. The window only output when watermarks make a progress. There are some reasons why the event time has not been advanced:
There are no data from the source
One of the source parallelisms doesn't have data
The time field extracted from the record should be millisecond instead of second.
Data should cover a longer time span than the window size to advance the event time.
The window will output if we change event time to processing time. Furthermore, we can monitor event time by checking the watermarks in the web dashboard[1] or print-debug it with a ProcessFunction which can lookup the current watermark.
[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_event_time.html#monitoring-current-event-time
Be sure you're setting environment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime).
Related
Let's say we have an EventTimeSlidingWindow with an EventTime trigger based on some watermark. If the watermark is generated very infrequently, say every five minutes, and the window sizes are say one minute, then will five window results get fired at the same time when the watermark progresses? i.e. in my output stream would I have the same timestamp for all of their output, which is the time when the watermark generator produced the watermark?
Not exactly. Flink registers a timer with a timestamp when the window will fire. That timestamp is the event which has caused the window to open. As the watermark advances it will cause all the timers to fire (more or less at once) with the timestamps registered previously. So you are right with the assumption that all the windows will fire but having timestamps when the window (timer) was registered.
But be aware that if you "only" create a watermark every 5 minutes the minimum delay of the window will be that 5 minutes. But still the timestamps will be correct.
There are a lot of late events in my Flink job so set allowedLateness() to 10mins (using TumblingEventTimeWindows and a complex AggregateFunction runs on every window)
Seems the aggregation happens on every late event but I'd like to fire less frequently.
Is there any trigger which fires only in every minute?
Do the triggers affect to late events?
Are there any triggers which effect only to the late events?
You could implement a custom Trigger with whatever behavior you desire.
If you look at the implementation of EventTimeTrigger, the default trigger for tumbling event time windows,
public TriggerResult onElement(Object element, long timestamp, TimeWindow window, TriggerContext ctx) throws Exception {
if (window.maxTimestamp() <= ctx.getCurrentWatermark()) {
// if the watermark is already past the window fire immediately
return TriggerResult.FIRE;
} else {
ctx.registerEventTimeTimer(window.maxTimestamp());
return TriggerResult.CONTINUE;
}
}
you'll see that whenever an event is assigned to the window after the stream's watermark has reached or surpassed the window's end, then the trigger returns FIRE. This is why every late event causes another firing.
An alternative would be to have no allowed lateness, but instead collect the late events into their own stream (using a side output), and then process the late events independently.
Just to clarify, the late events I mention below, are those late events still in the range of allowlatenes you set.
Is there any trigger which fires only in every minute ?
No. However you can customize your own Trigger, try using the event timer service to achieve that.
Do the triggers affect to late events ?
Yes. The late events will be referenced in trigger by calling onElement function.
Are there any triggers which effect only to the late events ?
You can filter the late events in custom trigger like this:
if (window.maxTimestamp() <= ctx.getCurrentWatermark()) {
return TriggerResult.FIRE;
In my opinion the best approach is to implement processes function with custom watermark.
I am new to flink, and am trying to learn the Event Time and Watermarks section.
Can you explain what is Watermarks, and what problem it solves? The example is not clear to me.
does it only need for event time (out of order processing)?
The purpose of the watermark is to define when a time-based window should fire.
Watermarks allow for the idea that events might be slightly out of order, so the time "extracted" from it might differ by some amount from where you would like to draw the "low water" mark for firing that window. For example if your data is generated from disparate sources that have varied latency before arrived (consider situation of distributed logging).However, you might not need this if your data is guaranteed to only have ascending timestamps, for example if it is generated from sensor readings.
So this goes hand in hand with the some of the pre-defined Watermark generators that Flink provides which, not surprisingly, line up with the options.
The watermark and late event handling is easy to understand, but how about early event? For example, if the original stream contains events happened from 3:00 to 4:00, but if I insert some events which happened from 6:00 to 7:00 into the stream, then how flink handles them? It would create separate window(s) for them and when the window expires, they get handled too?
Depending on the watermarking strategy, early events can advance the watermark and then cause subsequent "on time" events to be considered late.
Early events are not dropped but put into the corresponding window. The window is processed when the watermark passes the end timestamp of the window. So, Flink is able to maintain several windows at the same time.
I am working on Silverlight application and I want to make Image mousedouble click event,but there is no inbuilt mousedouble click in image control so I can do this..
Thanks...!!
The key to accomplishing this is to check for two things:
Measure the TimeSpan between two mouse clicks. Verify it is less than around 300 milliseconds.
Make certain the mouse has not moved more than a few pixels.
Try this http://www.michaelsnow.com/2010/05/10/silverlight-tip-of-the-day-17-double-click/
Regards.
The simplest way is to use a timer.
In the click event handler you have the following cases:
No timer. This is the first click, so start the timer and store the cursor location if necessary.
Timer running. This is potentially the second click. If the cursor hasn't moved then perform the single click action. Stop the timer.
The timer should stop itself after the double click interval (500 ms say) has expired.