Let's say we have an EventTimeSlidingWindow with an EventTime trigger based on some watermark. If the watermark is generated very infrequently, say every five minutes, and the window sizes are say one minute, then will five window results get fired at the same time when the watermark progresses? i.e. in my output stream would I have the same timestamp for all of their output, which is the time when the watermark generator produced the watermark?
Not exactly. Flink registers a timer with a timestamp when the window will fire. That timestamp is the event which has caused the window to open. As the watermark advances it will cause all the timers to fire (more or less at once) with the timestamps registered previously. So you are right with the assumption that all the windows will fire but having timestamps when the window (timer) was registered.
But be aware that if you "only" create a watermark every 5 minutes the minimum delay of the window will be that 5 minutes. But still the timestamps will be correct.
Related
My use case is to fire an event timer if the 2nd event did not come within 30 mins. Hence, I'll have 30 mins idless in between of the 2 events. My event timer triggers only if I send consequent events every atleast 1 sec, but does NOT if in between it's 30 mins gap, the onTimer never triggers. (Using PeriodicWatermarkAssigner)
How can I trigger if 30 mins gap is needed?
Event time timers are triggered by watermarks, which are only created from events. So if there are no events, there are no watermarks, and the timer will never fire. You'll need to use a processing time timer if the sources are completely idle.
When I use flink event time window, the window just doesn't trigger. How can I solve the problem, and are there any ways to debug?
As you are using the event time window, it is probably a watermark problem. The window only output when watermarks make a progress. There are some reasons why the event time has not been advanced:
There are no data from the source
One of the source parallelisms doesn't have data
The time field extracted from the record should be millisecond instead of second.
Data should cover a longer time span than the window size to advance the event time.
The window will output if we change event time to processing time. Furthermore, we can monitor event time by checking the watermarks in the web dashboard[1] or print-debug it with a ProcessFunction which can lookup the current watermark.
[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_event_time.html#monitoring-current-event-time
Be sure you're setting environment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime).
I am new to flink, and am trying to learn the Event Time and Watermarks section.
Can you explain what is Watermarks, and what problem it solves? The example is not clear to me.
does it only need for event time (out of order processing)?
The purpose of the watermark is to define when a time-based window should fire.
Watermarks allow for the idea that events might be slightly out of order, so the time "extracted" from it might differ by some amount from where you would like to draw the "low water" mark for firing that window. For example if your data is generated from disparate sources that have varied latency before arrived (consider situation of distributed logging).However, you might not need this if your data is guaranteed to only have ascending timestamps, for example if it is generated from sensor readings.
So this goes hand in hand with the some of the pre-defined Watermark generators that Flink provides which, not surprisingly, line up with the options.
The watermark and late event handling is easy to understand, but how about early event? For example, if the original stream contains events happened from 3:00 to 4:00, but if I insert some events which happened from 6:00 to 7:00 into the stream, then how flink handles them? It would create separate window(s) for them and when the window expires, they get handled too?
Depending on the watermarking strategy, early events can advance the watermark and then cause subsequent "on time" events to be considered late.
Early events are not dropped but put into the corresponding window. The window is processed when the watermark passes the end timestamp of the window. So, Flink is able to maintain several windows at the same time.
In my silverlight application I use a data grid. I have a column with session started time and a column with elapsed time. The elapsed time is "total minutes : seconds". In my model I have a property ElapsedTimeDisplay that compute and transform the elapsed time into a string.
How can update that each second?
Is there any possibility to update only that column?
Is there any other possibility of achieving this without using a trigger?
This depends on your setup, it could be as easy as adding a timer in your ViewModel (or maybe your model) that ticks every second, and raise the PropertyChanged event on the ElapsedTimeDisplay (which I suppose does the calculation in the getter). To do this, you need to have your ViewModel (or model) implement INotifyPropertyChanged.
Hope this helps :)