In Flink job I have a KeyedProcessFunction.
I have implemented a watermark strategy
val wmStrategy: WatermarkStrategy<MyInput> =
WatermarkStrategy.forMonotonousTimestamps<MyInput>()
.withTimestampAssigner { event: MyInput, _: Long -> event.getTimestampEvent() }
and then i apply it to my source data:
mysource.assignTimestampsAndWatermarks(wmStrategy)
When processElement is called a timer may be registered ctx.timerService().registerEventTimeTimer(timerWakeUpInstant.toEpochMilli()) and after that the ValueState is updated. Update is successful.
The next time processElement is called, valueState.value() returns null instead of the last updated value.
No clear() is called explicitly on the value state.
The timer is never triggered.
At the moment, I'm testing in a 'clean' environment, reading from a text file with data referring to only a key, and with parallelism = 1 running into my IDE.
Can you help me? Why the state is nullified? And why timer is not triggered?
I have tried myself: OnTimer is not called until the Function that has registered the timer receives a message that advences the watermark.
With event-time timers, the onTimer(...) method is called when the current watermark is advanced up to or beyond the timestamp of the time
The "current" watermark actually refers to the operator, and not the job. This was misleading for me, as i thought it was centralized.
Looking at some code sample in the documentation we can find a useful comment that may give us a hint:
//trigger event time timers by advancing the event time of the operator with a watermark
Related
Do timers in Flink get fired if they are set to a timestamp in the past? Since the current timestamp is greater than that of the timer service, would it get fired immediately or never get fired?
Also, we are trying to sort/order input events based on the event time by collecting/buffering them in a processing time based tumbling window just so we don't have to drop late events. Are there are any better solutions to address this?
Timers set to a timestamp in the past get triggered ASAP.
For sorting, see How to sort an out-of-order event time stream using Flink and How to sort a stream by event time using Flink SQL.
The example is very useful at first,it illustrates how keyedProcessFunction is working in Flink
there is something worth noticing, it suddenly came to me...
It is from Fraud Detector v2: State + Time part
It is reasonable to set a timer here, regarding the real application requirement part
override def onTimer(
timestamp: Long,
ctx: KeyedProcessFunction[Long, Transaction, Alert]#OnTimerContext,
out: Collector[Alert]): Unit = {
// remove flag after 1 minute
timerState.clear()
flagState.clear()
}
Here is the problem:
The TimeCharacteristic IS ProcessingTime which is determined by the system clock of the running machine, according to ProcessingTime property, the watermark will NOT be changed overtime, so that means onTimer will never be called, unless the TimeCharacteristic changes to eventTime
According the flink website:
An hourly processing time window will include all records that arrived at a specific operator between the times when the system clock indicated the full hour. For example, if an application begins running at 9:15am, the first hourly processing time window will include events processed between 9:15am and 10:00am, the next window will include events processed between 10:00am and 11:00am, and so on.
If the watermark doesn't change over time, will the window function be triggered? because the condition for a window to be triggered is when the watermark enters the end time of a window
I'm wondering the condition where the window is triggered or not doesn't depend on watermark in priocessingTime, even though the official website doesn't mention that at all, it will be based on the processing time to trigger the window
Hope someone can spend a little time on this,many thx!
Let me try to clarify a few things:
Flink provides two kinds of timers: event time timers, and processing time timers. An event time timer is triggered by the arrival of a watermark equal to or greater than the timer's timestamp, and a processing time timer is triggered by the system clock reaching the timer's timestamp.
Watermarks are only relevant when doing event time processing, and only purpose they serve is to trigger event time timers. They play no role at all in applications like the one in this DataStream API Code Walkthrough that you have referred to. If this application used event time timers, either directly, or indirectly (by using event time windows, or through one of the higher level APIs like SQL or CEP), then it would need watermarks. But since it only uses processing time timers, it has no use for watermarks.
BTW, this fraud detection example isn't using Flink's Window API, because Flink's windowing mechanism isn't a good fit for this application's requirements. Here we are trying to a match a pattern to a sequence of events within a specific timeframe -- so we want a different kind of "window" that begins at the moment of a special triggering event (a small transaction, in this case), rather than a TimeWindow (like those provided by Flink's Window API) that is aligned to the clock (i.e., 10:00am to 10:01am).
I want to know exactly
When will watermark value be set as Long.MaxValue? (On canceling a SourceFunction? Cancel a job through cli & web-panel? ... )
What does it means for an application? (End of the job? Job failure? with no re/start?)
And how should it be handled? (clearing all the states? what about timers? As I saw registering a new timer on this state will make application to run forever! If I should be able to persist a state in last-watermark to recover from it in later time/run, how I should persist a timer-state?)
The last watermark is emitted when your SourceFunction exits the run method and it means you have consumed all input.
Given this you should not need to clear as the job will be marked as finished once the watermark reaches all sinks.
I have a stream of elements in an EventTime stream to a TumblingWindow to an AggregationFunction to a FlatMap which maintains state over aggregations and generates alarms
The elements are coming over a network and occasionally there is a delay in events so that the TumblingWindow does not close for some time. In these cases there can be elements buffered in the window that when combined with what had been collected earlier would generate an alarm. Is there a way to close a TumblingWindow early in such a scenario?
Is there a recommended pattern to follow for something like this.
I tried the following with no success ... should it have worked
I combined a ProcessFunction, a custom trigger and added a boolean flushSignal to the Element in the stream. I inserted the ProcessFunction upstream of the window and starting a processing time timer in it, from the processingElementMethod
long timeout = ctx.timerService().currentProcessingTime() + 10000;
ctx.timerService().registerProcessingTimeTimer(timeout);
and save the element that was passed in to be processed in the keyed state
When the timer expired in the onTimer method I retrieved the element from the keyedstate, set the flushSignal set to true and passed it to the onTimer collector.
In the trigger the intent was to in the onElement method return TriggerResult.FIRE.
However, the Element instance did not make it to the custom trigger. In the TumblingWindow.assignWindows method the timestamp is invalid.
Does the collector in ProcessFunction.onTimer need to be correlated with the collector in the ProcessFunction.processElement?
Note that I element stream has time characteristic eventTime and the timer I am trying to use is a ProcessingTimeTimer, is that a problem?
Any suggestions would be much appreciated.
Thanks
Jim
I am using Flink's TimeWindow functionality to perform some computations. I am creating a 5 minute Window. However I want to create a one hour Window for only the first time. The next Windows I need are of 5 minutes.
Such that for the first hour, data is collected and my operation is performed on it. Once this is done, every five minutes the same operation is performed.
I figure out this can be implemented with a trigger but I am not sure which trigger I should use and how.
UPDATE: I don't think even triggers are helpful, from what I can get, they just define the time/count triggering per window, not when the first window is to be triggered.
This is not trivial to implement.
Given a KeyedStream you have to use a GlobalWindow and a custom stateful Trigger which "remembers" whether is has fired for the first time or not.
val stream: DataStream[(String, Int)] = ???
val result = stream
.keyBy(0)
.window(GlobalWindows.create())
.trigger(new YourTrigger())
.apply(new YourWindowFunction())
Details about GlobalWindow and Trigger are in the Flink Window documentation.