Apache Flink, what is serving delay in event stream? - apache-flink

Question

I have read few articles on Flink and while reading a blog on Flink I came across the phrase
"with at most 60 seconds serving delay (events are out of order by max. 1 minute) "
Is defining out of order events duration used for the technique "Watermarks" in Flink and if not then whats the internal purpose?

Answer 1

I'll try to briefly explain how to manage out of order events in Flink. Event time, out of order and watermarks are very close concepts, and I think you will understand that phrase better after you understand their relation.
Watermarks and out of orderness are concepts of event time based DataStreams. A watermark can be described as a time mark where you assume there aren't going to occur more events before the mark. There are several mechanisms to emit watermarks in Flink, i.e, you can set a watermark each time you receive an event. Also, time windows use the watermarks to check when is the right time to evaluate.
That said, the "watermarks" and "out of order" concepts are essentially the same, as you use the watermark to achieve that out of order management. In your case, to define that 60 second max delay it's as simple as setting the watermark 60 seconds before the max timestamp received.
There is a nice example on the official site about managing out of order events:
/**
* This generator generates watermarks assuming that elements come out of order to a certain degree only.
* The latest elements for a certain timestamp t will arrive at most n milliseconds after the earliest
* elements for timestamp t.
*/
public class BoundedOutOfOrdernessGenerator extends AssignerWithPeriodicWatermarks<MyEvent> {
private final long maxOutOfOrderness = 3500; // 3.5 seconds
private long currentMaxTimestamp;
#Override
public long extractTimestamp(MyEvent element, long previousElementTimestamp) {
long timestamp = element.getCreationTime();
currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp);
return timestamp;
}
#Override
public Watermark getCurrentWatermark() {
// return the watermark as current highest timestamp minus the out-of-orderness bound
return new Watermark(currentMaxTimestamp - maxOutOfOrderness);
}
}