Does Flink handle out-of-order tuples even in case one does not use a windowing operator?
For example:
withTimestampsAndWatermarks
.keyBy(...)
.map(...) // some stateful function
.addSink(...);
Will map wait to process elements until receiving the correct watermark or will it process the elements without waiting?
The problem is that the partitioned state that map holds could be affected by the out-of-order processing of tuples.
Thank you in advance
The short answer is no. Map operator doesn't work with watermarks at all.
You will get elements in the same order as in the input stream.
For further reference check the implementation of StreamMap operator where you can see watermark elements are just forwarded to the output.
Github source code
Related
I am having some trouble understanding the way windowing is implemented internally in Flink and could not find any article which explain this in depth. In my mind, there are two ways this can be done. Consider a simple window wordcount code as below
env.socketTextStream("localhost", 9999)
.flatMap(new Splitter())
.groupBy(0)
.window(Time.of(500, TimeUnit.SECONDS)).sum(1)
Method 1: Store all events for 500 seconds and at the end of the window, process all of them by applying the sum operation on the stored events.
Method 2: We use a counter to store a rolling sum for every window. As each event in a window comes, we do not store the individual events but keep adding 1 to previously stored counter and output the result at the end of the window.
Could someone kindly help to understand which of the above methods (or maybe a different approach) is used by Flink in reality. The reason is, there are pros and cons to both approach and is important to understand in order configure the resources for the cluster correctly.
eg: The Method 1 seems very close to batch processing and might potentially have issues related to spike in processing at every 500 sec interval while sitting idle otherwise etc while Method2 would need to maintain a common counter between all task managers.
sum is a reducing function as mentioned here(https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#reducefunction). Internally, Flink will apply reduce function to each input element, and simply save the reduced result in ReduceState.
For other windows functions, like windows.apply(WindowFunction). There is no aggregation so all input elements will be saved in the ListState.
This document(https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#window-functions) about windows stream mentions about how the internal elements are handled in Flink.
We have using CEP already to manipulate some events. We have been using patterns from cep to correlate the events and produce meaningful outputs. For example we have the pattern sequence
that a.followedBy b followed by c. So far the events were coming in order {a1,b1,c1}, {a2,b2,c2} etc.
If now the sequence of the events change and even the sequence is not completed for example {a1,b1,a2,c1,b2,a3,b3,c3} or sometimes the sequence is not complete etc, is it possible to detect this and produce correct outputs {a1,b1,c1} and {a3,b3,c3};
I have tried to enhance the patterns using iterative condition but it seems the missing events break the matching and nothing is produced as ouput.
When you use keyBy with CEP, then each key-partitioned stream is matched independently. So if, for example, you key the stream {a1,b1,a2,c1,b2,a3,b3,c3} by the digits (1, 2, and 3) then CEP will separately apply the pattern to these three streams
{a1,b1,c1}
{a2,b2}
{a3,b3,c3}
and the two streams that are complete will match.
I'm newbie to Flink and I'm wondering what's the behavior of minBy (guess for the maxBy is the same) if there are multiple records that have the minimum value. I noticed that Flink will output only one record in this case, but which one? The first, the last or a random one?
Thanks for help.
Note that as of FLIP-134 all of these relational methods on DataStreams, namely Windowed/KeyedStream#sum,min,max,minBy,maxBy, are planned to be deprecated. The entire DataSet API is also planned to eventually be deprecated as well.
The only long-term support for relational methods like these is what is provided by the Table and SQL APIs.
But to answer your question, minBy and maxBy work the same way.
The javadoc for DataSet#maxBy says
If multiple values with maximum value at the specified fields exist, a random one will be
picked.
while the javadocs for AllWindowedStream#maxBy(int positionToMaxBy) and KeyedStream#maxBy(int positionToMaxBy) say
If more elements have the same maximum value the operator returns the first by default.
and the javadocs for AllWindowedStream#maxBy(int positionToMaxBy, boolean first) and AllWindowedStream#maxBy(int positionToMaxBy, boolean first) explain that
If [first is] true, then the operator return the first element with the maximum value,
otherwise returns the last
There are multiple tasks here. One of the tasks is BookingInfoWithFraudAndDefaultAndMainSP -> TSAndWMBookingWithSPObjects. Lets call it task-1. At task-1, I assign a timestamp and generate watermark, I am using BoundedOutOfOrdernessTimestampExtractor with maxOutOfOrderness equal to 2min.
The next operator is where I window the data and do some aggregations on top it which are then sinked to Kafka. Lets call this chained task of Aggregating and Sinking, Task-2.
numLateRecordsDropped: Looking at this metric which tells the The number of records this operator/task has dropped due to arriving late.
Question: When I window elements, i have assigned 0 allowed Lateness. So it could have dropped some elements. But when I look at the metrics, since window is not an operator, there is no metric which can tell how many elements are being dropped by windows.
When I look at task-2 metrics, it shows a count for numLateRecordsDropped. What does it mean. How can Window aggregation task drop records. Or since it is aggregating windows, the count basically is the number of records dropped by windows.
The Window operator is the only place where Flink uses numLateRecordsDropped (and furthermore, the window aggregation function runs in the window operator), so yes, the count is the number of records dropped by the window.
The javadoc for the DataStream#assignAscendingTimestamps
* Assigns timestamps to the elements in the data stream and periodically creates
* watermarks to signal event time progress.
*
* This method is a shortcut for data streams where the element timestamp are known
* to be monotonously ascending within each parallel stream.
* In that case, the system can generate watermarks automatically and perfectly
* by tracking the ascending timestamps.
This method assumes that the the element timestamp are known to be monotonously ascending within each parallel stream. But in practice, almost no stream can give such guarantee that event timestamps are in ascending order.
I would like to conclude that this method should never be used,but I would ask if I have missed something(eg, when to use it)
generally I agree, it can be rarely used in practice. An exception is the following: If Kafka is used as a source with LogAppendTime, timestamp are in order per-partition. You can then use per-partition watermarking in Flink [1] with the AscendingTimestampExtractor and will have pretty optimal watermarking.
Cheers,
Konstantin
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-consumers-and-timestamp-extractionwatermark-emission
After reading the source code DataStream#assignAscendingTimestamps, it is using AscendingTimestampExtractor to extract the timestamp.
AscendingTimestampExtractor will keep the largest event timestamp seen so far. If the event time is out of order, it will print a log to warn that monotonously ascending timestamps is violated.
So, I think this class may be handy in practice for the case that doesn't allow laziness(the watermark may keep growing).