In Flink, stream windowing does not seem to work? - apache-flink

Question

I tried to enhance the Flink example displaying the usage of streams.
My goal is to use the windowing features (see the window function call).
I assume that the code below outputs the sum of last 3 numbers of the stream.
(the stream is opened thanks to nc -lk 9999 on ubuntu)
Actually, the output sums up ALL numbers entered. Switching to a time window produces the same result, i.e. no windowing produced.
Is that a bug? (version used: latest master on github )
object SocketTextStreamWordCount {
def main(args: Array[String]) {
val hostName = args(0)
val port = args(1).toInt
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create streams for names and ages by mapping the inputs to the corresponding objects
val text = env.socketTextStream(hostName, port)
val currentMap = text.flatMap { (x:String) => x.toLowerCase.split("\\W+") }
.filter { (x:String) => x.nonEmpty }
.window(Count.of(3)).every(Time.of(1, TimeUnit.SECONDS))
// .window(Time.of(5, TimeUnit.SECONDS)).every(Time.of(1, TimeUnit.SECONDS))
.map { (x:String) => ("not used; just to have a tuple for the sum", x.toInt) }
val numberOfItems = currentMap.count
numberOfItems print
val counts = currentMap.sum( 1 )
counts print
env.execute("Scala SocketTextStreamWordCount Example")
}
}

Answer 1

The problem seems to be that there is an implicit conversion from WindowedDataStream to DataStream. This implicit conversion calls flatten() on the WindowedDataStream.
What happens in your case is that the code gets expanded to this:
val currentMap = text.flatMap { (x:String) => x.toLowerCase.split("\\W+") }
.filter { (x:String) => x.nonEmpty }
.window(Count.of(3)).every(Time.of(1, TimeUnit.SECONDS))
.flatten()
.map { (x:String) => ("not used; just to have a tuple for the sum",x.toInt) }
What flatten() does is similar to a flatMap() on a collection. It takes the stream of windows which can be seen as a collection of collections ([[a,b,c], [d,e,f]]) and turns it into a stream of elements: [a,b,c,d,e,f].
This means that your count really operates only on the original stream that has been windowed and "de-windowed". This looks like it has never been windowed at all.
This is a problem and I will work on fixing this right away. (I'm one of the Flink committers.) You can track the progress here: https://issues.apache.org/jira/browse/FLINK-2096
The way to do it with the current API is this:
val currentMap = text.flatMap { (x:String) => x.toLowerCase.split("\\W+") }
.filter { (x:String) => x.nonEmpty }
.map { (x:String) => ("not used; just to have a tuple for the sum",x.toInt) }
.window(Count.of(3)).every(Time.of(1, TimeUnit.SECONDS))
WindowedDataStream has a sum() method so there will be no implicit insertion of the flatten() call. Unfortunately, count() is not available on WindowedDataStream so for this you have to manually add a 1 field to the tuple and count these.