Flink adding rebalance to stream cause to job failure when StreamExecutionEnvironment set using TimeCharacteristic.IngestionTime - apache-flink

Question

I am trying to run streaming job that consumes messages from Kafka transforms them and sinks to the Cassandra.
Current code snippet is failing
val env: StreamExecutionEnvironment = getExecutionEnv("dev")
env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime)
.
.
.
.
val source = env.addSource(kafkaConsumer)
.uid("kafkaSource")
.rebalance
val transformedObjects = source.process(new EnrichEventWithIngestionTimestamp)
.setParallelism(dataSinkParallelism)
sinker.apply(transformedObjects,dataSinkParallelism)
class EnrichEventWithIngestionTimestamp extends ProcessFunction[RawData, TransforemedObjects] {
override def processElement(rawData: RawData,
context: ProcessFunction[RawData, TransforemedObjects]#Context,
collector: Collector[TransforemedObjects]): Unit = {
val currentTimestamp=context.timerService().currentProcessingTime()
context.timerService().registerProcessingTimeTimer(currentTimestamp)
collector.collect(TransforemedObjects.fromRawData(rawData,currentTimestamp))
}
}
but if rebalance is commented out, or the job is changed to use TimeCharacteristic.EventTime and watermark assignment, as in fallowing snippet, then it works.
val env: StreamExecutionEnvironment = getExecutionEnv("dev")
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
.
.
val source = env.addSource(kafkaConsumer)
.uid("kafkaSource")
.rebalance
.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessRawDataTimestampExtractor[RawData](Time.seconds(1)))
val transformedObjects = source.map(rawData=>TransforemedObjects.fromRawData(rawData))
.setParallelism(dataSinkParallelism)
sinker.apply(transformedObjects,dataSinkParallelism)
The stack trace is:
java.lang.Exception: java.lang.RuntimeException: 1
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:217)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.processInput(SourceStreamTask.java:133)
at org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: 1
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
at org.apache.flink.streaming.api.collector.selector.DirectedOutput.collect(DirectedOutput.java:143)
at org.apache.flink.streaming.api.collector.selector.DirectedOutput.collect(DirectedOutput.java:45)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:727)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:705)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$AutomaticWatermarkContext.processAndCollect(StreamSourceContexts.java:176)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$AutomaticWatermarkContext.processAndCollectWithTimestamp(StreamSourceContexts.java:194)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collectWithTimestamp(StreamSourceContexts.java:409)
at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:91)
at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:156)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:715)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:203)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.flink.runtime.io.network.api.writer.RecordWriter.getBufferBuilder(RecordWriter.java:246)
at org.apache.flink.runtime.io.network.api.writer.RecordWriter.copyFromSerializerToTargetChannel(RecordWriter.java:169)
at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:154)
at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:120)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)
... 16 more
Am I doing something wrong?
Or there Is a limitation of using rebalance function when TimeCharacteristic set to IngestionTime?
Thank you in advance...

Answer 1

Can you provide the flink version that you are using?
Seems your issue is related to this Jira ticket
https://issues.apache.org/jira/browse/FLINK-14087
Did u only use rebalance once in your task? The recordWriter may share same channelSelector which decides where the record will be forwarded to. Your stack trace shows it is trying to select an out-of-bound channel.