Call to RocksDB.get times out when FlinkSQL job restarts - apache-flink

Question

We are running a streaming FlinkSQL job. If the job restarts (say due to a checkpoint failure), task managers log this error (see full stack trace below):
did not react to cancelling signal - interrupting; it is stuck for 30 seconds in method:
app//org.rocksdb.RocksDB.get(Native Method)
We see from the logs that rocksdb is closed exactly 30 seconds earlier:
Closed RocksDB State Backend. Cleaning up RocksDB working directory /flink-tmp/rocksdb/job_90331a967b94c5abd7b5377a55cc67ac_op_SlicingWindowOperator_4808b9a6cd8a2889e00c15fe1a792329__17_50__uuid_eb1b282c-8952-4b9f-b6d6-ee7be011d59f.
Is closing rocksdb causing the get operation to not return?
We are using Flink 1.15.0, and running a query like this:
INSERT INTO BigtableTable
SELECT CONCAT_WS('#', user_id, bucket) as rowkey, cell_timestamp, ROW(hllAttributeCount)
FROM (
SELECT
user_id,
window_end as cell_timestamp,
DATE_FORMAT(window_end, 'yyyy-MM-dd:HH') AS bucket,
STRING_HLL(attribute_to_count) AS hllAttributeCount
FROM TABLE(TUMBLE(TABLE inputTable, DESCRIPTOR(event_time), INTERVAL '5' MINUTES))
GROUP BY user_Id, window_start, window_end)
Full stack trace:
Task 'GlobalWindowAggregate[5] -> Calc[6] -> Sink: table[7] (4/50)#0' did not react to cancelling signal - interrupting; it is stuck for 30 seconds in method:
app//org.rocksdb.RocksDB.get(Native Method)
app//org.rocksdb.RocksDB.get(RocksDB.java:2084)
app//org.apache.flink.contrib.streaming.state.RocksDBValueState.value(RocksDBValueState.java:83)
app//org.apache.flink.table.runtime.operators.window.state.WindowValueState.value(WindowValueState.java:44)
app//org.apache.flink.table.runtime.operators.aggregate.window.combines.GlobalAggCombiner.combineAccumulator(GlobalAggCombiner.java:94)
app//org.apache.flink.table.runtime.operators.aggregate.window.combines.GlobalAggCombiner.combine(GlobalAggCombiner.java:85)
app//org.apache.flink.table.runtime.operators.aggregate.window.buffers.RecordsWindowBuffer.flush(RecordsWindowBuffer.java:112)
app//org.apache.flink.table.runtime.operators.aggregate.window.processors.AbstractWindowAggProcessor.prepareCheckpoint(AbstractWindowAggProcessor.java:203)
app//org.apache.flink.table.runtime.operators.window.slicing.SlicingWindowOperator.prepareSnapshotPreBarrier(SlicingWindowOperator.java:267)
app//org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.prepareSnapshotPreBarrier(RegularOperatorChain.java:89)
app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:300)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$12(StreamTask.java:1253)
app//org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$1997/0x0000000840efa440.run(Unknown Source)
app//org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1241)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1198)
app//org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)
app//org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)
app//org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)
app//org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:493)
app//org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74)
app//org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66)
app//org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)
app//org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$$Lambda$2010/0x0000000840efd040.apply(Unknown Source)
app//org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)
app//org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)
app//org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
app//org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
app//org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
app//org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
app//org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$1068/0x00000008409dfc40.runDefaultAction(Unknown Source)
app//org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
app//org.apache.flink.runtime.taskmanager.Task$$Lambda$1951/0x0000000840e47840.run(Unknown Source)
app//org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
app//org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
app//org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
app//org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
java.base#11.0.15/java.lang.Thread.run(Unknown Source)