Flink 1.15.1 Checkpoint Problem with gRPC - apache-flink

Question

I am trying to understand the Flink Checkpointing system (in PyFlink). This is why I created a playground for it. Here is my environment
env = StreamExecutionEnvironment.get_execution_environment()
config = Configuration(j_configuration=get_j_env_configuration(env._j_stream_execution_environment))
config.set_integer("python.fn-execution.bundle.size", 1000)
env.set_runtime_mode(RuntimeExecutionMode.STREAMING)
env.enable_checkpointing(3000)
env.get_checkpoint_config().set_checkpointing_mode(CheckpointingMode.AT_LEAST_ONCE)
env.get_checkpoint_config().set_checkpoint_storage(FileSystemCheckpointStorage('file:///home/ubuntu/checkpoints'))
env.get_checkpoint_config().set_fail_on_checkpointing_errors(True)
env.get_checkpoint_config().set_tolerable_checkpoint_failure_number(1)
and I use state with a simple map function as
class RandomRaiserMap(MapFunction):
def open(self, runtime_context: RuntimeContext):
state_desc = ValueStateDescriptor('cnt', Types.INT())
self.cnt_state = runtime_context.get_state(state_desc)
def map(self, value: Row):
cnt = self.cnt_state.value()
self.cnt_state.update(1 if cnt is None else cnt + 1)
if random.random() < 0.001:
raise Exception("Planted")
return Row(count=self.cnt_state.value(), **value.as_dict())
I used Kafka source (FlinkKafkaConsumer) as
deserialization_schema = JsonRowDeserializationSchema.builder() \
.type_info(Types.ROW_NAMED(["seq", "payload", "event_time"], [Types.INT(), Types.STRING(), Types.SQL_TIMESTAMP()])).build()
source = FlinkKafkaConsumer(
topics='test_checkpoint_sequence_2',
deserialization_schema=deserialization_schema,
properties={'bootstrap.servers': 'localhost:9092'}
)
Now, when start to execute job,
3> +I[1,273328,23fe377010c55a18,2022-08-02 12:36:50.325000]
3> +I[2,273329,79782a6af4a9badb,2022-08-02 12:36:50.336000]
3> +I[3,273330,d3537b0b9aedf6f9,2022-08-02 12:36:50.348000]
3> +I[4,273331,d600824835e4f04d,2022-08-02 12:36:50.359000]
3> +I[5,273332,e1e0851bfbe743ab,2022-08-02 12:36:50.370000]
3> +I[6,273333,27a547b6d4469ac0,2022-08-02 12:36:50.381000]
3> +I[7,273334,f12da6dbccbfc1be,2022-08-02 12:36:50.392000]
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 598, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 581, in _read_inputs
for elements in elements_iterator:
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
return self._next()
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/grpc/_channel.py", line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:38775 {grpc_message:"Multiplexer hanging up", grpc_status:1, created_time:"2022-08-02T12:36:53.274140888+00:00"}"
>
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 470, in input_elements
element = received.get(timeout=1)
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/queue.py", line 178, in get
raise Empty
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 289, in _execute
response = task()
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 362, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 606, in do_instruction
return getattr(self, request_type)(
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 644, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 988, in process_bundle
for element in data_channel.input_elements(instruction_id,
File "/home/ubuntu/miniconda3/envs/py38/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 473, in input_elements
raise RuntimeError('Channel closed prematurely.')
RuntimeError: Channel closed prematurely.
3> +I[188,273515,0fec22df999f7054,2022-08-02 12:36:52.362000]
3> +I[189,273516,9587bc4039b1b8d9,2022-08-02 12:36:52.373000]
3> +I[190,273517,e8e2d9dceb66ba15,2022-08-02 12:36:52.384000]
3> +I[191,273518,569c5513e994fa6b,2022-08-02 12:36:52.395000]
3> +I[192,273519,1f2809cd273204f9,2022-08-02 12:36:52.406000]
3> +I[193,273520,2f63560bdbeb2c96,2022-08-02 12:36:52.417000]
When a periodical checkpoint starts, I get this error. However, it manages to write a checkpoint on the given path (like 1f488f8d38f82d55cc342e1d0f676707) and if an error happens, it can recover as well. I am stuck with what the error can be.
I use
Ubuntu 22.04
Java 11
Python 3.8
Flink 1.15.1
apache-flink 1.15.1