Apache Flink TaskManager stuck on terminating - apache-flink

Question

The taskmanager could not terminating and remained in a hung state, so the systemd considered that it was running and did not restart it
{"time":"2022-05-23 12:46:42.189","loglevel":"ERROR","class":"org.apache.flink.runtime.taskexecutor.TaskManagerRunner","message":"Fatal error occurred while executing the TaskManager. Shutting it down...","host":"flink15"}
org.apache.flink.util.FlinkRuntimeException: Task did not exit gracefully within 764 + seconds.
at org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1791)
at java.lang.Thread.run(Thread.java:750)
{"time":"2022-05-23 12:46:52.201","loglevel":"ERROR","class":"org.apache.flink.runtime.taskexecutor.TaskManagerRunner","message":"Terminating TaskManagerRunner with exit code 1.","host":"flink15"}
org.apache.flink.util.FlinkException: Unexpected failure during runtime of TaskManagerRunner.
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:394)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerProcessSecurely$3(TaskManagerRunner.java:428)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:428)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:408)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:366)
Caused by: java.util.concurrent.TimeoutException: null
at org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1237)
at org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
at org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$15(FutureUtils.java:591)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{"time":"2022-05-23 12:46:52.203","loglevel":"INFO","class":"org.apache.flink.runtime.blob.PermanentBlobCache","message":"Shutting down BLOB cache","host":"flink15"}
{"time":"2022-05-23 12:46:52.204","loglevel":"INFO","class":"org.apache.flink.runtime.blob.TransientBlobCache","message":"Shutting down BLOB cache","host":"flink15"}
{"time":"2022-05-23 12:46:52.205","loglevel":"INFO","class":"org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager","message":"Shutting down TaskExecutorLocalStateStoresManager.","host":"flink15"}
{"time":"2022-05-23 12:46:52.205","loglevel":"INFO","class":"org.apache.flink.runtime.filecache.FileCache","message":"removed file cache directory /var/lib/flink/tmp/flink-dist-cache-436fc230-8c11-45fd-83b2-43f0105dd524","host":"flink15"}
{"time":"2022-05-23 12:46:52.206","loglevel":"INFO","class":"org.apache.flink.runtime.io.disk.FileChannelManagerImpl","message":"FileChannelManager removed spill file directory /var/lib/flink/tmp/flink-io-f2b13865-538e-4369-9143-186d783aca9a","host":"flink15"}
{"time":"2022-05-23 12:46:52.206","loglevel":"INFO","class":"org.apache.flink.runtime.io.disk.FileChannelManagerImpl","message":"FileChannelManager removed spill file directory /var/lib/flink/tmp/flink-netty-shuffle-de83c8a0-9806-4a4e-b40e-b9b481a31f40","host":"flink15"}
but java process continued to run on the system, and when we manually restarted taskmanager systemd forced it to kill
May 23 15:20:48 flink15 systemd[1]: Stopping Apache Flink Taskmanager...
May 23 15:20:48 flink15 taskmanager.sh[10486]: Stopping taskexecutor daemon (pid: 60193) on host flink15.
May 23 15:20:58 flink15 taskmanager.sh[10486]: Daemon taskexecutor didn't stop within 10 seconds. Killing it.
May 23 15:21:01 flink15 systemd[1]: flink-taskmanager.service: Main process exited, code=killed, status=9/KILL
May 23 15:21:01 flink15 systemd[1]: Stopped Apache Flink Taskmanager.
May 23 15:21:01 flink15 systemd[1]: flink-taskmanager.service: Unit entered failed state.
is there any way to avoid this problem? so that taskmanager is guaranteed to exit in case of errors?