Flink Lambda serialization error logged when running in standalone server - apache-flink

Question

Version: Flink 1.12, java 1.11.
No issues while runnning in local environment. When running in standalone cluster with below config
INFO [] - Loading configuration property: jobmanager.rpc.address, localhost
INFO [] - Loading configuration property: jobmanager.rpc.port, 6123
INFO [] - Loading configuration property: taskmanager.cpu.cores, 1.79
INFO [] - Loading configuration property: taskmanager.memory.task.heap.size, 4096m
INFO [] - Loading configuration property: taskmanager.memory.task.off-heap.size, 4096m
INFO [] - Loading configuration property: taskmanager.memory.managed.size, 128m
INFO [] - Loading configuration property: taskmanager.memory.network.min, 64m
WARN [] - Error while trying to split key and value in configuration file /Users/vgamini/tools/flink-1.12.0/conf/flink-conf.yaml:44: "taskmanager.memory.network.max:64m"
INFO [] - Loading configuration property: jobmanager.memory.flink.size, 4096m
INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
INFO [] - Loading configuration property: parallelism.default, 1
I am seeing the below error in task manager logs - but the job went to running state
2020-12-29 15:16:31,322 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: Custom Source (1/3)#0 (5c850a62dc24ac6ccea8da166d5cc8f6) switched from DEPLOYING to FAILED.
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate outputs in order.
at org.apache.flink.streaming.api.graph.StreamConfig.getOutEdgesInOrder(StreamConfig.java:470) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriters(StreamTask.java:1138) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriterDelegate(StreamTask.java:1122) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:290) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:277) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:73) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:69) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1373) [flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:700) [flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547) [flink-dist_2.12-1.12.0.jar:1.12.0]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.keySelector of type org.apache.flink.api.java.functions.KeySelector in instance of org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2190) ~[?:?]
at java.io.ObjectStreamClass$FieldReflector.checkObjectFieldValueTypes(ObjectStreamClass.java:2153) ~[?:?]
at java.io.ObjectStreamClass.checkObjFieldValueTypes(ObjectStreamClass.java:1407) ~[?:?]
at java.io.ObjectInputStream.defaultCheckFieldValues(ObjectInputStream.java:2371) ~[?:?]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2278) ~[?:?]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2087) ~[?:?]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594) ~[?:?]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2355) ~[?:?]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2249) ~[?:?]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2087) ~[?:?]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594) ~[?:?]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430) ~[?:?]
at java.util.ArrayList.readObject(ArrayList.java:928) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1160) ~[?:?]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2216) ~[?:?]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2087) ~[?:?]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594) ~[?:?]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430) ~[?:?]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:576) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:562) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:550) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:511) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.streaming.api.graph.StreamConfig.getOutEdgesInOrder(StreamConfig.java:467) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
Is it just a warning? or will there be any impact because of this error.
Usecase:
Take a kafka input stream and key by a field to apply window grouping.
Note: While creating the fat jar - all the recommendations has been followed.
Do anyone faced the same issue?
The execution shows 2 tasks while the logs show tasks as 1/3. Can somebody help understanding it?

Answer 1

This problem occurs when you run Flink in Application Mode, but only your JobManager has the job jar in its classpath (or the usrlib/ folder).
To fix this issue, make the job jar available on your TaskManger instances as well.
You can do this by either building a custom docker image, containing the required jar(s), or by mounting them.
The exception in Flink 1.13.1 looked like this for me:
2021-08-27 11:44:45,788 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Job 857fbf37156b20a8d445b9e3d9a465ab reached terminal state FAILED.
org.apache.flink.util.SerializedThrowable: Recovery is suppressed by NoRestartBackoffTimeStrategy
at org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.howToHandleFailure(AdaptiveScheduler.java:1073) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.scheduler.adaptive.Executing.handleAnyFailure(Executing.java:88) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.scheduler.adaptive.Executing.updateTaskExecutionState(Executing.java:114) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.lambda$updateTaskExecutionState$4(AdaptiveScheduler.java:471) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.scheduler.adaptive.State.tryCall(State.java:142) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.updateTaskExecutionState(AdaptiveScheduler.java:468) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_302]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_302]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_302]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_302]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.11-1.13.1.jar:1.13.1]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.11-1.13.1.jar:1.13.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) [flink-dist_2.11-1.13.1.jar:1.13.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.13.1.jar:1.13.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.11-1.13.1.jar:1.13.1]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.11-1.13.1.jar:1.13.1]
Caused by: org.apache.flink.util.SerializedThrowable: Could not instantiate outputs in order.
at org.apache.flink.streaming.api.graph.StreamConfig.getOutEdgesInOrder(StreamConfig.java:485) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriters(StreamTask.java:1338) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriterDelegate(StreamTask.java:1322) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:327) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:308) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:76) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:72) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_302]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_302]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_302]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_302]
at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1524) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:730) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_302]
Caused by: org.apache.flink.util.SerializedThrowable: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.keySelector of type org.apache.flink.api.java.functions.KeySelector in instance of org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301) ~[?:1.8.0_302]
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431) ~[?:1.8.0_302]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2411) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) ~[?:1.8.0_302]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) ~[?:1.8.0_302]
at java.util.ArrayList.readObject(ArrayList.java:799) ~[?:1.8.0_302]
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_302]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_302]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) ~[?:1.8.0_302]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) ~[?:1.8.0_302]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.api.graph.StreamConfig.getOutEdgesInOrder(StreamConfig.java:482) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriters(StreamTask.java:1338) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriterDelegate(StreamTask.java:1322) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:327) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:308) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:76) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:72) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_302]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_302]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_302]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_302]
at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1524) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:730) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.11-1.13.1.jar:1.13.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_302]