Infinite loop in Camel - Rabbit MQ - apache-camel

I have a small server-route, which takes a message from queue.in and puts in queue.out.
When I put ONE message in queue.in, the server runns in an infinite loop. I wonder what I'm missing in terms of configuration. This seems like a very simple route.
The small server-route:
<camelContext id="camel-server" xmlns="http://camel.apache.org/schema/spring">
<jmxAgent disabled="false" onlyRegisterProcessorWithCustomId="false"
createConnector="true" usePlatformMBeanServer="true"
registerNewRoutes="true" statisticsLevel="All" includeHostName="false"
mask="false" id="agent" />
<endpoint id="queue.in" uri="${queue.url.input_queue}" />
<endpoint id="queue.out" uri="${queue.url.output_queue}" />
<route id="rabbitRoute">
<from uri="ref:queue.in" />
<bean ref="multiplier" /><!-- a simple bean that appends "BeanHello" to input msg-->
<to uri="ref:queue.out" />
</route>
</camelContext>
The rabbit queue properties:
queue.url.input_queue=rabbitmq://localhost:5672/ex1?queue=input_queue&routingKey=input_queue&connectionFactory=#customConnectionFactory&autoDelete=false&autoAck=true
queue.url.output_queue=rabbitmq://localhost:5672/ex1?queue=output_queue&routingKey=output_queue&connectionFactory=#customConnectionFactory&autoDelete=false&autoAck=true
Snippet of log file - see recursive part at the end of log:
The string "BeanHello" is added multiple times, that means the message is going to queue.in again and again instead of going to queue.out.
2014-12-14 15:00:16,692 [main ] DEBUG DefaultComponentResolver - Found component: rabbitmq via type: org.apache.camel.component.rabbitmq.RabbitMQComponent via: META-INF/services/org/apache/camel/component/rabbitmq
2014-12-14 15:00:16,699 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=components,name="rabbitmq"
2014-12-14 15:00:16,711 [main ] DEBUG DefaultComponent - Creating endpoint uri=[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue], path=[localhost:5672/ex1]
2014-12-14 15:00:16,721 [main ] DEBUG IntrospectionSupport - Configured property: autoAck on bean: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue] with value: true
2014-12-14 15:00:16,725 [main ] DEBUG IntrospectionSupport - Configured property: autoDelete on bean: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue] with value: false
2014-12-14 15:00:16,726 [main ] DEBUG IntrospectionSupport - Configured property: queue on bean: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue] with value: input_queue
2014-12-14 15:00:16,727 [main ] DEBUG IntrospectionSupport - Configured property: routingKey on bean: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue] with value: input_queue
2014-12-14 15:00:16,728 [main ] DEBUG RabbitMQComponent - Creating RabbitMQEndpoint with host localhost:5672 and exchangeName: ex1
2014-12-14 15:00:16,729 [main ] DEBUG SpringCamelContext - rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue converted to endpoint: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue] by component: org.apache.camel.component.rabbitmq.RabbitMQComponent#2be25129
2014-12-14 15:00:16,730 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=endpoints,name="rabbitmq://localhost:5672/ex1\?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue"
2014-12-14 15:00:16,730 [main ] DEBUG DefaultManagementAgent - MBean already registered with ObjectName: org.apache.camel:context=camel-server,type=endpoints,name="rabbitmq://localhost:5672/ex1\?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue"
2014-12-14 15:00:16,743 [main ] DEBUG SpringCamelContext - Using ComponentResolver: org.apache.camel.impl.DefaultComponentResolver#1bd7643a to resolve component with name: bean
2014-12-14 15:00:16,743 [main ] DEBUG DefaultComponentResolver - Found component: bean in registry: null
2014-12-14 15:00:16,743 [main ] DEBUG DefaultComponentResolver - Found component: bean via type: org.apache.camel.component.bean.BeanComponent via: META-INF/services/org/apache/camel/component/bean
2014-12-14 15:00:16,748 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=components,name="bean"
2014-12-14 15:00:16,758 [main ] DEBUG DefaultChannel - Initialize channel for target: 'Bean[ref:multiplier]'
2014-12-14 15:00:16,765 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=tracer,name=BacklogTracer
2014-12-14 15:00:16,774 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=tracer,name=BacklogDebugger
2014-12-14 15:00:16,788 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=errorhandlers,name="DefaultErrorHandlerBuilder(ref:CamelDefaultErrorHandlerBuilder)"
2014-12-14 15:00:16,789 [main ] DEBUG DefaultComponent - Creating endpoint uri=[ref://queue.out], path=[queue.out]
2014-12-14 15:00:16,789 [main ] DEBUG SpringCamelContext - ref://queue.out converted to endpoint: Endpoint[ref://queue.out] by component: org.apache.camel.component.ref.RefComponent#2182fb16
2014-12-14 15:00:16,790 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=endpoints,name="ref://queue.out"
2014-12-14 15:00:16,792 [main ] DEBUG DefaultComponent - Creating endpoint uri=[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue], path=[localhost:5672/ex1]
2014-12-14 15:00:16,794 [main ] DEBUG IntrospectionSupport - Configured property: autoAck on bean: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue] with value: true
2014-12-14 15:00:16,795 [main ] DEBUG IntrospectionSupport - Configured property: autoDelete on bean: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue] with value: false
2014-12-14 15:00:16,797 [main ] DEBUG IntrospectionSupport - Configured property: queue on bean: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue] with value: output_queue
2014-12-14 15:00:16,798 [main ] DEBUG IntrospectionSupport - Configured property: routingKey on bean: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue] with value: output_queue
2014-12-14 15:00:16,798 [main ] DEBUG RabbitMQComponent - Creating RabbitMQEndpoint with host localhost:5672 and exchangeName: ex1
2014-12-14 15:00:16,800 [main ] DEBUG SpringCamelContext - rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue converted to endpoint: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue] by component: org.apache.camel.component.rabbitmq.RabbitMQComponent#2be25129
2014-12-14 15:00:16,801 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=endpoints,name="rabbitmq://localhost:5672/ex1\?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue"
2014-12-14 15:00:16,801 [main ] DEBUG DefaultManagementAgent - MBean already registered with ObjectName: org.apache.camel:context=camel-server,type=endpoints,name="rabbitmq://localhost:5672/ex1\?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue"
2014-12-14 15:00:16,802 [main ] DEBUG DefaultChannel - Initialize channel for target: 'To[ref:queue.out]'
2014-12-14 15:00:16,808 [main ] INFO SpringCamelContext - AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
2014-12-14 15:00:16,808 [main ] INFO SpringCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
2014-12-14 15:00:16,809 [main ] DEBUG SpringCamelContext - Warming up route id: rabbitRoute having autoStartup=true
2014-12-14 15:00:16,809 [main ] DEBUG RouteService - Starting services on route: rabbitRoute
2014-12-14 15:00:16,811 [main ] DEBUG RouteService - Starting child service on route: rabbitRoute -> Pipeline[[Channel[BeanProcessor[org.apache.camel.example.server.Treble(0x64eb29e1)]], Channel[sendTo(Endpoint[ref://queue.out])]]]
2014-12-14 15:00:16,811 [main ] DEBUG DefaultErrorHandler - Redelivery enabled: false on error handler: DefaultErrorHandler[Instrumentation:bean[BeanProcessor[org.apache.camel.example.server.Treble(0x64eb29e1)]]]
2014-12-14 15:00:16,818 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=producers,name=RabbitMQProducer(0x89c08d3)
2014-12-14 15:00:16,823 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=threadpools,name="RabbitMQProducer(0x89c08d3)"
2014-12-14 15:00:16,824 [main ] DEBUG DefaultExecutorServiceManager - Created new ThreadPool for source: Producer[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue] with name: CamelRabbitMQProducer[output_queue]. -> org.apache.camel.util.concurrent.RejectableThreadPoolExecutor#69e4d7d[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0][CamelRabbitMQProducer[output_queue]]
2014-12-14 15:00:16,890 [main ] DEBUG RabbitMQProducer - Created connection: amqp://guest#127.0.0.1:5672/
2014-12-14 15:00:16,911 [main ] DEBUG ProducerCache - Adding to producer cache with key: Endpoint[ref://queue.out] for producer: Producer[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=output_queue&routingKey=output_queue]
2014-12-14 15:00:16,912 [main ] DEBUG DefaultErrorHandler - Redelivery enabled: false on error handler: DefaultErrorHandler[Instrumentation:to[sendTo(Endpoint[ref://queue.out])]]
2014-12-14 15:00:16,912 [main ] DEBUG RouteService - Starting child service on route: rabbitRoute -> Pipeline[[Channel[BeanProcessor[org.apache.camel.example.server.Treble(0x64eb29e1)]], Channel[sendTo(Endpoint[ref://queue.out])]]]
2014-12-14 15:00:16,912 [main ] DEBUG RouteService - Starting child service on route: rabbitRoute -> BeanProcessor[org.apache.camel.example.server.Treble(0x64eb29e1)]
2014-12-14 15:00:16,919 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=processors,name="bean1"
2014-12-14 15:00:16,920 [main ] DEBUG RouteService - Starting child service on route: rabbitRoute -> Channel[BeanProcessor[org.apache.camel.example.server.Treble(0x64eb29e1)]]
2014-12-14 15:00:16,920 [main ] DEBUG RouteService - Starting child service on route: rabbitRoute -> sendTo(Endpoint[ref://queue.out])
2014-12-14 15:00:16,928 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=processors,name="to1"
2014-12-14 15:00:16,928 [main ] DEBUG RouteService - Starting child service on route: rabbitRoute -> Channel[sendTo(Endpoint[ref://queue.out])]
2014-12-14 15:00:16,939 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=routes,name="rabbitRoute"
2014-12-14 15:00:16,939 [main ] DEBUG TimerListenerManager - Added TimerListener: org.apache.camel.management.mbean.ManagedSuspendableRoute#19f7f0f0
2014-12-14 15:00:16,940 [main ] DEBUG SpringCamelContext - Route: rabbitRoute >>> EventDrivenConsumerRoute[Endpoint[ref://queue.in] -> Pipeline[[Channel[BeanProcessor[org.apache.camel.example.server.Treble(0x64eb29e1)]], Channel[sendTo(Endpoint[ref://queue.out])]]]]
2014-12-14 15:00:16,940 [main ] DEBUG SpringCamelContext - Starting consumer (order: 1000) on route: rabbitRoute
2014-12-14 15:00:16,945 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=consumers,name=RabbitMQConsumer(0x753af01f)
2014-12-14 15:00:16,947 [main ] DEBUG DefaultManagementAgent - Registered MBean with ObjectName: org.apache.camel:context=camel-server,type=threadpools,name="RabbitMQEndpoint(0xad9be15)"
2014-12-14 15:00:16,947 [main ] DEBUG DefaultExecutorServiceManager - Created new ThreadPool for source: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue] with name: RabbitMQConsumer. -> org.apache.camel.util.concurrent.RejectableThreadPoolExecutor#5d4e5a43[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0][RabbitMQConsumer]
2014-12-14 15:00:16,947 [main ] DEBUG RabbitMQConsumer - Using executor org.apache.camel.util.concurrent.RejectableThreadPoolExecutor#5d4e5a43[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0][RabbitMQConsumer]
2014-12-14 15:00:16,951 [main ] DEBUG RabbitMQConsumer - Created connection: amqp://guest#127.0.0.1:5672/
2014-12-14 15:00:16,952 [main ] DEBUG RabbitMQConsumer - Created channel: AMQChannel(amqp://guest#127.0.0.1:5672/,1)
2014-12-14 15:00:16,957 [main ] INFO SpringCamelContext - Route: rabbitRoute started and consuming from: Endpoint[rabbitmq://localhost:5672/ex1?autoAck=true&autoDelete=false&connectionFactory=%23customConnectionFactory&queue=input_queue&routingKey=input_queue]
2014-12-14 15:00:16,958 [main ] DEBUG ultManagementLifecycleStrategy - Load performance statistics disabled
2014-12-14 15:00:16,958 [main ] INFO SpringCamelContext - Total 1 routes, of which 1 is started.
2014-12-14 15:00:16,962 [main ] INFO SpringCamelContext - Apache Camel 2.15-SNAPSHOT (CamelContext: camel-server) started in 0.889 seconds
2014-12-14 15:00:16,963 [main ] DEBUG MainSupport - Starting Spring ApplicationContext: org.springframework.context.support.ClassPathXmlApplicationContext#3db853a5
2014-12-14 15:00:16,964 [main ] DEBUG SpringCamelContext - onApplicationEvent: org.springframework.context.event.ContextStartedEvent[source=org.springframework.context.support.ClassPathXmlApplicationContext#3db853a5: startup date [Sun Dec 14 15:00:14 PST 2014]; root of context hierarchy]
2014-12-14 15:01:01,671 [abbitMQConsumer] DEBUG MethodInfo - Setting bean invocation result on the IN message: ###MsgFromClient#BeanHello
2014-12-14 15:01:01,676 [abbitMQConsumer] DEBUG SendProcessor - >>>> Endpoint[ref://queue.out] Exchange[Message: ###MsgFromClient#BeanHello ]
2014-12-14 15:01:01,683 [abbitMQConsumer] DEBUG MethodInfo - Setting bean invocation result on the IN message: ###MsgFromClient#BeanHello #BeanHello
2014-12-14 15:01:01,684 [abbitMQConsumer] DEBUG SendProcessor - >>>> Endpoint[ref://queue.out] Exchange[Message: ###MsgFromClient#BeanHello #BeanHello ]
2014-12-14 15:01:01,687 [abbitMQConsumer] DEBUG MethodInfo - Setting bean invocation result on the IN message: ###MsgFromClient#BeanHello #BeanHello #BeanHello
2014-12-14 15:01:01,687 [abbitMQConsumer] DEBUG SendProcessor - >>>> Endpoint[ref://queue.out] Exchange[Message: ###MsgFromClient#BeanHello #BeanHello #BeanHello ]
2014-12-14 15:01:01,698 [abbitMQConsumer] DEBUG MethodInfo - Setting bean invocation result on the IN message: ###MsgFromClient#BeanHello #BeanHello #BeanHello #BeanHello
2014-12-14 15:01:01,698 [abbitMQConsumer] DEBUG SendProcessor - >>>> Endpoint[ref://queue.out] Exchange[Message: ###MsgFromClient#BeanHello #BeanHello #BeanHello #BeanHello ]

Resolution:
Changed the uri added BridgedEndpoint=true. I do not understand the mechanics behind that, and why such extra complications for rabbitmq specifically.
queue.url.input_queue=rabbitmq://localhost:5672/ex1?queue=input_queue&routingKey=input_queue&connectionFactory=#customConnectionFactory&autoDelete=false&autoAck=true&BridgeEndpoint=true
queue.url.output_queue=rabbitmq://localhost:5672/ex1?queue=output_queue&routingKey=output_queue&connectionFactory=#customConnectionFactory&autoDelete=false&autoAck=true&BridgeEndpoint=true

I had the same problem and it is fixed by adding between input and output (http://camel.apache.org/constant.html):
.setHeader("rabbitmq.ROUTING_KEY", constant("output_queue"))
According http://camel.apache.org/rabbitmq.html:
The routing key to use when binding a consumer queue to the exchange.
For producer routing keys, you set the header (see header section)

Related

Flink Standalone K8 - Cannot create HA service - NullPointerException

I currently have a Flink (1.12) cluster running in Standalone Kubernetes (v1.16).
For our purposes, we have went with an application cluster mode deployment.
To make our flink cluster more resilient to failures we want to add HA to our current setup, and I have gone through the documentation and followed the example configurations recommended for our given setup (here).
flink-conf.yaml
jobmanager.rpc.address: {{ $fullName }}-jobmanager
jobmanager.rpc.port: 6123
jobmanager.memory.process.size: 1600m
taskmanager.numberOfTaskSlots: 2
taskmanager.rpc.port: 6122
taskmanager.memory.process.size: 1728m
blob.server.port: 6124
queryable-state.proxy.ports: 6125
parallelism.default: 2
scheduler-mode: reactive
execution.checkpointing.interval: 10s
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.cluster-id: thoros-cluster-1
high-availability.storageDir: s3:///company-flink-{{ .Values.environment }}/recovery
job.yaml (excerpt)
...
restartPolicy: OnFailure
containers:
- name: jobmanager
image: "{{ .Values.thoros.image.repository }}:{{ .Chart.AppVersion }}"
imagePullPolicy: {{ default "Always" .Values.thoros.image.pullPolicy }}
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
envFrom:
- configMapRef:
name: {{ $fullName }}
# The following args overwrite the value of jobmanager.rpc.address configured in the configuration config map to POD_IP.
args: [
"standalone-job",
"--host",
"$(POD_IP)",
"--job-classname",
"com.company.beam.Main"]
There are of course a couple of other configurations I am leaving out (happy to provide those if needed)
To test, I have set the Job parallelism to 2 (which spins up two JobManagers, one of which should be standby)
When trying to deploy this to K8 the JobManager pods fail immediately with the following error - I am not sure what may be missing here aside from the fact that something seems to be missing hence causing the Nullpointerexception?
2021-08-20 12:06:55,133 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services.
2021-08-20 12:06:55,176 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 100.107.0.5:6123, bind address 0.0.0.0:6123.
2021-08-20 12:06:56,956 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2021-08-20 12:06:57,067 INFO akka.remote.Remoting [] - Starting remoting
2021-08-20 12:06:57,469 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink#100.107.0.5:6123]
2021-08-20 12:06:57,687 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink#100.107.0.5:6123
2021-08-20 12:06:58,671 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting StandaloneApplicationClusterEntryPoint down with application status FAILED. Diagnostics org.apache.flink.util.FlinkException: Could not create the ha services from the instantiated HighAvailabilityServicesFactory org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
Caused by: java.lang.NullPointerException
at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
at org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)q
at org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
... 9 more
.
2021-08-20 12:06:58,684 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopping Akka RPC service.
2021-08-20 12:06:58,754 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon.
2021-08-20 12:06:58,767 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports.
2021-08-20 12:06:58,833 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down.
2021-08-20 12:06:58,882 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopped Akka RPC service.
2021-08-20 12:06:58,882 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Could not start cluster entrypoint StandaloneApplicationClusterEntryPoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneApplicationClusterEntryPoint.
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:201) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585) [flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85) [flink-dist_2.12-1.12.5.jar:1.12.5]
Caused by: org.apache.flink.util.FlinkException: Could not create the ha services from the instantiated HighAvailabilityServicesFactory org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
... 2 more
Caused by: java.lang.NullPointerException
at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175) ~[flink-dist_2.12-1.12.5.jar:1.12.5]
This issue was due to using high-availability.cluster-id when it should be kubernetes.cluster-id.

Flink job always show "CREATED" status

I am trying to run Flink job in k8s environment, the cluster looks ok. I can see jobmanager and taskmanager are running well from UI. but when I tried to run a Flink job, UI shows that job is running, but the tasks status always keep "CREATED".
I made a GET request to get the metrics of that job and found that the task status is "SCHEDULED".
I have no idea that where the cluster have issue, anyone can give some indicators or suggestions on how to deal with it.
and i suspect that jobmanager can contact with taskmanager, but the taskmanager looks running well from UI. I also check that the resource of cluster, it's enough.
Need your help, thanks!!
my flink cluster(standalone) is running on kubernetes. One master pod, one worker pod, but none of them has any system level logs. I am not sure where can configure this. but when I try to execute a wordcount.jar example on it, it will show me below error logs:
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185)
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179)
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503)
at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49)
at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710)
at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287)
at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255)
at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086)
at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748)
at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435)
at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726)
at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537)
at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources.
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
... 47 more
Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
... 27 more
Caused by: java.util.concurrent.TimeoutException
... 25 more
taskmanager error log:
2020-09-04 09:09:53 DEBUG a.r.t.n.NettyTransport [] - Remote connection to [/192.168.3.147:55194] was disconnected because of [id: 0xcf013b4a, /192.168.3.147:55194 :> /172.16.0.210:6122] DISCONNECTED
2020-09-04 09:09:53 DEBUG a.r.t.ProtocolStateActor [] - Association between local [tcp://flink#172.16.0.210:6122] and remote [tcp://flink#192.168.3.147:55194] was disassociated because the ProtocolStateActor failed: Unknown
2020-09-04 09:10:00 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:10:10 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:10:20 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:10:30 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:10:40 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:10:51 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:10:53 DEBUG a.r.t.n.NettyTransport [] - Remote connection to [/192.168.3.147:56740] was disconnected because of [id: 0xeab6536f, /192.168.3.147:56740 :> /172.16.0.210:6122] DISCONNECTED
2020-09-04 09:10:53 DEBUG a.r.t.ProtocolStateActor [] - Association between local [tcp://flink#172.16.0.210:6122] and remote [tcp://flink#192.168.3.147:56740] was disassociated because the ProtocolStateActor failed: Unknown
2020-09-04 09:10:56 INFO o.a.f.r.t.TaskExecutor [] - Receive slot request c37da0fd74b14bd257a4ecce33c06d79 for job 69482630ce87466bb580bff416c284e5 from resource manager with leader id 00000000000000000000000000000000.
2020-09-04 09:10:56 DEBUG o.a.f.r.m.MemoryManager [] - Initialized MemoryManager with total memory size 37580964 and page size 32768.
2020-09-04 09:10:56 INFO o.a.f.r.t.TaskExecutor [] - Allocated slot for c37da0fd74b14bd257a4ecce33c06d79.
2020-09-04 09:10:56 INFO o.a.f.r.t.DefaultJobLeaderService [] - Add job 69482630ce87466bb580bff416c284e5 for job leader monitoring.
2020-09-04 09:10:56 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - New leader information for job 69482630ce87466bb580bff416c284e5. Address: akka.tcp://flink#flink:6123/user/rpc/jobmanager_12, leader id: 00000000000000000000000000000000.
2020-09-04 09:10:56 INFO o.a.f.r.t.DefaultJobLeaderService [] - Try to register at job manager akka.tcp://flink#flink:6123/user/rpc/jobmanager_12 with leader id 00000000-0000-0000-0000-000000000000.
2020-09-04 09:10:56 DEBUG o.a.f.r.r.a.AkkaRpcService [] - Try to connect to remote RPC endpoint with address akka.tcp://flink#flink:6123/user/rpc/jobmanager_12. Returning a org.apache.flink.runtime.jobmaster.JobMasterGateway gateway.
2020-09-04 09:10:56 INFO o.a.f.r.t.DefaultJobLeaderService [] - Resolved JobManager address, beginning registration
2020-09-04 09:10:56 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager attempt 1 (timeout=100ms)
2020-09-04 09:10:56 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager (akka.tcp://flink#flink:6123/user/rpc/jobmanager_12) attempt 1 timed out after 100 ms
2020-09-04 09:10:56 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager attempt 2 (timeout=200ms)
2020-09-04 09:10:56 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager (akka.tcp://flink#flink:6123/user/rpc/jobmanager_12) attempt 2 timed out after 200 ms
2020-09-04 09:10:56 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager attempt 3 (timeout=400ms)
2020-09-04 09:10:56 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager (akka.tcp://flink#flink:6123/user/rpc/jobmanager_12) attempt 3 timed out after 400 ms
2020-09-04 09:10:56 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager attempt 4 (timeout=800ms)
2020-09-04 09:10:57 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager (akka.tcp://flink#flink:6123/user/rpc/jobmanager_12) attempt 4 timed out after 800 ms
2020-09-04 09:10:57 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager attempt 5 (timeout=1600ms)
2020-09-04 09:10:59 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager (akka.tcp://flink#flink:6123/user/rpc/jobmanager_12) attempt 5 timed out after 1600 ms
2020-09-04 09:10:59 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager attempt 6 (timeout=3200ms)
2020-09-04 09:11:01 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:11:02 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager (akka.tcp://flink#flink:6123/user/rpc/jobmanager_12) attempt 6 timed out after 3200 ms
2020-09-04 09:11:02 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Registration at JobManager attempt 7 (timeout=6400ms)
2020-09-04 09:11:06 DEBUG o.a.f.r.t.TaskExecutor [] - Free slot with allocation id c37da0fd74b14bd257a4ecce33c06d79 because: The slot c37da0fd74b14bd257a4ecce33c06d79 has timed out.
2020-09-04 09:11:06 DEBUG o.a.f.r.t.s.TaskSlotTableImpl [] - Free slot TaskSlot(index:2, state:ALLOCATED, resource profile: ResourceProfile{cpuCores=1.0000000000000000, taskHeapMemory=12.800mb (13421771 bytes), taskOffHeapMemory=0 bytes, managedMemory=35.840mb (37580964 bytes), networkMemory=8.960mb (9395241 bytes)}, allocationId: c37da0fd74b14bd257a4ecce33c06d79, jobId: 69482630ce87466bb580bff416c284e5).
java.lang.Exception: The slot c37da0fd74b14bd257a4ecce33c06d79 has timed out.
at org.apache.flink.runtime.taskexecutor.TaskExecutor.timeoutSlot(TaskExecutor.java:1653) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskexecutor.TaskExecutor.access$2800(TaskExecutor.java:173) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskexecutor.TaskExecutor$SlotActionsImpl.lambda$timeoutSlot$1(TaskExecutor.java:1940) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.12-1.11.1.jar:1.11.1]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-dist_2.12-1.11.1.jar:1.11.1]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.12-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.12-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-dist_2.12-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.actor.Actor.aroundReceive(Actor.scala:517) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.actor.Actor.aroundReceive$(Actor.scala:515) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.11.1.jar:1.11.1]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.11.1.jar:1.11.1]
2020-09-04 09:11:06 INFO o.a.f.r.t.DefaultJobLeaderService [] - Remove job 69482630ce87466bb580bff416c284e5 from job leader monitoring.
2020-09-04 09:11:06 DEBUG o.a.f.r.t.DefaultJobLeaderService [] - Retrying registration towards akka.tcp://flink#flink:6123/user/rpc/jobmanager_12 was cancelled.
2020-09-04 09:11:06 DEBUG o.a.f.r.s.TaskExecutorLocalStateStoresManager [] - Releasing local state under allocation id c37da0fd74b14bd257a4ecce33c06d79.
2020-09-04 09:11:11 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:11:20 DEBUG a.a.LocalActorRefProvider(akka://flink) [] - Resolve (deserialization) of path [temp/$y] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-09-04 09:11:21 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:11:24 DEBUG a.a.LocalActorRefProvider(akka://flink) [] - Resolve (deserialization) of path [temp/$E] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-09-04 09:11:24 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from b128e6525f090a1f2a909ea501307f1d.
2020-09-04 09:11:24 DEBUG a.a.LocalActorRefProvider(akka://flink) [] - Resolve (deserialization) of path [temp/$z] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-09-04 09:11:24 DEBUG a.a.LocalActorRefProvider(akka://flink) [] - Resolve (deserialization) of path [temp/$A] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-09-04 09:11:24 DEBUG a.a.LocalActorRefProvider(akka://flink) [] - Resolve (deserialization) of path [temp/$B] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-09-04 09:11:24 DEBUG a.a.LocalActorRefProvider(akka://flink) [] - Resolve (deserialization) of path [temp/$C] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-09-04 09:11:24 DEBUG a.a.LocalActorRefProvider(akka://flink) [] - Resolve (deserialization) of path [temp/$D] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-09-04 09:11:31 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:11:34 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from b128e6525f090a1f2a909ea501307f1d.
2020-09-04 09:11:41 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:11:44 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from b128e6525f090a1f2a909ea501307f1d.
2020-09-04 09:11:51 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:11:53 DEBUG a.r.t.n.NettyTransport [] - Remote connection to [/192.168.3.147:58278] was disconnected because of [id: 0xaaf4e1c4, /192.168.3.147:58278 :> /172.16.0.210:6122] DISCONNECTED
2020-09-04 09:11:53 DEBUG a.r.t.ProtocolStateActor [] - Association between local [tcp://flink#172.16.0.210:6122] and remote [tcp://flink#192.168.3.147:58278] was disassociated because the ProtocolStateActor failed: Unknown
2020-09-04 09:11:54 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from b128e6525f090a1f2a909ea501307f1d.
2020-09-04 09:12:01 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:12:04 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from b128e6525f090a1f2a909ea501307f1d.
2020-09-04 09:12:11 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:12:21 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:12:31 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:12:41 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:12:51 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:12:53 DEBUG a.r.t.n.NettyTransport [] - Remote connection to [/192.168.3.147:59802] was disconnected because of [id: 0x6fea1015, /192.168.3.147:59802 :> /172.16.0.210:6122] DISCONNECTED
2020-09-04 09:12:53 DEBUG a.r.t.ProtocolStateActor [] - Association between local [tcp://flink#172.16.0.210:6122] and remote [tcp://flink#192.168.3.147:59802] was disassociated because the ProtocolStateActor failed: Unknown
2020-09-04 09:13:01 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:13:11 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:13:21 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:13:31 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:13:41 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:13:51 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.
2020-09-04 09:13:53 DEBUG a.r.t.n.NettyTransport [] - Remote connection to [/192.168.3.147:33116] was disconnected because of [id: 0xc7769526, /192.168.3.147:33116 :> /172.16.0.210:6122] DISCONNECTED
2020-09-04 09:13:53 DEBUG a.r.t.ProtocolStateActor [] - Association between local [tcp://flink#172.16.0.210:6122] and remote [tcp://flink#192.168.3.147:33116] was disassociated because the ProtocolStateActor failed: Unknown
2020-09-04 09:14:01 DEBUG o.a.f.r.t.TaskExecutor [] - Received heartbeat request from d5ddacd80a913c1ae961a83cbe58a598.

Unable to consume messages with Camel 3.3 and Google Pubsub

I have a small groovy script with Apache camel that is attempting to pull messages from Google Pubsub. This works well on Camel 2.25.1, but camel 3.3.0 does not. Here is the script. I also looked at the Camel 2.x to 3.0 migration guide, but I am unable to find anything relevant to this issue, but perhaps I am mistaken and I would need some help.
My groovy version is 3.0.4 and I am using JDK 11.0.6
#Grapes([
#Grab(group='org.apache.camel', module='camel-core', version='3.3.0'),
#Grab(group='org.apache.camel', module='camel-google-pubsub', version='3.3.0'),
#Grab(group='org.slf4j', module='slf4j-api', version='1.7.30'),
#Grab(group='ch.qos.logback', module='logback-classic', version='1.2.3'),
])
import org.apache.camel.CamelContext
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.builder.RouteBuilder
main()
def main() {
println "downloaded camel dependencies"
CamelContext context = new DefaultCamelContext()
context.addRoutes(new ReadFromQueueWithPubsub())
context.start()
addShutdownHook { context.stop() }
synchronized(this){ this.wait() }
}
class ReadFromQueueWithPubsub extends RouteBuilder {
#Override
void configure() {
from("google-pubsub://my_gcp_project:transaction-test-subscriber")
.to("log:input?showAll=true")
}
}
Here is the snippet of logs in debug mode. I can't see anything that's relevant here, but I may be mistaken.
2020-05-28 07:14:51 INFO [o.a.c.i.e.AbstractCamelContext ] - Apache Camel 3.3.0 (CamelContext: camel-1) is starting
2020-05-28 07:14:51 DEBUG [o.a.c.i.e.AbstractCamelContext ] - Using ClassResolver=org.apache.camel.impl.engine.DefaultClassResolver#25478603, PackageScanClassResolver=org.apache.camel.impl.engine.DefaultPackageScanClassResolver#3c7dbf1f, ApplicationContextClassLoader=null, RouteController=org.apache.camel.impl.engine.DefaultRouteController#2efbe938
2020-05-28 07:14:51 INFO [o.a.c.i.e.AbstractCamelContext ] - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
2020-05-28 07:14:51 DEBUG [o.a.c.i.e.AbstractCamelContext ] - Using HeadersMapFactory: org.apache.camel.impl.engine.DefaultHeadersMapFactory#12459991
2020-05-28 07:14:51 DEBUG [o.a.c.i.e.AbstractCamelContext ] - Using ReactiveExecutor: org.apache.camel.impl.engine.DefaultReactiveExecutor#43114981
2020-05-28 07:14:51 DEBUG [o.a.c.i.e.AbstractCamelContext ] - Warming up route id: route1 having autoStartup=true
2020-05-28 07:14:51 DEBUG [o.a.c.s.DefaultProducer ] - Starting producer: Producer[log://input?showAll=true]
2020-05-28 07:14:52 DEBUG [o.a.c.i.e.AbstractCamelContext ] - Route: route1 >>> Route[google-pubsub://my_gcp_project:transaction-test-subscriber -> null]
2020-05-28 07:14:52 DEBUG [o.a.c.i.e.AbstractCamelContext ] - Starting consumer (order: 1000) on route: route1
2020-05-28 07:14:52 DEBUG [o.a.c.s.DefaultConsumer ] - Init consumer: Consumer[google-pubsub://my_gcp_project:transaction-test-subscriber]
2020-05-28 07:14:52 DEBUG [o.a.c.s.DefaultConsumer ] - Starting consumer: Consumer[google-pubsub://my_gcp_project:transaction-test-subscriber]
2020-05-28 07:14:52 INFO [o.a.c.c.g.p.GooglePubsubConsumer ] - Starting Google PubSub consumer for my_gcp_project/transaction-test-subscriber
2020-05-28 07:14:52 DEBUG [o.a.c.i.e.BaseExecutorServiceManager ] - Created new ThreadPool for source: google-pubsub://my_gcp_project:transaction-test-subscriber with name: GooglePubsubConsumer[transaction-test-subscriber]. -> org.apache.camel.util.concurrent.RejectableThreadPoolExecutor#7d726fa1[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0][GooglePubsubConsumer[transaction-test-subscriber]]
2020-05-28 07:14:52 INFO [o.a.c.i.e.AbstractCamelContext ] - Route: route1 started and consuming from: google-pubsub://my_gcp_project:transaction-test-subscriber
2020-05-28 07:14:52 INFO [o.a.c.i.e.AbstractCamelContext ] - Total 1 routes, of which 1 are started
2020-05-28 07:14:52 INFO [o.a.c.i.e.AbstractCamelContext ] - Apache Camel 3.3.0 (CamelContext: camel-1) started in 0.078 seconds
2020-05-28 07:14:52 DEBUG [o.a.c.c.g.p.GooglePubsubConsumer ] - Subscribing to projects/my_gcp_project/subscriptions/transaction-test-subscriber
Closing this out: It looks like there's a bug open in JIRA for this CAMEL-15064 that affects versions >= 3.2

can not invoke chromedriver on Linux (debian) machine

Resources: VM -Debian Linux , Selenium 4 , Chrome V78, ChromeDriver , Jenkins , Maven, JAVA 8
**Issue:** I can not invoke chromedriver on Linux (debian) machine.
I can see message "Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code."
on Jenkins.
Maven command : clean install test
Log Trace:::
[JENKINS REMOTING CAPACITY]===>channel started
Executing Maven: -B -f /var/lib/jenkins/workspace/Maven/pom.xml clean install test
[INFO] Scanning for projects...
[WARNING]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # Land ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 8 source files to /var/lib/jenkins/workspace/Maven/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) # Land ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
[INFO ] 2019-09-18 21:51:26.644 [main] BaseClass - [32mINFO[m - Creating Screenshot folder for this current execution.
directory is : /var/lib/jenkins/workspace/Maven/screenshots
directory is exist /var/lib/jenkins/workspace/Maven/logs
directory is exist /var/lib/jenkins/workspace/Maven/test-output
[INFO ] 2019-09-18 21:51:26.655 [main] SetUpTest - [32mINFO[m - ============================================================
[INFO ] 2019-09-18 21:51:26.655 [main] SetUpTest - [32mINFO[m - ====== Initializing Driver, Starting Browser Session =========
[INFO ] 2019-09-18 21:51:26.655 [main] SetUpTest - [32mINFO[m - ============================================================
projectPath is: /var/lib/jenkins/workspace/Maven
[INFO ] 2019-09-18 21:51:26.656 [main] BaseClass - [32mINFO[m - projectPath is: /var/lib/jenkins/workspace/Maven
[INFO ] 2019-09-18 21:51:26.656 [main] BaseClass - [32mINFO[m - Driver value is : null
[INFO ] 2019-09-18 21:51:26.656 [main] BaseClass - [32mINFO[m - chrome driver is here
Starting ChromeDriver 78.0.3904.11 (eaaae9de6b8999773fa33f92ce1e1bbe294437cf-refs/branch-heads/3904#{#86}) on port 4878
Only local connections are allowed.
**Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.**
[INFO ] 2019-09-18 21:51:27.930 [main] SetUpTest -
[INFO ] 2019-09-18 21:51:27.930 [main] SetUpTest - ============ createAccount Test case is Started. ==============
[INFO ] 2019-09-18 21:51:27.931 [main] SetUpTest -
[INFO ] 2019-09-18 21:51:27.932 [main] TestListener -
[INFO ] 2019-09-18 21:51:27.933 [main] TestListener - ===== createAccount - Test case is Skipped =======
[INFO ] 2019-09-18 21:51:27.934 [main] TestListener -
[INFO ] 2019-09-18 21:51:27.934 [main] TestListener - ===== createAccount - Test case is Failed =======
[INFO ] 2019-09-18 21:51:27.941 [main] BaseClass - ******** Disposing Browser Driver **********
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[ERROR] There are test failures. This is the final result
----------------------------------------------------------------
**My code :**
This is my Selenium code:
if (browserName.equalsIgnoreCase("chrome")) {
System.setProperty("webdriver.chrome.driver", "/usr/bin/chromedriver");
ChromeOptions options = new ChromeOptions();
options.addArguments("--verbose");
options.addArguments("--whitelisted-ips=''");
options.addArguments("--disable-extensions");
options.addArguments("--disable-notifications");
options.addArguments("disable-infobars");
options.addArguments("--incognito");
options.addArguments("--disable-gpu");
options.addArguments("--no-sandbox");
options.addArguments("--disable-gpu --disable-software-rasterizer");
options.addArguments("--disable-gpu-sandbox");
options.addArguments("--disable-features=VizDisplayCompositor");
These are options I have tried on chromedriver so far
driver = new ChromeDriver(options); // This line is not being invoked successfully.
------------------------------------------
What I tried so far:
1. I tried firefox browser - which is not launching as well
2. I tried to "chmod 777" files, drivers and folders as per requirement
3. I set up few chrome "options.addArguments" to "disbale GPU" , "headless" and "no sandbox" as per google
4. I looked in VM that if there are already multiple instance opened for Chromedriver via "htop" and other commands to kill the older sessions but I did not see any chrome because it's never been launched
5. https://stgconsulting.com/running-automated-selenium-tests-from-jenkins-on-linux/
6. https://tecadmin.net/setup-selenium-chromedriver-on-ubuntu
Resources: Maven, Chrome,ChromeDriver, Linux(Debian), Java, Testng, jenkins
Link I followed for basic Linux (Debian) set up : https://stgconsulting.com/running-automated-selenium-tests-from-jenkins-on-linux/
Before solution we have been seeing this below error:
[INFO ] 2019-09-20 09:41:19.341 [main] BaseClass - browserName value is : chrome
[INFO ] 2019-09-20 09:41:19.341 [main] BaseClass - chrome driver is here
Starting ChromeDriver 78.0.3904.11 (eaaae9de6b8999773fa33f92ce1e1bbe294437cf-refs/branch-heads/3904#{#86}) on port 29688
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[INFO ] 2019-09-20 09:41:23.904 [main] SetUpTest - ==========================================================
[INFO ] 2019-09-20 09:41:23.904 [main] SetUpTest - =========== createAccount Test case is Started.
After solution I am not seeing failure/error and I am able to execute lest on Linux (Debian):
[INFO ] 2019-09-20 08:46:30.675 [main] BaseClass - chrome driver is here
Starting ChromeDriver 75.0.3770.140 (2d9f97485c7b07dc18a74666574f19176731995c-refs/branch-heads/3770#{#1155}) on port 9770
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[1568994392.814][WARNING]: This version of ChromeDriver has not been tested with Chrome version 76.
Sep 20, 2019 8:46:33 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[INFO ] 2019-09-20 08:46:33.227 [main] BaseClass - Browser is Chrome
solution steps and information:
(1) You can have latest Chrome Linux Browser on your box
(2) Go here https://chromedriver.chromium.org/downloads and do bruteforce of ChromeDriver versions
(3) I am using Chrome linux 77 (as of now 9/20/2019) and chromedriver 76.0.3809.25 , worked for me
(4) "Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code" - got nothing with firewall
(5) Check your Chmod 777 permission for chromeDriver
It's a BROWSER AND CHROME DRIVER MISMTACH , ALL YOU NEED TO DO IS TO CHECK THE COMPATIBILITY

Zap export report is not working in Jenkins

Trying to export report using ZAP in Jenkins.
Getting below errors :-
[ZAP Jenkins Plugin] INITIALIZATION [ SUCCESSFUL ]
REQUIRED PLUGIN(S) ARE MISSING
[ZAP Jenkins Plugin] SHUTDOWN [ START ]
and in local OWASP ZAP/zap.log:-
2018-11-18 09:52:48,551 [main ] INFO Options ParamCertificate - Unsafe SSL renegotiation disabled.2018-11-18 09:52:49,684 [main ] INFO ENGINE - open start - state not modified 2018-11-18 09:52:50,085 [main ] INFO ENGINE - dataFileCache open start2018-11-18 09:52:50,134 [main ] INFO ENGINE - dataFileCache open end2018-11-18 09:52:50,498 [ZAP-daemon] INFO ExtensionFactory - Loading extensions2018-11-18 09:52:50,746 [ZAP-daemon] ERROR ExtensionAutoUpdate - Unable to load the configuration org.apache.commons.configuration.ConfigurationException: Unable to load the configuration
I resolved this issue:- Added add-on exportreport-alpha-5 plugin in the ZAP Home directory (in plugin directory) .

Resources