I am trying to deploy a .war file called LicenseManager.war. The data server is hosted on another machine with an ip address. I am the database drivers have been correctly imported. After trying to copy and paste the LicenseManager.war file in deployments folder, the deployment fails and a LicenseManager.war.failed is created. The following is the server log :
2017-09-07 14:38:47,802 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 59) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
2017-09-07 14:38:47,825 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 59) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
2017-09-07 14:38:47,859 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 59) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
2017-09-07 14:38:47,859 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 59) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
2017-09-07 14:38:48,415 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0006: Starting Services for CDI deployment: LicenseManager.war
2017-09-07 14:38:48,602 INFO [org.jboss.weld.Version] (MSC service thread 1-1) WELD-000900: 2.3.3 (redhat)
2017-09-07 14:38:48,851 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0009: Starting weld service for deployment LicenseManager.war
2017-09-07 14:38:49,303 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 59) WFLYCLINF0002: Started client-mappings cache from ejb container
2017-09-07 14:38:49,642 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "LicenseManager.war")]) - failure description: {"WFLYCTL0180: Services with missing/unavailable dependencies" => [
"jboss.persistenceunit.\"LicenseManager.war#LicenseManager\" is missing [jboss.naming.context.java.jboss.manageIDDatasource]",
"jboss.persistenceunit.\"LicenseManager.war#LicenseManager\".__FIRST_PHASE__ is missing [jboss.naming.context.java.jboss.manageIDDatasource]"
]}
2017-09-07 14:38:49,734 INFO [org.jboss.as.server] (ServerService Thread Pool -- 34) WFLYSRV0010: Deployed "LicenseManager.war" (runtime-name : "LicenseManager.war")
2017-09-07 14:38:49,755 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0184: New missing/unsatisfied dependencies:
service jboss.naming.context.java.jboss.manageIDDatasource (missing) dependents: [service jboss.persistenceunit."LicenseManager.war#LicenseManager", service jboss.persistenceunit."LicenseManager.war#LicenseManager".__FIRST_PHASE__]
2017-09-07 14:38:50,197 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management
2017-09-07 14:38:50,198 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990
2017-09-07 14:38:50,198 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: JBoss EAP 7.0.0.GA (WildFly Core 2.1.2.Final-redhat-1) started (with errors) in 18730ms - Started 578 of 983 services (135 services failed or missing dependencies, 385 services are lazy, passive or on-demand)
2017-09-07 14:38:50,410 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 59) WFLYCLINF0003: Stopped client-mappings cache from ejb container
2017-09-07 14:38:50,541 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment LicenseManager.war (runtime-name: LicenseManager.war) in 301ms
2017-09-07 14:38:50,724 INFO [org.jboss.as.repository] (DeploymentScanner-threads - 1) WFLYDR0002: Content removed from location /home/aboshi/EAP-7.0.0/standalone/data/content/7e/7e872f95551f1c9ff56e422fcc5a18b44a4e12/content
2017-09-07 14:38:50,724 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0009: Undeployed "LicenseManager.war" (runtime-name: "LicenseManager.war")
2017-09-07 14:38:50,783 INFO [org.jboss.as.controller] (DeploymentScanner-threads - 1) WFLYCTL0183: Service status report
WFLYCTL0184: New missing/unsatisfied dependencies:
service jboss.deployment.unit."LicenseManager.war".WeldBootstrapService (missing) dependents: [service jboss.deployment.unit."LicenseManager.war".component."com.sun.faces.taglib.html_basic.CommandButtonTag".WeldInstantiator, service jboss.deployment.unit."LicenseManager.war".component."com.sun.faces.taglib.jsf_core.RegexValidatorTag".WeldInstantiator, service jboss.deployment.unit."LicenseManager.war".component."com.sun.faces.taglib.jsf_core.SelectItemTag".WeldInstantiator, service jboss.deployment.unit."LicenseManager.war".component.LicenseTimerBean.WeldInterceptorBindingsService, WFLYCTL0208: ... and 63 more ]
service jboss.deployment.unit."LicenseManager.war".WeldStartService (missing) dependents: [service jboss.deployment.unit."LicenseManager.war".component."com.sun.faces.taglib.html_basic.CommandButtonTag".WeldInstantiator, service jboss.deployment.unit."LicenseManager.war".component."com.sun.faces.taglib.jsf_core.RegexValidatorTag".WeldInstantiator, service jboss.deployment.unit."LicenseManager.war".component."com.sun.faces.taglib.jsf_core.SelectItemTag".WeldInstantiator, service jboss.deployment.unit."LicenseManager.war".component.LicenseTimerBean.WeldInterceptorBindingsService, WFLYCTL0208: ... and 64 more ]
service jboss.deployment.unit."LicenseManager.war".beanmanager (missing) dependents: [service jboss.persistenceunit."LicenseManager.war#LicenseManager"]
service jboss.deployment.unit."LicenseManager.war".component.LicenseManagerAction.CREATE (missing) dependents: [service jboss.deployment.unit."LicenseManager.war".component.LicenseManagerAction.START] service jboss.deployment.unit."LicenseManager.war".component.LicenseManagerAction.JndiBindingsService (missing) dependents: [service jboss.deployment.unit."LicenseManager.war".jndiDependencyService]
service
Also, one of the main errors is :
6:34:19,720 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "LicenseManager.war")]) - failure description: {"WFLYCTL0180: Services with missing/unavailable dependencies" => [
"jboss.persistenceunit.\"LicenseManager.war#LicenseManager\" is missing [jboss.naming.context.java.jboss.manageIDDatasource]",
"jboss.persistenceunit.\"LicenseManager.war#LicenseManager\".__FIRST_PHASE__ is missing [jboss.naming.context.java.jboss.manageIDDatasource]"
]}
This is way too much to understand. Also, a similar configuration ran successfully on a windows 7 OS where again, the database was external. But this one is on RHEL 7.4. Can some please help me understand what is going on? Thank you very much!!
Regards,
Abhi Joe
It seems JBoss is not finding the datasource where it expects it:
jboss.naming.context.java.jboss.manageIDDatasource
You might first turn logging to 'debug' then start the server-- grep for any lines that deal with manageIDDatasource.
It's quite possible the deployment of the datasource wasn't as smooth as could be. Look for examples of doing this, then when there are no errors things should be better. (If there are still questions about the validity of the datasource, then maybe write a small servlet to retrieve and test it)
Related
I am trying MDB in jboss server 7.1.0 GA. When I am deploying my MDB I am getting error and server is not started:
11:02:57,788 ERROR [org.jboss.as.controller.management-operation]
(Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed -
address: ([("deployment" => "MDBExample.jar")]) - failure description:
{ "WFLYCTL0412: Required services that are not installed:" =>
["jboss.ra.hornetq-ra"], "WFLYCTL0180: Services with
missing/unavailable dependencies" =>
["jboss.deployment.unit.\"MDBExample.jar\".component.MessageMDBSample.CREATE
is missing [jboss.ra.hornetq-ra]"] }11:02:57,808 INFO
[org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010:
Deployed "MDBExample.jar" (runtime-name : "MDBExample.jar")
11:02:57,825 INFO [org.jboss.as.controller] (Controller Boot Thread)
WFLYCTL0183: Service status report WFLYCTL0184: New
missing/unsatisfied dependencies: service jboss.ra.hornetq-ra
(missing) dependents: [service
jboss.deployment.unit."MDBExample.jar".component.MessageMDBSample.CREATE]
I have a SolrCloud cluster (6.6) setup with external Zookeeper Ensemble (3.4.8) of 5 nodes. Recently, one machine (ip1:port1) that run 1 Zookeeper with id=1 went down. This is what I've done to replace zookeeper:
Start zookeeper in another machine with the same id (=1).
Change zoo.cfg in 4 live zookeeper to match new zookeeper server and restart.
Update ZK_HOST variable in solr.in.sh to match new zookeeper server.
Restart solr.
After that, my solr cluster seemed to functioning well, but in solr.log, it looked like solr client and zookeeper servers still try to connect to the old zookeeper:
Solr log
2017-12-01 15:04:38.782 WARN (Timer-0-SendThread(ip1:port1)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 30029ms for sessionid 0x0
2017-12-01 15:04:40.807 WARN (Timer-0-SendThread(ip1:port1)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 31030ms for sessionid 0x0
Zookeeper log:
2017-12-01 13:53:57,972 [myid:] - INFO [main-SendThread(ip1:port1):ClientCnxn$SendThread#1032] - Opening socket connection to server ip1:port1. Will not attempt to authenticate using SASL (unknown error)
2017-12-01 13:54:03,972 [myid:] - WARN [main-SendThread(ip1:port1):ClientCnxn$SendThread#1162] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2017-12-01 13:54:05,074 [myid:] - INFO [main-SendThread(ip1:port1):ClientCnxn$SendThread#1032] - Opening socket connection to server ip1:port1. Will not attempt to authenticate using SASL (unknown error)
2017-12-01 13:54:06,974 [myid:] - WARN [main-SendThread(ip1:port1):ClientCnxn$SendThread#1162] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
I've done some search in add/remove zookeeper but didn't find a document for it. My zookeeper version (3.4.7) is not supported for dynamic reconfiguration (which is in zookeeper 3.5).
Is there a way I can manually remove/add zookeeper server from ensemble?
Thanks for your attention!
Iam trying to deploy a sample camel war application created using maven archetype camel-archetype-web:
https://mvnrepository.com/artifact/org.apache.camel.archetypes/camel-archetype-web/2.10.0.redhat-60092
Iam getting below info and warning in logs ,the camel application is not running.
The application is placed in below link.
https://drive.google.com/file/d/0B7junC5UmfFgNlFJVnFtOENtd2s/view?usp=sharing
Server log:
10:11:49,488 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "sampleapp-0.0.1-SNAPSHOT.war" (runtime-name: "sampleapp-0.0.1-SNAPSHOT.war")
10:11:51,324 WARN [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015960: Class Path entry jaxb-api.jar in /C:/Temp/FSW/jboss-eap-6.1/bin/content/sampleapp-0.0.1-SNAPSHOT.war/WEB-INF/lib/jaxb-impl-2.2.5.jar does not point to a valid jar for a Class-Path reference.
10:11:51,340 WARN [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015960: Class Path entry activation.jar in /C:/Temp/FSW/jboss-eap-6.1/bin/content/sampleapp-0.0.1-SNAPSHOT.war/WEB-INF/lib/jaxb-impl-2.2.5.jar does not point to a valid jar for a Class-Path reference.
10:11:51,355 WARN [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015960: Class Path entry jsr173_1.0_api.jar in /C:/Temp/FSW/jboss-eap-6.1/bin/content/sampleapp-0.0.1-SNAPSHOT.war/WEB-INF/lib/jaxb-impl-2.2.5.jar does not point to a valid jar for a Class-Path reference.
10:11:51,371 WARN [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015960: Class Path entry jaxb1-impl.jar in /C:/Temp/FSW/jboss-eap-6.1/bin/content/sampleapp-0.0.1-SNAPSHOT.war/WEB-INF/lib/jaxb-impl-2.2.5.jar does not point to a valid jar for a Class-Path reference.
10:11:51,527 INFO [org.jboss.web] (ServerService Thread Pool -- 110) JBAS018210: Register web context: /sampleapp-0.0.1-SNAPSHOT
10:11:51,543 INFO [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/sampleapp-0.0.1-SNAPSHOT]] (ServerService Thread Pool -- 110) No Spring WebApplicationInitializer types detected on classpath
10:11:51,558 INFO [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/sampleapp-0.0.1-SNAPSHOT]] (ServerService Thread Pool -- 110) Initializing Spring root WebApplicationContext
10:11:52,812 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) JBAS018565: Replaced deployment "sampleapp-0.0.1-SNAPSHOT.war" with deployment "sampleapp-0.0.1-SNAPSHOT.war"
Any Help appreciated.
I am trying to implement cronscheduledroutepolicy in camel route and it is not working as expected. I am expecting switchyard Route to stop and start at cron trigger. I had added routePolicyRef="customRoutePolicy" and autoStartup="false" attributes in route definition.
logs:
23:07:01,530 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) JBAS016009: Stopping weld service for deployment Thunderhead-POC-PDF-1.1-SNAPSHOT.jar
23:07:01,558 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment Thunderhead-POC-PDF-1.1-SNAPSHOT.jar (runtime-name: Thunderhead-POC-PDF-1.1-SNAPSHOT.jar) in 30ms
23:07:01,559 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "Thunderhead-POC-PDF-1.1-SNAPSHOT.jar" (runtime-name: "Thunderhead-POC-PDF-1.1-SNAPSHOT.jar")
23:07:01,657 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) JBAS016002: Processing weld deployment Thunderhead-POC-PDF-1.1-SNAPSHOT.jar
23:07:01,716 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) JBAS016005: Starting Services for CDI deployment: Thunderhead-POC-PDF-1.1-SNAPSHOT.jar
23:07:01,718 INFO [org.switchyard] (MSC service thread 1-4) Deploying SwitchYard application 'Thunderhead-POC-PDF-1.1-SNAPSHOT.jar'
23:07:01,720 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) JBAS016008: Starting weld service for deployment Thunderhead-POC-PDF-1.1-SNAPSHOT.jar
23:07:01,764 INFO [org.switchyard] (MSC service thread 1-4) Starting SwitchYard service
23:07:01,768 INFO [org.apache.camel.management.ManagementStrategyFactory] (MSC service thread 1-4) JMX enabled.
23:07:01,771 INFO [org.switchyard.common.camel.SwitchYardCamelContext] (MSC service thread 1-4) Apache Camel 1.1.1-p5-redhat-1 (CamelContext: camel-33) is starting
23:07:01,805 INFO [org.apache.camel.impl.converter.DefaultTypeConverter] (MSC service thread 1-4) Loaded 181 type converters
23:07:01,814 INFO [org.switchyard.common.camel.SwitchYardCamelContext] (MSC service thread 1-4) Total 0 routes, of which 0 is started.
23:07:01,814 INFO [org.switchyard.common.camel.SwitchYardCamelContext] (MSC service thread 1-4) Apache Camel 1.1.1-p5-redhat-1 (CamelContext: camel-33) started in 0.043 seconds
23:07:01,967 INFO [org.quartz.simpl.SimpleThreadPool] (MSC service thread 1-4) Job execution threads will use class loader of thread: MSC service thread 1-4
23:07:01,978 INFO [org.quartz.core.SchedulerSignalerImpl] (MSC service thread 1-4) Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
23:07:01,979 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Quartz Scheduler v.1.8.5 created.
23:07:01,979 INFO [org.quartz.simpl.RAMJobStore] (MSC service thread 1-4) RAMJobStore initialized.
23:07:01,980 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler meta-data: Quartz Scheduler (v1.8.5) 'DefaultQuartzScheduler-camel-33' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
23:07:01,980 INFO [org.quartz.impl.StdSchedulerFactory] (MSC service thread 1-4) Quartz scheduler 'DefaultQuartzScheduler-camel-33' initialized from an externally provided properties instance.
23:07:01,980 INFO [org.quartz.impl.StdSchedulerFactory] (MSC service thread 1-4) Quartz scheduler version: 1.8.5
23:07:01,980 INFO [org.apache.camel.component.quartz.QuartzComponent] (MSC service thread 1-4) Starting Quartz scheduler: DefaultQuartzScheduler-camel-33
23:07:01,981 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler DefaultQuartzScheduler-camel-33_$_NON_CLUSTERED started.
23:07:01,982 INFO [org.apache.camel.routepolicy.quartz.ScheduledRoutePolicy] (MSC service thread 1-4) Scheduled trigger: triggerGroup-route19.trigger-START-route19 for action: START on route route19
23:07:02,001 INFO [org.switchyard.common.camel.SwitchYardCamelContext] (MSC service thread 1-4) Skipping starting of route route19 as its configured with autoStartup=false
23:07:02,153 INFO [org.switchyard.common.camel.SwitchYardCamelContext] (MSC service thread 1-4) Route: direct:{urn:com.example.switchyard:Thunderhead-POC-PDF:1.0}ESBService started and consuming from: Endpoint[direct://%7Burn:com.example.switchyard:Thunderhead-POC-PDF:1.0%7DESBService]
23:07:02,217 INFO [org.switchyard.common.camel.SwitchYardCamelContext] (MSC service thread 1-4) Route: V1CamelJmsBindingModel/ESBService#jms1#-275801100 started and consuming from: Endpoint[jms://queue:Document.Eu.In.Deferred?connectionFactory=%23ConnectionFactory&replyTo=Document.Eu.Out.Result]
23:07:02,319 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018565: Replaced deployment "Thunderhead-POC-PDF-1.1-SNAPSHOT.jar" with deployment "Thunderhead-POC-PDF-1.1-SNAPSHOT.jar"
Iam trying similar to this Throttling based on time interval need to pick messages from queue only during cron trigger,but the attached component starts listening the queue once deployment is done.
can you please tell me what might be the issue?
Camel Route:
<?xml version="1.0" encoding="ASCII"?>
<routes xmlns="http://camel.apache.org/schema/spring">
<route streamCache="true" routePolicyRef="customRoutePolicy" autoStartup="false">
<from uri="switchyard://ESBService" />
<log message="Header is: ${headers.ResponseType}" />
<choice>
<when>
<simple>${headers.ResponseType} == 'XMLANDPDF'</simple>
<bean ref="CreateStub" method="setAttachmentInHeader()" />
<to uri="xslt:xslt/create-response.xslt" />
</when>
<otherwise>
<bean ref="CreateStub" method="setAttachmentInBody()" />
</otherwise>
</choice>
<log message="Response is: ${body}" />
</route>
</routes>
Route Policy:
package com.example.switchyard.Thunderhead_POC_PDF;
import javax.enterprise.inject.Produces;
import javax.inject.Named;
import org.apache.camel.routepolicy.quartz.CronScheduledRoutePolicy;
public class MyRoutePolicy{
#Produces #Named("customRoutePolicy")
public static CronScheduledRoutePolicy createRoutePolicy ()
{
CronScheduledRoutePolicy policy=new CronScheduledRoutePolicy();
policy.setRouteStartTime("0 27 18 ? * *");
policy.setRouteStopGracePeriod(10000);
policy.setRouteStartTime("0 29 18 ? * *");
return policy;
}
}
Switchyard xml:
<?xml version="1.0" encoding="UTF-8"?>
<sy:switchyard xmlns:camel="urn:switchyard-component-camel:config:1.1" xmlns:jms="urn:switchyard-component-camel-jms:config:1.1" xmlns:sca="http://docs.oasis-open.org/ns/opencsa/sca/200912" xmlns:sy="urn:switchyard-config:switchyard:1.1" name="Thunderhead-POC-PDF" targetNamespace="urn:com.example.switchyard:Thunderhead-POC-PDF:1.0">
<sca:composite name="Thunderhead-POC-PDF" targetNamespace="urn:com.example.switchyard:Thunderhead-POC-PDF:1.0">
<sca:component name="Component">
<camel:implementation.camel>
<camel:xml path="route.xml"/>
</camel:implementation.camel>
<sca:service name="ESBService">
<sy:interface.esb inputType="java.lang.String" outputType="java.io.InputStream"/>
</sca:service>
</sca:component>
<sca:service name="ESBService" promote="Component/ESBService">
<sy:interface.esb inputType="java.lang.String" outputType="java.io.InputStream"/>
<jms:binding.jms name="jms1">
<jms:contextMapper includes="sourceSystemID.*,ResponseType.*"/>
<jms:queue>Document.Eu.In.Deferred</jms:queue>
<jms:connectionFactory>#ConnectionFactory</jms:connectionFactory>
<jms:replyTo>Document.Eu.Out.Result</jms:replyTo>
</jms:binding.jms>
</sca:service>
</sca:composite>
</sy:switchyard>
When I try to run my flow on Apache Flink standalone cluster I see the following exception:
java.lang.IllegalStateException: Update task on instance aaa0859f6af25decf1f5fc1821ffa55d # app-2 - 4 slots - URL: akka.tcp://flink#192.168.38.98:46369/user/taskmanager failed due to:
at org.apache.flink.runtime.executiongraph.Execution$6.onFailure(Execution.java:954)
at akka.dispatch.OnFailure.internal(Future.scala:228)
at akka.dispatch.OnFailure.internal(Future.scala:227)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at scala.runtime.AbstractPartialFunction.applyOrElse(AbstractPartialFunction.scala:28)
at scala.concurrent.Future$$anonfun$onFailure$1.apply(Future.scala:136)
at scala.concurrent.Future$$anonfun$onFailure$1.apply(Future.scala:134)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink#192.168.38.98:46369/user/taskmanager#1804590378]] after [10000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333)
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)
Seems like port 46369 blocked by firewall. It is true because I read configuration section and open these ports only:
6121:
comment: Apache Flink TaskManager (Data Exchange)
6122:
comment: Apache Flink TaskManager (IPC)
6123:
comment: Apache Flink JobManager
6130:
comment: Apache Flink JobManager (BLOB Server)
8081:
comment: Apache Flink JobManager (Web UI)
The same ports described in flink-conf.yaml:
jobmanager.rpc.address: app-1.stag.local
jobmanager.rpc.port: 6123
jobmanager.heap.mb: 1024
taskmanager.heap.mb: 2048
taskmanager.numberOfTaskSlots: 4
taskmanager.memory.preallocate: false
blob.server.port: 6130
parallelism.default: 4
jobmanager.web.port: 8081
state.backend: jobmanager
restart-strategy: none
restart-strategy.fixed-delay.attempts: 2
restart-strategy.fixed-delay.delay: 60s
So, I have two questions:
This exception related to blocked ports. Right?
Which ports should I open on firewall for standalone Apache Flink cluster?
UPDATE 1
I found configuration problem in masters and slaves files (I skip new line separators between hosts described in these files). I fixed it and now I see other exceptions:
flink--taskmanager-0-app-1.stag.local.log
flink--taskmanager-0-app-2.stag.local.log
I have 2 nodes:
app-1.stag.local (with running job and task managers)
app-2.stag.local (with running task manager)
As you can see from these logs the app-1.stag.local task manager can't connect to other task manager:
java.io.IOException: Connecting the channel failed: Connecting to remote task manager + 'app-2.stag.local/192.168.38.98:35806' has failed. This might indicate that the remote task manager has been lost.
but app-2.stag.local has open port:
2016-03-18 16:24:14,347 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 39 ms). Listening on SocketAddress /192.168.38.98:35806
So, I think problem related to firewall but I don't understand where I can configure this port (or range of ports) in Apache Flink.
I have found a problem: taskmanager.data.port parameter was set to 0 by default (but documentation say what it should be set to 6121).
So, I set this port in flink-conf.yaml and now all works fine.