Error opening socket to server while connecting to database - database

When my application server is running, Jboss is not able to connect to DB. I am running docker DB2 image and not using local DB. I am running Docker on Virtual Box. Application is opening socket to VirtualBox IP instead of localhost. Error says:
Error says:
Cannot create PoolableConnectionFactory : Exception java.net.ConnectException: Error opening socket to server /192.168.56.1 on port 50,000 with message: Connection refused: connect. ERRORCODE=-4499, SQLSTATE=08001....Where 192.168.56.1 is my Virtual Box Ip. I can connect my DB2 docker image through DB2 client running on local but not with my application.
Logs:
2019-02-11 18:20:16,806 INFO [ost-startStop-1 ation.ConfigurationClassParser] - Properties location [/opt/demandtec/common/conf/jms.properties] not resolvable: class path resource [opt/demandtec/common/conf/jms.properties] cannot be opened because it does not exist
2019-02-11 18:20:17,877 INFO [ost-startStop-1 redAnnotationBeanPostProcessor] - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2019-02-11 18:20:18,683 INFO [ost-startStop-1 dtec.message.DestReserverBeans] - Loading defalt queue topic definitions
2019-02-11 18:20:18,736 INFO [ost-startStop-1 mandtec.global.DTServiceConfig] - init jndi template
2019-02-11 18:20:18,764 INFO [ost-startStop-1 dtec.global.DTRemotingServices] - Start Locate Registry with port: 3099
2019-02-11 18:20:22,685 INFO [ost-startStop-1 ssage.activemq.DTAMQJmsService] - amq connect meta data created
2019-02-11 18:20:22,691 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: QueueConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,697 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: TopicConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,698 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: PromoCPQueueConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,698 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: PerishableQueueConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,698 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: AMEQueueConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,698 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: FarmQueueConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,720 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: IAMTopicConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,720 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: CTSQueueConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,721 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: GlobalQueueConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,722 INFO [ost-startStop-1 .message.InitConnectionFactory] - export: GlobalTopicConnectionFactory as interface com.ibm.demandtec.mq.RemoteConnectFactory
2019-02-11 18:20:22,731 INFO [ost-startStop-1 ndtec.mq.DTMQConnectionFactory] - Init class: org.apache.activemq.ActiveMQConnectionFactory, with: {brokerURL=tcp://IBM217-PC0BY3X9:61616, trustAllPackages=true}
2019-02-11 18:20:22,992 INFO [ost-startStop-1 ndtec.mq.DTMQConnectionFactory] - Init class: org.apache.activemq.ActiveMQConnectionFactory, with: {brokerURL=tcp://IBM217-PC0BY3X9:61616, clientID=IBM217-PC0BY3X9.IAM, trustAllPackages=true}
2019-02-11 18:20:23,357 WARN [ost-startStop-1 .commons.dbcp2.BasicDataSource] - The requested JMX name [demandtec.com_optimization] was not valid and will be ignored.
2019-02-11 18:20:53,117 INFO [pool-2-thread-1 .demandtec.global.DTMDBService] - Startup AdminMDB
2019-02-11 18:20:54,241 INFO [pool-2-thread-1 .demandtec.global.DTMDBService] - Startup IAM as clientid:IBM217-PC0BY3X9_appiam
2019-02-11 18:30:31,001 WARN [ost-startStop-1 emandtec.dal.ConnectionHandler] - JBossPoolException! Failed getting connection to Optimization for demandtec.com
2019-02-11 18:30:31,001 WARN [ost-startStop-1 emandtec.dal.ConnectionHandler] - cause: Cannot create PoolableConnectionFactory ([jcc][t4][2043][11550][4.22.29] Exception java.net.ConnectException: Error opening socket to server /192.168.56.1 on port 50,000 with message: Connection refused: connect. ERRORCODE=-4499, SQLSTATE=08001)
2019-02-11 18:30:31,001 WARN [ost-startStop-1 emandtec.dal.ConnectionHandler] - will attempt one more time.
2019-02-11 18:30:31,001 WARN [ost-startStop-1 .commons.dbcp2.BasicDataSource] - The requested JMX name [demandtec.com_optimization] was not valid and will be ignored.
Database: DB2; OS: Windows7/ Docker running on Oracle VirtualBox; Framework: J2EE; App-Server: Jboss

I think I have the same problem using ibmcom/db2 community edition image. It works well when I start the container on MacOS. Then if I start the container in virtualbox environment, jdbc connection fails.
[jcc][10521][13706]Command : java com.ibm.db2.jcc.DB2Jcc -user db2inst1 -password ******** -url jdbc:db2://0.0.0.0:50000/testdb
[jcc][10512][13714]Failed to create connection.
SQLCODE: -4499
SQLSTATE: 08001
Message: [jcc][t4][2030][11211][4.12.55] A communication error occurred during operations on the connection's underlying socket, socket input stream,
or socket output stream. Error location: Reply.fill() - insufficient data (-1). Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
Someone suspect it is because of ipv6. So I run docker run -itd --name mydb2 --privileged=true -p 50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=test123 -e DBNAME=testdb -v /home/vagrant/database:/database ibmcom/db2 to start a container. Then get into container with docker exec -it mydb2 bash. Remove serveral ipv6 lines and keep only following in /etc/hosts file.
127.0.0.1 localhost
172.17.0.2 def55f9a8b0f
Quit and remove this container. Restart container with command docker run -itd --name mydb2 --privileged=true -p 0.0.0.0:50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=test123 -e DBNAME=testdb -v /home/vagrant/database:/database ibmcom/db2.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4fbaec4a46ac ibmcom/db2 "/var/db2_setup/lib/…" 7 minutes ago Up 7 minutes 22/tcp, 55000/tcp, 60006-60007/tcp, 0.0.0.0:50000->50000/tcp mydb2
Wait for a few moments, JDBC connection works.
[jcc][10516][13709]Test Connection Successful.
DB product version = SQL11056
DB product name = DB2/LINUXX8664
DB URL = jdbc:db2://127.0.0.1:50000/testdb
DB Drivername = IBM Data Server Driver for JDBC and SQLJ
DB OS Name = Linux

For anyone having this issue on podman just set rootfull to true and repeat the setup process
podman machine set --rootful=true

In our scenario we solved just adding the option
privileged: true
to the db2 service in the docker-compose.yml file

Related

Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host in taskmanager

When I start my apache flink 1.10 taskmanager service in kubernetes(v1.15.2) cluster,it shows logs like this:
2020-05-01 08:34:55,847 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink#flink-jobmanager:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink#flink-jobmanager:6123/user/resourcemanager..
2020-05-01 08:34:55,847 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host
2020-05-01 08:34:55,848 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#flink-jobmanager:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#flink-jobmanager:6123]] Caused by: [java.net.NoRouteToHostException: No route to host]
2020-05-01 08:35:08,874 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host
2020-05-01 08:35:08,877 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#flink-jobmanager:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#flink-jobmanager:6123]] Caused by: [java.net.NoRouteToHostException: No route to host]
2020-05-01 08:35:08,878 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink#flink-jobmanager:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink#flink-jobmanager:6123/user/resourcemanager..
2020-05-01 08:35:21,907 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host
and the taskmanager could not registered success, and I logged into taskmanager and find out I could success ping jobmanager liket this:
flink#flink-taskmanager-54d85f57c7-nl9cf:~$ ping flink-jobmanager
PING flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171) 56(84) bytes of data.
64 bytes from flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171): icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171): icmp_seq=2 ttl=64 time=0.076 ms
64 bytes from flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171): icmp_seq=3 ttl=64 time=0.079 ms
so why this would happen and what should I do to fix it?
Try to install nmap in your kubernetes taskmanger's pod container:
apt-get udpate
apt-get install nmap -y
then scan the jobmanager and make sure the pod's expose port 6123 is accessable(in my case ,I found could not access the port 6123 from current pod).
nmap -T4 <your-jobmanager's-pod-ip>
Hope this help.

The sqoop is not working on my ubuntu 18.04 with hadoop 3.1.3

I am getting below error in my Ubutnttu(18.0.4) machine while launching sqoop(1.4.7,Hadoop-3.1.3)
command used:
sqoop import --connect jdbc:mysql://localhost/myhadoop --username hiveuser --password xxxx --table employee --split-by --target-dir /employee2
Error:
2020-04-30 15:28:01,570 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
Thu Apr 30 15:28:01 IST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2020-04-30 15:28:01,727 INFO db.DBInputFormat: Using read commited transaction isolation
2020-04-30 15:28:01,736 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
2020-04-30 15:28:01,772 INFO mapred.LocalJobRunner: map task executor complete.
2020-04-30 15:28:01,801 WARN mapred.LocalJobRunner: job_local1054959073_0001
java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class employee not found
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class employee not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2638)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:270)
at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:266)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:527)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Class employee not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2542)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)
... 12 more
2020-04-30 15:28:02,089 INFO mapreduce.Job: Job job_local1054959073_0001 running in uber mode : false
2020-04-30 15:28:02,094 INFO mapreduce.Job: map 0% reduce 0%
2020-04-30 15:28:02,104 INFO mapreduce.Job: Job job_local1054959073_0001 failed with state FAILED due to: NA
2020-04-30 15:28:02,143 INFO mapreduce.Job: Counters: 0
2020-04-30 15:28:02,151 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2020-04-30 15:28:02,155 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 3.8645 seconds (0 bytes/sec)
2020-04-30 15:28:02,160 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2020-04-30 15:28:02,160 INFO mapreduce.ImportJobBase: Retrieved 0 records.
2020-04-30 15:28:02,160 ERROR tool.ImportTool: Import failed: Import job failed!
Please advise
You have to specify --bin-dir for sqoop import. You can specify any directory.
From official documentation
The import process compiles the source into .class and .jar files; these are ordinarily stored under /tmp. You can select an alternate target directory with --bindir. For example, --bindir /scratch.
sqoop import --connect jdbc:mysql://localhost/myhadoop --username hiveuser --password xxxx --table employee --split-by --target-dir /employee2
--bindir /tmp

Cannot initialize database connection from spring boot application

I am able to successfully connect to my local database, by specifying following URL in application.properties file of my spring boot application:
application.properties
spring.datasource.url=jdbc:sqlserver://localhost:1433;databaseName=mantaDB;integratedSecurity=true
And here is my output:
2019-Aug-29 11:14:41.298 INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting... -
2019-Aug-29 11:14:41.753 INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed. -
2019-Aug-29 11:14:41.914 INFO o.h.jpa.internal.util.LogHelper - HHH000204: Processing PersistenceUnitInfo [_ name: default_ ...] -
2019-Aug-29 11:14:42.102 INFO org.hibernate.Version - HHH000412: Hibernate Core {5.3.7.Final} -
2019-Aug-29 11:14:42.396 INFO o.h.annotations.common.Version - HCANN000001: Hibernate Commons Annotations {5.0.4.Final} -
2019-Aug-29 11:14:42.827 INFO org.hibernate.dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.SQLServerDialect -
2019-Aug-29 11:14:44.157 INFO o.s.o.j.LocalContainerEntityManagerFactoryBean - Initialized JPA EntityManagerFactory for persistence unit 'default' -
Now, I want to change application to point to my dev db box:
application.properties
spring.datasource.url=jdbc:sqlserver://STUDENT12.site,28001;databaseName=mantaDB;integratedSecurity=true
But, I am getting following error:
2019-Aug-29 11:16:48.089 INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting... -
2019-Aug-29 11:17:18.682 ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Exception during pool initialization. -
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host STUDENT12.site,28001, port 1433 has failed. Error: "STUDENT12.site,28001. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:228)
at com.microsoft.sqlserver.jdbc.SQLServerException.ConvertConnectExceptionToSQLServerException(SQLServerException.java:285)
Clearly, I can get to the database.
Does anyone know what i am doing wrong?
The JDBC Url should look like:
jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]
Building the Connection URL
And the error makes it clear that you are not connecting on port 28001:
connection to the host JD1LSTWLSLMC101.dcsr.site,28001, port 1433 has
failed
So
spring.datasource.url=jdbc:sqlserver://JD1LSTWLSLMC101.dcsr.site:28001; . . .

SonarQube don't start

I'm trying to install SonarQube on my CentOS7 server.
When i try to access via web I see this:
web_page
It's supposed to show a SonarQube page right??
Here i put my logs and configs:
web.log:
2019.08.29 17:24:54 INFO web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2019.08.29 17:25:05 INFO web[][o.s.p.ProcessEntryPoint] Starting web
2019.08.29 17:25:05 INFO web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
2019.08.29 17:25:06 INFO web[][o.e.plugins] [Kurt Wagner] modules [], plugins [], sites []
2019.08.29 17:25:06 INFO web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
2019.08.29 17:25:06 INFO web[][o.s.s.p.LogServerVersion] SonarQube Server / 6.4.0.25310 / ad64a17b531c0e1f6fef0ce7e4d0d0b060977754
2019.08.29 17:25:06 INFO web[][o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://localhost/sonar
2019.08.29 17:25:06 ERROR web[][o.s.s.p.Platform] Web server startup failed
java.lang.IllegalStateException: Can not connect to database. Please check connectivity and settings (see the properties prefixed by 'sonar.jdbc.').
at org.sonar.db.DefaultDatabase.checkConnection(DefaultDatabase.java:108)
at org.sonar.db.DefaultDatabase.start(DefaultDatabase.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.invokeMethod(ReflectionLifecycleStrategy.java:110)
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.start(ReflectionLifecycleStrategy.java:89)
at org.sonar.core.platform.ComponentContainer$1.start(ComponentContainer.java:320)
at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84)
at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169)
at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132)
at org.picocontainer.behaviors.Stored.start(Stored.java:110)
at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:1016)
at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:1009)
at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:767)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143)
at org.sonar.server.platform.platformlevel.PlatformLevel.start(PlatformLevel.java:88)
at org.sonar.server.platform.Platform.start(Platform.java:231)
at org.sonar.server.platform.Platform.startLevel1Container(Platform.java:190)
at org.sonar.server.platform.Platform.init(Platform.java:86)
at org.sonar.server.platform.web.PlatformServletContextListener.contextInitialized(PlatformServletContextListener.java:43)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4727)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5189)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1409)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (FATAL: password authentication failed for user "sonar")
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)
at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:323)
at org.sonar.db.DefaultDatabase.checkConnection(DefaultDatabase.java:106)
... 30 common frames omitted
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "sonar"
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:451)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:223)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:211)
at org.postgresql.Driver.makeConnection(Driver.java:407)
at org.postgresql.Driver.connect(Driver.java:275)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
... 35 common frames omitted
sonar.log:
2019.08.29 17:28:37 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.08.29 17:28:37 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.08.29 17:28:37 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped
--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
2019.08.29 17:28:40 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2019.08.29 17:28:40 INFO app[][o.s.a.p.JavaProcessLauncherImpl] Launch process[es]: /usr/java/jdk1.8.0_131/jre/bin/java -Djava.awt.headless=true -Xmx1G -Xms256m -Xss256k -Djna.nosys=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonarqube/temp -cp ./lib/common/*:./lib/search/* org.sonar.search.SearchServer /opt/sonarqube/temp/sq-process6725369398987844378properties
2019.08.29 17:28:45 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2019.08.29 17:28:45 INFO app[][o.s.a.p.JavaProcessLauncherImpl] Launch process[web]: /usr/java/jdk1.8.0_131/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonarqube/temp -cp ./lib/common/*:./lib/server/*:/opt/sonarqube/lib/jdbc/postgresql/postgresql-9.4.1209.jre7.jar org.sonar.server.app.WebServer /opt/sonarqube/temp/sq-process3428525909039640490properties
systemctl status httpd:
[oksmart#CLOUDSVRSONAR01 logs]$ systemctl status httpd -l
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-08-29 16:59:00 CEST; 33min ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 5697 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
Main PID: 5834 (httpd)
Status: "Total requests: 50; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─5834 /usr/sbin/httpd -DFOREGROUND
├─5835 /usr/sbin/httpd -DFOREGROUND
├─5836 /usr/sbin/httpd -DFOREGROUND
├─5837 /usr/sbin/httpd -DFOREGROUND
├─5839 /usr/sbin/httpd -DFOREGROUND
├─5900 /usr/sbin/httpd -DFOREGROUND
├─5901 /usr/sbin/httpd -DFOREGROUND
├─5903 /usr/sbin/httpd -DFOREGROUND
├─5904 /usr/sbin/httpd -DFOREGROUND
├─5905 /usr/sbin/httpd -DFOREGROUND
└─5906 /usr/sbin/httpd -DFOREGROUND
Aug 29 16:59:00 CLOUDSVRSONAR01 systemd[1]: Starting The Apache HTTP Server...
Aug 29 16:59:00 CLOUDSVRSONAR01 httpd[5834]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using fe80::250:56ff:fe01:114a. Set the 'ServerName' directive globally to suppress this message
Aug 29 16:59:00 CLOUDSVRSONAR01 systemd[1]: Started The Apache HTTP Server.
/etc/httpd/conf.d/sonar.oksmart.es.conf :
<VirtualHost *:80>
ServerName localhost
# ProxyPreserveHost On
# ProxyPass / http://localhost:9000/
# ProxyPassReverse / http://localhost:9000/
TransferLog /var/log/httpd/sonar.oksmart.es_access.log
ErrorLog /var/log/httpd/sonar.oksmart.es_error.log
</VirtualHost>
NOTE: I commented the proxy options because if i comment out those lines, i get an error on the web page.
sonar.sh :
DEF_APP_NAME="SonarQube"
DEF_APP_LONG_NAME="SonarQube"
APP_NAME="${DEF_APP_NAME}"
APP_LONG_NAME="${DEF_APP_LONG_NAME}"
WRAPPER_CMD="./wrapper"
WRAPPER_CONF="../../conf/wrapper.conf"
PRIORITY=
PIDDIR="."
RUN_AS_USER=root
...
sonar.service:
[Unit]
Description=SonarQube service
After=syslog.target network.target
[Service]
Type=forking
ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop
User=root
Group=sonar
Restart=always
[Install]
WantedBy=multi-user.target
Any idea of what am i doing wrong??
Firewall is disabled.
Thanks all!
Looks like the web container failed to startup because of database authentication.
Besides that, you commented out your proxy config, that's why you're seeing default Apache homepage.

Flink HA Cluster JobManager issues

I have a setup with flink 1.2 cluster, made up of 3 JobManagers and 2 TaskManagers. I start the Zookeeper Quorum from JobManager1, I get confirmation Zookeeper starts on the other 2 JobManagers then I start a Flink job on this JobManager1.
The flink-conf.yaml is the same on all 5 VMs this means jobmanager.rpc.address: points to JobManager1 everywhere.
If I turn off the VM running JobManager1 I would expect Zookeeper to say one of the remaining JobManagers is the leader and the TaskManagers should reconnect to it. Instead I get in the TaskManagers' logs a lot of these messages
2017-03-14 14:13:21,827 INFO org.apache.flink.runtime.taskmanager.TaskManager - Trying to register at JobManager akka.tcp://flink#1.2.3.4:43660/user/jobmanager (attempt 11, timeout: 30 seconds)
2017-03-14 14:13:21,836 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:43660] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:43660]] Caused by: [Connection refused: /1.2.3.4:43660]
I modified the original IP to 1.2.3.4 for confidentiality and because it's always the same IP (of JobManager1).
More logs:
2017-03-15 10:28:28,655 INFO org.apache.flink.core.fs.FileSystem - Ensuring all FileSystem streams are closed for Async calls on Source: Custom Source -> Flat Map (1/1)
2017-03-15 10:28:38,534 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
2017-03-15 10:28:46,606 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:44779]] Caused by: [Connection refused: /1.2.3.4:44779]
2017-03-15 10:28:52,431 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:44779]] Caused by: [Connection refused: /1.2.3.4:44779]
2017-03-15 10:29:02,435 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:44779]] Caused by: [Connection refused: /1.2.3.4:44779]
2017-03-15 10:29:10,489 INFO org.apache.flink.runtime.taskmanager.TaskManager - TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink#1.2.3.4:44779/user/jobmanager: Old JobManager lost its leadership.
2017-03-15 10:29:10,490 INFO org.apache.flink.runtime.taskmanager.TaskManager - Cancelling all computations and discarding all cached data.
2017-03-15 10:29:10,491 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to fail task externally Source: Custom Source -> Flat Map (1/1) (75fd495cc6acfd72fbe957e60e513223).
2017-03-15 10:29:10,491 INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source -> Flat Map (1/1) (75fd495cc6acfd72fbe957e60e513223) switched from RUNNING to FAILED.
java.lang.Exception: TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink#1.2.3.4:44779/user/jobmanager: Old JobManager lost its leadership.
at org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:1074)
at org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleJobManagerLeaderAddress(TaskManager.scala:1426)
at org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:286)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:122)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2017-03-15 10:29:10,512 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Source: Custom Source -> Flat Map (1/1) (75fd495cc6acfd72fbe957e60e513223).
2017-03-15 10:29:10,515 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to fail task externally Flat Map (1/1) (dd555e0437867c3180a1ecaf0a9f4d04).
2017-03-15 10:29:10,515 INFO org.apache.flink.runtime.taskmanager.Task - Flat Map (1/1) (dd555e0437867c3180a1ecaf0a9f4d04) switched from RUNNING to FAILED.
java.lang.Exception: TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink#1.2.3.4:44779/user/jobmanager: Old JobManager lost its leadership.
at org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:1074)
at org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleJobManagerLeaderAddress(TaskManager.scala:1426)
at org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:286)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:122)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2017-03-15 10:29:10,516 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Flat Map (1/1) (dd555e0437867c3180a1ecaf0a9f4d04).
2017-03-15 10:29:10,516 INFO org.apache.flink.runtime.taskmanager.TaskManager - Disassociating from JobManager
2017-03-15 10:29:10,525 INFO org.apache.flink.runtime.blob.BlobCache - Shutting down BlobCache
2017-03-15 10:29:10,542 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:44779]] Caused by: [Connection refused: /1.2.3.4:44779]
2017-03-15 10:29:10,546 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Custom Source -> Flat Map (1/1) (75fd495cc6acfd72fbe957e60e513223).
2017-03-15 10:29:10,548 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Flat Map (1/1) (dd555e0437867c3180a1ecaf0a9f4d04).
2017-03-15 10:29:10,551 INFO org.apache.flink.core.fs.FileSystem - Ensuring all FileSystem streams are closed for Flat Map (1/1)
2017-03-15 10:29:10,552 INFO org.apache.flink.runtime.taskmanager.TaskManager - Trying to register at JobManager akka.tcp://flink#1.2.3.5:43893/user/jobmanager (attempt 1, timeout: 500 milliseconds)
2017-03-15 10:29:10,567 INFO org.apache.flink.core.fs.FileSystem - Ensuring all FileSystem streams are closed for Source: Custom Source -> Flat Map (1/1)
2017-03-15 10:29:10,632 INFO org.apache.flink.runtime.taskmanager.TaskManager - Successful registration at JobManager (akka.tcp://flink#1.2.3.5:43893/user/jobmanager), starting network stack and library cache.
2017-03-15 10:29:10,633 INFO org.apache.flink.runtime.taskmanager.TaskManager - Determined BLOB server address to be /1.2.3.5:42830. Starting BLOB cache.
2017-03-15 10:29:10,633 INFO org.apache.flink.runtime.blob.BlobCache - Created BLOB cache storage directory /tmp/blobStore-d97e08db-d2f1-4f00-a7d1-30c2f5823934
2017-03-15 10:29:15,551 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:44779]] Caused by: [Connection refused: /1.2.3.4:44779]
2017-03-15 10:29:20,571 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:44779]] Caused by: [Connection refused: /1.2.3.4:44779]
2017-03-15 10:29:25,582 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:44779]] Caused by: [Connection refused: /1.2.3.4:44779]
2017-03-15 10:29:30,592 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#1.2.3.4:44779] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink#1.2.3.4:44779]] Caused by: [Connection refused: /1.2.3.4:44779]
Does anyone know why the TaskManagers are not trying to reconnect to one of the remaining JobManagers (like 1.2.3.5 above)?
Thanks!
For everyone facing the same issue, HA requires you to provide a DFS location accessible from all nodes. I had backend state checkpoint directory and zookeeper storage directory pointing on each VM to a local filesystem location and when one of the JobManagers went down the new leader couldn't resume the running jobs because of lack of information / location not accessible.
Edit: Since this was asked, the file I modified (In the case of Apache Flink 1.2 (https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/config.html)) was
conf/flink-conf.yaml
I set
state.backend.fs.checkpointdir
high-availability.zookeeper.storageDir
to AWS S3 paths .accessible from both TaskManagers and JobManagers.

Resources