Overview
I learn to migrate an Amazon RDS for Oracle Database to Amazon Redshift referring this tutorial https://docs.aws.amazon.com/dms/latest/sbs/CHAP_RDSOracle2Redshift.html
Trouble
I had a trouble the below question.
I would like migrate Oralce DB to Amazon Redshift with AWS SCT
I managed to get out of trouble with the help of people. However, I had a new trouble.
I try to connect to Amazon Redshift with AWS SCT. AWS Redshift failed to connect.
images
I saw the log file. An error was recorded in the log.
2020-12-31 20:56:16.358 [ 78] LOADER ERROR Connection to 'jdbc:redshift://oracletoredshiftdwusingdms-redshiftcluster-1dll5wg4tqddk.cnxpo3loreqp.us-east-1.redshift.amazonaws.com:5439/test' wasn't established. ERROR: code: 500150; message: [Amazon](500150) Error setting/closing connection: Connection timed out: connect.
2020-12-31 20:56:16.358 [ 78] LOADER ERROR Error chain:
[Amazon](500150) Error setting/closing connection: Connection timed out: connect.
------------------------------
java.sql.SQLException: [Amazon](500150) Error setting/closing connection: Connection timed out: connect.
at com.amazon.redshift.client.PGClient.connect(Unknown Source)
at com.amazon.redshift.client.PGClient.<init>(Unknown Source)
at com.amazon.redshift.core.PGJDBCConnection.connect(Unknown Source)
at com.amazon.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.amazon.jdbc.common.AbstractDriver.connect(Unknown Source)
at com.amazon.sct.dbloader.JdbcDriverAdapter.connect(JdbcDriverAdapter.java:30)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
at com.amazon.sct.dbloader.DbLoaderDataSource.testConnection(DbLoaderDataSource.java:109)
at com.amazon.sct.dbloader.SqlLoaderEngine.testConnection(SqlLoaderEngine.java:1706)
at com.amazon.sct.dbloader.DbLoader.checkConnection(DbLoader.java:706)
at com.amazon.sct.dbloader.DbLoader.connect(DbLoader.java:406)
at com.amazon.sct.dbloader.DbLoaderContainer.checkAccessibility(DbLoaderContainer.java:30)
at com.amazon.sct.task.TestConnectionTask.call(TestConnectionTask.java:40)
at com.amazon.sct.task.TestConnectionTask.call(TestConnectionTask.java:18)
at javafx.concurrent.Task$TaskCallable.call(Task.java:1425)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Caused by: com.amazon.support.exceptions.GeneralException: [Amazon](500150) Error setting/closing connection: Connection timed out: connect.
... 17 more
Caused by: java.net.ConnectException: Connection timed out: connect
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:476)
at java.base/sun.nio.ch.Net.connect(Net.java:468)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:694)
at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
at com.amazon.redshift.client.PGClient.connect(Unknown Source)
at com.amazon.redshift.client.PGClient.<init>(Unknown Source)
at com.amazon.redshift.core.PGJDBCConnection.connect(Unknown Source)
at com.amazon.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.amazon.jdbc.common.AbstractDriver.connect(Unknown Source)
at com.amazon.sct.dbloader.JdbcDriverAdapter.connect(JdbcDriverAdapter.java:30)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
at com.amazon.sct.dbloader.DbLoaderDataSource.testConnection(DbLoaderDataSource.java:109)
at com.amazon.sct.dbloader.SqlLoaderEngine.testConnection(SqlLoaderEngine.java:1706)
at com.amazon.sct.dbloader.DbLoader.checkConnection(DbLoader.java:706)
at com.amazon.sct.dbloader.DbLoader.connect(DbLoader.java:406)
at com.amazon.sct.dbloader.DbLoaderContainer.checkAccessibility(DbLoaderContainer.java:30)
at com.amazon.sct.task.TestConnectionTask.call(TestConnectionTask.java:40)
at com.amazon.sct.task.TestConnectionTask.call(TestConnectionTask.java:18)
at javafx.concurrent.Task$TaskCallable.call(Task.java:1425)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-31 20:56:16.359 [ 78] GENERAL INFO Test connecting to Amazon Redshift database finished in 0:00:21.113 sec with memory consumption of 516.25 MB (497.79 MB .. 516.25 MB).
2020-12-31 20:56:16.359 [ 78] GENERAL INFO Test connecting to Amazon Redshift database statistics:
GENERAL: 0:00:21.113 sec
2020-12-31 20:56:16.365 [ 19] GENERAL ERROR com.amazon.sct.dbloader.DbLoaderHandledException: Connection wasn't established. Check connection properties.
java.util.concurrent.ExecutionException: com.amazon.sct.dbloader.DbLoaderHandledException: Connection wasn't established. Check connection properties.
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at com.amazon.sct.task.launcher.CommonTaskLauncher.lambda$run$0(CommonTaskLauncher.java:39)
at com.sun.javafx.binding.ExpressionHelper$Generic.fireValueChangedEvent(ExpressionHelper.java:360)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:80)
at javafx.beans.property.ObjectPropertyBase.fireValueChangedEvent(ObjectPropertyBase.java:106)
at javafx.beans.property.ObjectPropertyBase.markInvalid(ObjectPropertyBase.java:113)
at javafx.beans.property.ObjectPropertyBase.set(ObjectPropertyBase.java:147)
at javafx.concurrent.Task.setState(Task.java:698)
at javafx.concurrent.Task$TaskCallable.lambda$call$2(Task.java:1455)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
at com.sun.glass.ui.win.WinApplication._enterNestedEventLoopImpl(Native Method)
at com.sun.glass.ui.win.WinApplication._enterNestedEventLoop(WinApplication.java:201)
at com.sun.glass.ui.Application.enterNestedEventLoop(Application.java:509)
at com.sun.glass.ui.EventLoop.enter(EventLoop.java:107)
at com.sun.javafx.tk.quantum.QuantumToolkit.enterNestedEventLoop(QuantumToolkit.java:635)
at javafx.stage.Stage.showAndWait(Stage.java:465)
at javafx.scene.control.HeavyweightDialog.showAndWait(HeavyweightDialog.java:162)
at javafx.scene.control.Dialog.showAndWait(Dialog.java:346)
at com.amazon.sct.handler.CreateConnectionHandler.createConnection(CreateConnectionHandler.java:51)
at com.amazon.sct.viewmodel.AppViewModel.createConnection(AppViewModel.java:1406)
at com.amazon.sct.viewmodel.AppViewModel.createTargetConnection(AppViewModel.java:774)
at com.amazon.sct.view.AppView.loadTarget(AppView.java:348)
at com.amazon.sct.view.AppView.lambda$initConnectionMenusBindings$27(AppView.java:1077)
at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)
at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)
at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)
at javafx.event.Event.fireEvent(Event.java:198)
at javafx.scene.Scene$MouseHandler.process(Scene.java:3862)
at javafx.scene.Scene.processMouseEvent(Scene.java:1849)
at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2590)
at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:409)
at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:299)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:447)
at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:412)
at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:446)
at com.sun.glass.ui.View.handleMouseEvent(View.java:556)
at com.sun.glass.ui.View.notifyMouse(View.java:942)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.amazon.sct.dbloader.DbLoaderHandledException: Connection wasn't established. Check connection properties.
at com.amazon.sct.dbloader.DbLoader.checkConnection(DbLoader.java:721)
at com.amazon.sct.dbloader.DbLoader.connect(DbLoader.java:406)
at com.amazon.sct.dbloader.DbLoaderContainer.checkAccessibility(DbLoaderContainer.java:30)
at com.amazon.sct.task.TestConnectionTask.call(TestConnectionTask.java:40)
at com.amazon.sct.task.TestConnectionTask.call(TestConnectionTask.java:18)
at javafx.concurrent.Task$TaskCallable.call(Task.java:1425)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
... 1 more
What I try
[Amazon](500150) Error setting/closing connection: Connection timed out: connect.
I think the above error is similar to the issues.
So, I launched the Security Groups console and add [Rule: Type=Redshift, Source=MyIP].
I saved it and I try to connect again.
Unfortunately, AWS Redshift failed to connect with same error.
I have no idea to do next time.
What I should do?
To troubleshoot this issue, you can use following steps;
(1) To troubleshoot faster, instead of your java application, you can use ping command to check network reachability from your host/machine/laptop/desktop.
(2) If step one is successful, then try following commands on your host/machine/laptop/desktop using Windows command prompt or Windows PowerShell or Any Linux shell/command line tool;
nc -vz -w 3 "${DB_HOST}" "${DB_PORT}"
OR
telnet "${DB_HOST}" "${DB_PORT}"
(3) If commands from step (1) or (2) are failing with "Connection timed out" error, then
(3.1) Use VPC Reachability Analyzer : Create and analyze path with Source type Network Interfaces and Source as Network Interface Id of the machine where you application is deployed. Furthermore, select Destination type as Network Interfaces and Destination as network Interface Id of the AWS Red Shift. Put Destination port as 5439, keep Protocol as TCP.
Note: It takes few minutes for completion of this path analysis.
OR
(3.2) If you have enabled VPC Flow Logs, then you can trace vpc flow logs to check, which AWS resource is rejecting the network traffic. For VPC flow logs information please refer this document from AWS.
Related
I am running Nifi on windows machine and would like to establish a connection to the MS SQL Server on the same machine.
I downloaded the JDBC driver from Microsoft and put mssql-jdbc-11.2.1.jre11.jar to the lib folder of Nifi. Below is the properties for Controller Service Detail in Nifi to connect to the Sql server.
Database Connection URL: jdbc:sqlserver://127.0.0.1;databaseName=test_database;encrypt=true;user=nifireader;password=***;
Database Driver Class Name: com.microsoft.sqlserver.jdbc.SQLServerDriver
I also created a user inside Sql Server and gave it a read access to use it for Nifi. But I get the SSL error and could not solve it so far. Is there any other steps that I need to take to establish a connection?
2022-12-09 17:47:05,052 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=f76ca380-0184-1000-3a89-b878a90723ed] Unable to execute SQL select query SELECT * FROM tbl_tags due to org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create PoolableConnectionFactory (The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:7fcd7f53-9532-44da-96ce-56eac78049f6). No FlowFile to route to failure
org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create PoolableConnectionFactory (The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:7fcd7f53-9532-44da-96ce-56eac78049f6)
at org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:573)
at org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:550)
at org.apache.nifi.dbcp.DBCPService.getConnection(DBCPService.java:55)
at jdk.internal.reflect.GeneratedMethodAccessor111.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
at com.sun.proxy.$Proxy85.getConnection(Unknown Source)
at org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:255)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1356)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:7fcd7f53-9532-44da-96ce-56eac78049f6)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:653)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:531)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:731)
at org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:560)
... 20 common frames omitted
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:7fcd7f53-9532-44da-96ce-56eac78049f6
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:3806)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1906)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:3329)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:2950)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:2790)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1663)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1064)
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:374)
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:106)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:649)
... 23 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:369)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:312)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:478)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:456)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:199)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1382)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1295)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:416)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:388)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1795)
... 32 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:233)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:110)
at com.microsoft.sqlserver.jdbc.HostNameOverrideX509TrustManager.checkServerTrusted(SQLServerTrustManager.java:86)
at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1510)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
... 44 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 52 common frames omitted
I have a Spring Boot rest service which retrieves data from a SQL Server Database. It runs fine on local but on deploying to the Cloud environment, the POD fails to come up and we are seeing certificate issues, I see the below logs:
[main] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Exception during pool initialization.
com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:<random generated id>
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:3680)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:2113)
.............................................
.............................................
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
.............................................
.............................................
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:2021)
... 58 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
.............................................
.............................................
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSChannel$HostNameOverrideX509TrustManager.checkServerTrusted(IOBuffer.java:1702)
at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(Unknown Source)
... 71 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
... 79 more
The connection string: jdbc:sqlserver://<HOST_NAME>:<PORT_NAME>;instanceName=<INSTANCE_NAME>;databaseName=<DB_NAME>;
I have tried adding encrypt=true; trustServerCertificate=true; and trustServerCertificate=false; as suggested on different posts but it still fails.
I have seen this error for different URLs in the past and the fix would be to download the Certificate (using the browser) and adding it to the Java KeyStore but I am not sure what to do for Databases. Also, we wouldn't have access to the Java Keystore on the cloud env.
Can someone help/guide on how to fix this issue?
Thanks in advance.
I am trying to import data from SQL Server to Google Cloud Storage which I will later on upload to BigQuery. I am doing all of this through Google's Cloud Shell.
I have done the initial steps of downloading Sqoop and Sql server JDBC file and downloading and then uploading it to a specific google cloud storage. I have also created a Google Dataproc cluster to submit the Sqoop job but when I try to use the submission code it throws me a couple of error.
I am following this process (https://medium.com/datamindedbe/import-sql-server-data-in-bigquery-d640441d5d56), in my case I am trying to extract one table first.Code to submit a job through dataproc
WHAT I HAVE TRIED
I do have the SQL server jdbc .jar (mssql-jdbc-8.2.1.jre8.jar) file
in cloud storage with other dependent files
I have also checked my TCP/IP connection in my SQL Server 2014 and it
is in the recommended state as the error suggests
CODE I USED TO SUBMIT A SQOOP JOB TO DATAPROC CLUSTER
CLUSTERNAME="sqoop-cluster"
BUCKET="gs://sqoop-bucket-20092021"
libs=`gsutil ls $BUCKET/jars | paste -sd, --`
JDBC_STR="jdbc:sqlserver://RUKSQLRS01:1433;databaseName=RUKDataWarehouse"
SQL_USER="RUKSQLDataWarehouse_Reporting"
SQL_PASS="gs://sqoop-bucket-20092021/creds/sqoop.password"
TABLE="LBD_Task"
SCHEMA="dbo"
gcloud dataproc jobs submit hadoop \
--region europe-west2 \
--cluster="$CLUSTERNAME"\
--jars=$libs \
--class=org.apache.sqoop.Sqoop \
-- \
import \
-Dorg.apache.sqoop.splitter.allow_text_splitter=true \
-Dmapreduce.job.user.classpath.first=true \
--connect "$JDBC_STR" \
--username "$SQL_USER" \
--password-file "$SQL_PASS" \
--table "$SCHEMA.$TABLE" \
--warehouse-dir "$BUCKET/output/$TABLE" \
--num-mappers 1 \
--as-avrodatafile
ERROR I AM GETTING
21/09/22 11:30:46 WARN tool.SqoopTool: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
21/09/22 11:30:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
21/09/22 11:30:48 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
21/09/22 11:30:48 INFO manager.SqlManager: Using default fetchSize of 1000
21/09/22 11:30:48 INFO tool.CodeGenTool: Beginning code generation
21/09/22 11:31:02 ERROR manager.SqlManager: Error executing statement: com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host RUKSQLRS01, port 1433 has failed. Error: "RUKSQLRS01. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host RUKSQLRS01, port 1433 has failed. Error: "RUKSQLRS01. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:227)
at com.microsoft.sqlserver.jdbc.SQLServerException.ConvertConnectExceptionToSQLServerException(SQLServerException.java:284)
at com.microsoft.sqlserver.jdbc.SocketFinder.findSocket(IOBuffer.java:2435)
at com.microsoft.sqlserver.jdbc.TDSChannel.open(IOBuffer.java:635)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2010)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1687)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1528)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:866)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:569)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:904)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1872)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1671)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunClassShim.main(HadoopRunClassShim.java:19)
21/09/22 11:31:02 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1677)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunClassShim.main(HadoopRunClassShim.java:19)
This seems to be a networking issue. Your SQL server is outside of GCP and you are trying to access it through hostname. You need to use either external IP and setup firewall rules on the SQL server side to allow accessing from GCP, or setup VPN between your GCP VPC network and the SQL server's network and access SQL server through internal IP.
I am trying to implement solrcloud.I foollowed doc from official resource https://cwiki.apache.org/confluence/display/solr/Getting+Started+with+SolrCloud .It works fine with embeded zookeper but it is recomended to use external zookeper. I insalled zookeper on my system created data dictionary zookeper on my home folder.I created sub folders named 1 and 2 and created myid file with text 1 and two respectively i each folder as mentioned in doc.I created config files for zookeper zoo.cnfg
clientPort=2181
initLimit=5
syncLimit=2
server.1=localhost:2879:3879
server.2=localhost:2888:3888
and zoo2.cnfg
initLimit=5
syncLimit=2
clientPort=2182
server.1=localhost:2878:3878
server.2=localhost:2888:3888
Next I run cd
bin/zkServer.sh start zoo.cfg
bin/zkServer.sh start zoo2.cfg
And its started sucessfully. next I run
bin/solr start -e cloud -z localhost:2181,localhost:2182
system ask me no of shards etc like in getting started i select port for node1 8990 and for node 2 8991. It gives error
Waiting to see Solr listening on port 8991 [/] Still not seeing Solr listening on 8991 after 30 seconds!
WARN - 2015-10-30 09:47:04.827; [ ] org.apache.zookeeper.ClientCnxn$SendThread; Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
WARN - 2015-10-30 09:47:05.929; [ ] org.apache.zookeeper.ClientCnxn$SendThread; Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
WARN - 2015-10-30 09:47:06.030; [ ] org.apache.zookeeper.ClientCnxn$SendThread; Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
WARN - 2015-10-30 09:47:07.131; [ ] org.apache.zookeeper.ClientCnxn$SendThread; Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
WARN - 2015-10-30 09:47:07.232; [ ] org.apache.zookeeper.ClientCnxn$SendThread; Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Where I am missing ? gone through many docs but apche doc is not proper for external zookeper setup.
Your Zookeeper ensemble must have an impair number of nodes : 1, 3, 5, etc...
If you want to test ZK clustering feature than you have to set up at least 3 ZK instances. In this case, don't forget :
To set correctly the ZK server id in the file myid, that must be created in the directory dataDir, referenced by your zoo.cfg.
Separate the dataDir and dataLogDir for each ZK instance.
We have installed spring-xd version 1.1.0 and trying to stream data from sql server via JDBC driver using xd-singlenode. After deploying the stream, it is giving us error that:
port 1433/databasename is not valid
For testing purpose we also fetched data from postgressql and it is connecting to the database.
We have created stream as:
stream create connectiontest --definition "jdbc --url=jdbc:sqlserver://databaseIP:1433/Databasename
--driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver --username=sample
--password=***** --query= 'SELECT * FROM schema.tablename' |log" --deploy
Error is:
1.1.0.RELEASE ERROR task-scheduler-3 handler.LoggingHandler - org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: The port number 1433/Databasename is not valid.
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:245)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:457)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy127.call(Unknown Source)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:298)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:292)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The port number 1433/Databasename is not valid.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:190)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:691)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1012)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:307)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:200)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:699)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:633)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:186)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:204)
... 21 more
Any help will be much appreciated.
This was answered in the GitHub issue here - https://github.com/spring-projects/spring-xd/issues/1521
If you need to add a database name as a property in the JDBC URL you need to enclose the URL in single quotes like:
--url='jdbc:sqlserver://databaseIP:1433;database=Database_name'