I am attempting to connect to Microsoft SQL Server using Sqoop. I have installed the JDBC driver from Microsoft by following the instructions for the Sqoop Connector and the JDBC Driver. Next I attempt to list the databases on the server. I have tried the following commands:
sqoop list-databases --connect 'jdbc:sqlserver://<HOST>' --username <USER> --password <PASS>
sqoop list-databases --connect 'jdbc:sqlserver://<HOST>;username=<USER>;password=<PASS>'
sqoop list-databases --connect 'jdbc:sqlserver://<HOST>;username=<USER>;password=<PASS>' --username <USER> --password <PASS>
Each of these commands produces the same error messages.
13/01/02 10:44:52 ERROR sqoop.ConnFactory: Error loading ManagerFactory information from file <MY SQOOP DIRECTORY>/conf/managers.d/mssqoop-sqlserver: java.io.IOException: the content of connector file must be in form of key=value
at org.apache.sqoop.ConnFactory.addManagersFromFile(ConnFactory.java:219)
at org.apache.sqoop.ConnFactory.loadManagersFromConfDir(ConnFactory.java:294)
at org.apache.sqoop.ConnFactory.instantiateFactories(ConnFactory.java:85)
at org.apache.sqoop.ConnFactory.<init>(ConnFactory.java:62)
at com.cloudera.sqoop.ConnFactory.<init>(ConnFactory.java:36)
at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:200)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:44)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
13/01/02 10:45:08 ERROR manager.CatalogQueryManager: Failed to list databases
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host <HOST>, port 1433 has failed. Error: "connect timed out. Verify the connection properties, check that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port, and that no firewall is blocking TCP connections to the port.".
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:171)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1033)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:817)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:700)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:842)
at java.sql.DriverManager.getConnection(DriverManager.java:579)
at java.sql.DriverManager.getConnection(DriverManager.java:221)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:665)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:56)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
I have connected to the database using Microsoft SQL Server Management Studio to ensure the database is operation and that that the host/username/password are all correct. Additionally I have ensured that the port is open and that MSSQL is on the other side with the following.
sudo nmap -sS -p 1433 <HOST>
Starting Nmap 5.21 ( http://nmap.org ) at 2013-01-02 11:04 PST
Nmap scan report for <HOST> (<HOST IP>)
Host is up (0.00070s latency).
rDNS record for <HOST IP: <HOST FQDN>
PORT STATE SERVICE
1433/tcp filtered ms-sql-s
Nmap done: 1 IP address (1 host up) scanned in 0.33 seconds
Any suggestions on where I should go from here? I have not been able to find any documentation on this error. Thanks
I am currently attempting to verify that the SQL server is reachable using OSQL from FreeTDS. Will update this post with my findings.
After much searching and talking to many people I have determined that this issue is being caused by the port being blocked. I am still not 100% certain why this error in particular occurred. If I try with an invalid username or password it will successfully tell me that they are invalid. It is just when it comes to making the actual query that the port is restricted. Most likely different ports are used.
Related
I have PostgreSQL running on Ubuntu Server and I want to make remote connection with PostgreSQL running on port 5432.
I've checked if I can ping the public IP of ubuntu server from my machine and that works fine.
Next I've changed two files on ubuntu server first I've changed postgresql.conf which looks as below
#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------
# - Connection Settings -
listen_addresses = '*' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
port = 5432 # (change requires restart)
Next I've added two lines in pg_hba.conf as below
host all all 0.0.0.0/0 trust
host all all ::/0 trust
Finally I checked if firewall is running by running sudo ufw verbose which outputted inactive.
As per my understanding I've allowed PostgreSQL to accept remote connection and firewall is also not present hence nothing is blocking. Still I get the following error.
psycopg2.OperationalError: connection to server at "XXX.XXX.XXX.XXX", port 5432 failed: Connection timed out (0x0000274C/10060)
Is the server running on that host and accepting TCP/IP connections?
How can I fix this error?
Edit
Although I can ping and ssh to the Ubuntu server using public IP but can not telnet.
I checked if port 5432 is open using this link but it turned out to be closed.
i'm facing the following problem.
I've got a PostgreSQL database on a remote server. In order to access it, i need to pass through a SSH Tunnel.
So, the steps of my test are:
create the tunneling
Connect to the db
perform query / queries
I'm able to create the tunnel by using SSHLibrary, and it works:
SSHLibrary.Open Connection 10.xxx.xxx.xx
SSHLibrary.Login MySSH_User MySSH_Password delay=1
SSHLibrary.Create Local SSH Tunnel 9191 ${IP_DATABASE} 22
So i go on by using DatabaseLibrary to perform the connection (and i've checked all the fields are right):
DatabaseLibrary.Connect To Database psycopg2 ${DB_NAME} ${USER_DB} ${PASSWORD_DB} ${IP_DATABASE} ${PORT_DB}
but i get the following error:
OperationalError: could not connect to server: Connection timed out (0x0000274C/10060)
Is the server running on host "10.xxx.xxx.xx" and accepting
TCP/IP connections on port 5432?
How can i solve it? I think i need to specify in some way that the database connection must be done by using the tunnel but i don't know how to do it.
Here's the complete test code:
*** Settings ***
Library DatabaseLibrary
Library SSHLibrary
*** Variables ***
${IP_DATABASE} 10.xxx.xxx.xx
${PORT_DB} 5432
${DB_NAME} MyNameDB
${SCHEMA_DATABASE} MySchemaDB
${USER_DB} MyUserDB
${PASSWORD_DB} MyPasswordDB
*** Keywords ***
*** Test Cases ***
Connect To DB Via SSH
SSHLibrary.Open Connection 10.xxx.xxx.xx
SSHLibrary.Login MySSH_User MySSH_Password delay=1
SSHLibrary.Create Local SSH Tunnel 9191 ${IP_DATABASE} 22
Sleep 2s
DatabaseLibrary.Connect To Database psycopg2 ${DB_NAME} ${USER_DB} ${PASSWORD_DB} ${IP_DATABASE} ${PORT_DB}
I would like to do it by using RF, Anyone can help?
Thank's a lot
Ssh tunnels should work in a way that that you establish SSH connection to another host and expose a PORT in localhost that will then tunnel a traffic from that port in localhost to a IP/PORT in the end point of the tunnel ..
So, lets say you have a host at 192.168.100.10 where your robotframework is running and it needs to connect to a database in 192.168.50.100:
If the postgres is accessible via ssh directly:
Connect To DB Via SSH
${IP_DATABASE}= 192.168.50.100
${DB_PORT}= 5432
SSHLibrary.Open Connection ${IP_DATABASE}
SSHLibrary.Login MySSH_User MySSH_Password delay=1
SSHLibrary.Create Local SSH Tunnel ${DB_PORT} ${IP_DATABASE} ${DB_PORT}
DatabaseLibrary.Connect To Database psycopg2 ${DB_NAME} ${USER_DB} ${PASSWORD_DB} 127.0.0.1 ${PORT_DB}
Now, if you have to use ssh jump host, eg, connect to a ssh server that will then connect to another server where postgres is running, all you need to do is change the tunnel ip address where the traffic will be forward to:
Connect To DB Via SSH
${IP_DATABASE}= 192.168.50.100
${IP_SSH_JUMPHOST}= 192.168.50.1
${DB_PORT}= 5432
SSHLibrary.Open Connection ${IP_JUMPHOST}
SSHLibrary.Login MySSH_User MySSH_Password delay=1
SSHLibrary.Create Local SSH Tunnel ${DB_PORT} ${IP_DATABASE} ${DB_PORT}
DatabaseLibrary.Connect To Database psycopg2 ${DB_NAME} ${USER_DB} ${PASSWORD_DB} 127.0.0.1 ${PORT_DB}
I am trying to import data from SQL Server to Google Cloud Storage which I will later on upload to BigQuery. I am doing all of this through Google's Cloud Shell.
I have done the initial steps of downloading Sqoop and Sql server JDBC file and downloading and then uploading it to a specific google cloud storage. I have also created a Google Dataproc cluster to submit the Sqoop job but when I try to use the submission code it throws me a couple of error.
I am following this process (https://medium.com/datamindedbe/import-sql-server-data-in-bigquery-d640441d5d56), in my case I am trying to extract one table first.Code to submit a job through dataproc
WHAT I HAVE TRIED
I do have the SQL server jdbc .jar (mssql-jdbc-8.2.1.jre8.jar) file
in cloud storage with other dependent files
I have also checked my TCP/IP connection in my SQL Server 2014 and it
is in the recommended state as the error suggests
CODE I USED TO SUBMIT A SQOOP JOB TO DATAPROC CLUSTER
CLUSTERNAME="sqoop-cluster"
BUCKET="gs://sqoop-bucket-20092021"
libs=`gsutil ls $BUCKET/jars | paste -sd, --`
JDBC_STR="jdbc:sqlserver://RUKSQLRS01:1433;databaseName=RUKDataWarehouse"
SQL_USER="RUKSQLDataWarehouse_Reporting"
SQL_PASS="gs://sqoop-bucket-20092021/creds/sqoop.password"
TABLE="LBD_Task"
SCHEMA="dbo"
gcloud dataproc jobs submit hadoop \
--region europe-west2 \
--cluster="$CLUSTERNAME"\
--jars=$libs \
--class=org.apache.sqoop.Sqoop \
-- \
import \
-Dorg.apache.sqoop.splitter.allow_text_splitter=true \
-Dmapreduce.job.user.classpath.first=true \
--connect "$JDBC_STR" \
--username "$SQL_USER" \
--password-file "$SQL_PASS" \
--table "$SCHEMA.$TABLE" \
--warehouse-dir "$BUCKET/output/$TABLE" \
--num-mappers 1 \
--as-avrodatafile
ERROR I AM GETTING
21/09/22 11:30:46 WARN tool.SqoopTool: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
21/09/22 11:30:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
21/09/22 11:30:48 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
21/09/22 11:30:48 INFO manager.SqlManager: Using default fetchSize of 1000
21/09/22 11:30:48 INFO tool.CodeGenTool: Beginning code generation
21/09/22 11:31:02 ERROR manager.SqlManager: Error executing statement: com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host RUKSQLRS01, port 1433 has failed. Error: "RUKSQLRS01. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host RUKSQLRS01, port 1433 has failed. Error: "RUKSQLRS01. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:227)
at com.microsoft.sqlserver.jdbc.SQLServerException.ConvertConnectExceptionToSQLServerException(SQLServerException.java:284)
at com.microsoft.sqlserver.jdbc.SocketFinder.findSocket(IOBuffer.java:2435)
at com.microsoft.sqlserver.jdbc.TDSChannel.open(IOBuffer.java:635)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2010)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1687)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1528)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:866)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:569)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:904)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1872)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1671)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunClassShim.main(HadoopRunClassShim.java:19)
21/09/22 11:31:02 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1677)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunClassShim.main(HadoopRunClassShim.java:19)
This seems to be a networking issue. Your SQL server is outside of GCP and you are trying to access it through hostname. You need to use either external IP and setup firewall rules on the SQL server side to allow accessing from GCP, or setup VPN between your GCP VPC network and the SQL server's network and access SQL server through internal IP.
This is my docker-compose.yml file:
version: "3.9"
services:
api:
# configuration of API here
db:
image: mcr.microsoft.com/mssql/server
restart: always
container_name: Database
volumes:
- ./Data:/var/opt/mssql
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=lksU2o412f7tBj58t07B
- MSSQL_PID=Express
ports:
- 1433:1433
command: >
sh -c
"
tail -f /dev/null
"
As you can see, I'm using the official docker image for MS SQL Server, mcr.microsoft.com/mssql/server.
My laptop is Ubuntu 20.04 LTS.
I run docker-compose up -d and then docker exec -it db sh to get interactive shell.
Then I write this command:
/opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P 'lksU2o412f7tBj58t07B'
And I get this response:
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : TCP Provider: Error code 0x2749.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online..
This is the official image and I have not changed anything.
Also in my api container, I can ping db or Database. But I can't telnet port 1433.
Also from my host machine I can telnet localhost 1433, but I get this response:
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
This is a terrible experience in working with containers from Microsoft. Aren't containers supposed to work out of the box? Isn't container technology there only to solve these stupid problems?
What should I do?
Ok, here's the issue: I want to upload a DB I've got locally and I'm using in localhost, but, when I use:
PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U root app_db > app_db.dump
I always get:
pg_dump: [archiver (db)] connection to database "app_db" failed: could not connect to server: Connection refused
Is the server running on host "localhost" and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" and accepting
TCP/IP connections on port 5432?
I think the problem is that I'm writing localhost instead of my heroku app address, can anyone tell me how can I get the correct address?
You need to have postgresql installed and running