Docker Container: Unable to connect to SQL Server by name - sql-server

I created two container on the same network and one of them as a Sql Server instance running. In the other container (with SQL Tools) i'm able to connect to the SQL using the IP Address, but if i swith to machine name it fails.
I already tried do ping the machine and the dns is solving the right IP, i also tried dnslookup and it's also working. Does anyone as a clue on how to fix this?
Full test scenario:
Created new network
docker network create --driver=bridge specsnet
Run SQL Container
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Password!123456' -p 1401:1433 -d --name=TestDBServer --net=specsnet --rm microsoft/mssql-server-linux:latest
Run New Container with SQL Tools (to test connection)
docker run -it --net=specsnet --rm --entrypoint "bash" mcr.microsoft.com/mssql-tools:latest
Loaded some tools for troubleshooting (into SQL Tools container)
apt-get update | apt-get install telnet -y | apt-get install iputils-ping -y | apt-get install dnsutils -y | apt-get install nmap -y | apt-get install nano -y
Test Connection with IP (Success - IP was 172.18.0.2)
sqlcmd -S tcp:172.18.0.2,1433 -U sa -P 'Password!123456'
Test Connection with Name (Fails)
sqlcmd -S tcp:TestDBServer,1433 -U sa -P 'Password!123456'

So as Bjoern suggested i created a docker compose file and after doing some test i realized the issue was not fixed.
Then i started to manipulate the file, tweaking the properties, and discovered the problem was on the SQL container name (the container name had upper case letters). I set the SQL container name to "testdbserver" and everything worked fine.
Docker Compose File
version: '2'
services:
testdbserver:
image: microsoft/mssql-server-linux:latest
ports:
["1401:1433"]
environment:
SA_PASSWORD: Password!123456
ACCEPT_EULA: Y
networks:
- specsnet
sqltools:
image: mcr.microsoft.com/mssql-tools:latest
depends_on:
- testdbserver
networks:
- specsnet
networks:
specsnet:
driver: bridge
ipam:
config:
- subnet: 10.5.0.0/16
gateway: 10.5.0.1
Start SQL Tools Container on Bash Mode
docker-compose run sqltools bash
Execute SQL Test Connection (works now)
sqlcmd -S tcp:testdbserver,1433 -U sa -P 'Password!123456'

The TestDBServer part of your sqlcmd references a so-called Server Alias. These server aliases are not accessible due to the way you have set up the Docker networking currently. If you switch to Docker compose to set up the networking for you it should work.
An alternative approach would be to --link the containers in question together.

I had the same issue when using a gateway and a microservice:
Create network:
docker network create mynet
Add containers:
docker network connect mynet my-gateway
docker network connect mynet my_service_8080
Doing so, I wasn't able to communicate between both containers when using container name. E.g. when connecting to the gateway container and doing a curl to the service:
curl -i http://my_service_8080:8080/management/info
HTTP/1.1 400
Transfer-Encoding: chunked
Date: Tue, 23 Feb 2021 12:51:47 GMT
Connection: close
After doing a lot of research on the net which led to nothing, I finally had the idea to provide an alias name:
docker network connect --alias gateway mynet my-gateway
docker network connect --alias myservice mynet my_service_8080
Now, I can successfully communicate between the containers by using alias name as host name:
curl -i http://myservice:8080/management/info
HTTP/1.1 200
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/vnd.spring-boot.actuator.v2+json;charset=UTF-8
Transfer-Encoding: chunked
Date: Tue, 23 Feb 2021 12:54:36 GMT
{"name":"myservice-app","description":"my service","version":"20210202"}
EDIT:
After some more digging, I found that communication using container names doesn't work if the name (or alias) contains an underline character. So if you experience problems, check your container names.

Related

What configuration should I provide in docker-compose.yml to allow a spring boot docker container to connect to a remote database?

I try to start 2 containers with the following docker compose file:
version: '2'
services:
client-app:
image: client-app:latest
build: ./client-app/Dockerfile
volumes:
- ./client-app:/usr/src/app
ports:
- 3000:8000
spring-boot-server:
build: ./spring-boot-server/Dockerfile
volumes:
- ./spring-boot-server:/usr/src/app
ports:
- 7000:7000
The spring boot server tries to connect to a remote database server which is on another host and network. Docker successfully starts the client-app containers but fails to start the spring-boot-server. This log is showing that the server crashed because it has failed to connect to the remote database:
2021-01-25 21:02:28.393 INFO 1 --- [main] com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Starting...
2021-01-25 21:02:29.553 ERROR 1 --- [main] com.zaxxer.hikari.pool.HikariPool: HikariPool-1 - Exception during pool initialization.
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
The Dockerfiles of both containers create valid images by which I can run manually the containers. It looks like there are some default network restrictions on containers started by a composing file.
Docker compose version running on Ubuntu:
docker-compose version 1.8.0, build unknown
=============================================
FURTHER INVESTIGATIONS:
I had created a Dockerfile
FROM ubuntu
RUN apt-get update
RUN apt-get install -y mysql-client
CMD mysql -A -P 3306 -h 8.8.8.8 --user=root --password=mypassword -e "SELECT VERSION()" mydatabase
along with a docker-compose.yml
version: '2'
services:
test-remote-db-compose:
build: .
ports:
- 8000:8000
to test aside the connectivity alone with the remote database. The test passed with success.
The problem has been misteriously solved, after doing this , a host mashine reboot and docker-compose up --build.

Cannot access to dockerized SQL Server instance

I have simple docker-compose.yml which contains two services only, my-api and sql-server.
version: '3.0'
services:
sql-server:
image: mcr.microsoft.com/mssql/server:2019-latest
hostname: sql-server
container_name: sql-server
ports:
- "1433:1433"
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=MyPassword01*
- MSSQL_PID=Express
my-api:
ports:
- "8080:5000"
depends_on:
- sql-server
... ommited for clarity
When I docker-compose up --build the containers are ready (I can verify with docker ps)
a7a47b89a17a mcr.microsoft.com/mssql/server:2019-latest "/opt/mssql/bin/perm…" 12 minutes ago Up 11 minutes 0.0.0.0:1433->1433/tcp sql-server
but I cannot access my SQL Server using SSMS.
SSMS login window:
Server Name: localhost,1433
Authentication: SQL Server Authentication
Username: sa
Password: MyPassword01*
Error:
Cannot connect to localhost,1433.
Login failed for user 'sa'. (.Net SqlClient Data Provider)
PS: I also tried with
Server Name: sql-server,1433
but still cannot access
Execute the below code which will display the public ipaddress.
Instead of localhost use this ipaddress
docker inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}" mssqltrek-con1.
I followed the below link to achieve the same.
https://www.sqlshack.com/sql-server-with-a-docker-container-on-windows-server-2016/
I guess you have to debug further. The first thing I guess you can do is to open the bash inside the container and try to connect to your SQL database from the container.
docker exec -it sql-server "bash"
Once inside the container bash, then connect with sqlcmd,
/opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "<YourNewStrong#Passw0rd>"
If you fail to connect inside the container, then I have to assume your SA password is somehow different when you set up the SQL server. But if this is not an issue, that means, you can connect to SQL server inside the container, then you can be rest assured it is something wrong with SSMS on port 1433 in your computer. Make sure the server name is correct.

How can I allow connections by specifying docker-compose host names in postgres's pg_hba.conf file?

I'm trying to allow a connection from one Docker container to a postgres container by specifying the host name of the client container in the server's pg_hba.conf file. Postgres's documentation indicates that a host name can be specified, rather than an IP address. Since I'm using Docker Compose to start the two containers, they should be accessible to each other by container name using Docker Compose's DNS. I don't want to open up all IP addresses for security reasons, and when I eventually add access for additional containers, it will be much easier to just specify the container name in the pg_hba.conf file rather than assign static IP addresses to each of them. However, when I attempt to do this, it fails with a message such as this:
psql: FATAL: no pg_hba.conf entry for host "192.168.208.3", user "postgres", database "postgres", SSL off
Here's a minimum reproducible example of what I'm trying to do:
I use the following Docker Compose file:
version: '3'
services:
postgresdb:
image: postgres:9.4
container_name: postgres-server
ports:
- "5432:5432"
volumes:
- "postgres-data:/var/lib/postgresql/data"
postgres-client:
image: postgres:9.4
container_name: postgres-client
depends_on:
- postgres-server
volumes:
postgres-data:
After running docker-compose up, I exec into the server container and modify the pg_hba.conf file in /var/lib/postgresql/data to look like this:
host all postgres postgres-client trust
I then restart the postgres server (docker-compose down then docker-compose up) and it loads the modified pg_hba.conf from the mounted volume.
I exec into the client container and attempt to connect to the postgres server:
docker exec -it postgres-client /bin/bash
psql -U postgres -h postgres-server postgres
This is where I get an error such as the following:
psql: FATAL: no pg_hba.conf entry for host "192.168.208.3", user "postgres", database "postgres", SSL off
I can't seem to find anything online that shows how to get this working. I've found examples where they just open up all or a range of IP addresses, but none where they get the use of a host name working. Here are some related questions and information:
https://www.postgresql.org/docs/9.4/auth-pg-hba-conf.htm
Allow docker container to connect to a local/host postgres database
https://dba.stackexchange.com/questions/212020/using-host-names-in-pg-hba-conf
Any ideas on how to get this working the way I would expect it to work using Docker Compose?
You need to add the full qualified host name of the client container in pg_hba.conf.
host all postgres postgres-client.<network_name> trust
e.g:
host all postgres postgres-client.postgreshostresolution_default trust
If no network has been defined, network_name is <project_name>_default.
By default project_name is the folder the docker-compose.yml resides.
To get the network names you may also call
docker inspect postgres-client | grep Networks -A1
or
docker network ls
to get a list of all docker networks currently defined on your docker host

Kafka-Connect For MSSQL Invalid value java.sql.SQLException: No suitable driver found for for configuration

I am trying to connect kafka-connect to my local mssql with localhost:3030
I am receiving this error when I try to make a new connection for mssql. in centos 7(linux). Mssql data is from an external IP(windows), my consumer is inside of linux environment.
"No suitable driver found for configuration".
connect-distributed.properties is shown below;
plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors,
I added "ojdbc7-12.1.0.2.jar" file under /opt/connectors/kafka-connect-jdbc/ but still I am receiving error messages. I dont know what is wrong.
Also my connect-console-source.properties
name=source-sqlserver-user
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
topic.prefix=my-timestamp
connection.url=jdbc:sqlserver://externalIP;database=database;username=username;password=password
version: '2'
services:
kafka-cluster:
image: landoop/fast-data-dev:cp3.3.0
environment:
ADV_HOST: 127.0.0.1
RUNTESTS: 0
ports:
- 2181:2181 # Zookeeper
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect ports
- 9581-9585:9581-9585 # JMX Ports
- 9092:9092 # Kafka Broker
ojdbc7-12.1.0.2.jar is the JDBC driver for Oracle.
For MS SQL you need the MS SQL JDBC driver
Edit: Since you're using Docker to run Kafka Connect, you need to make the JDBC JAR file available to the Kafka Connect worker before it runs. You can't just run the Docker container and copy the JDBC driver into it, because you need to restart Kafka Connect afterwards.
To workaround this, you can mount the JAR from your local machine to the relevant path in the container. The relevant path is whereever the Kafka Connect JDBC jar is. Looking at the fast-data-dev image it's in
root#fast-data-dev / $ ls -l /opt/confluent-3.3.0/share/java/kafka-connect-jdbc
total 6544
-rw-r--r-- 1 root root 133842 Jul 28 2017 kafka-connect-jdbc-3.3.0.jar
-rw-r--r-- 1 root root 658466 Jul 28 2017 postgresql-9.4-1206-jdbc41.jar
-rw-r--r-- 1 root root 5575351 Jul 28 2017 sqlite-jdbc-3.8.11.2.jar
So you can run
docker run --rm --net=host --volume ~/Downloads/mssql-jdbc-7.4.1.jre8.jar:/opt/confluent-3.3.0/share/java/kafka-connect-jdbc/mssql-jdbc-7.4.1.jre8.jar landoop/fast-data-dev:cp3.3.0
or mount it in your Docker Compose with a volumes config:
version: '2'
services:
kafka-cluster:
image: landoop/fast-data-dev:cp3.3.0
environment:
ADV_HOST: 127.0.0.1
RUNTESTS: 0
ports:
- 2181:2181 # Zookeeper
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect ports
- 9581-9585:9581-9585 # JMX Ports
- 9092:9092 # Kafka Broker
volumes:
- ~/Downloads/mssql-jdbc-7.4.1.jre8.jar:/opt/confluent-3.3.0/share/java/kafka-connect-jdbc/mssql-jdbc-7.4.1.jre8.jar
It's worth noting that Confluent Platform 3.3.0 is really old - the latest is 5.3.1. If you want to see an up-to-date example of running Kafka, Kafka Connect, SQL Server etc with JDBC driver automatically set up see this example here.

Debezium Connector for SQL Server - connection refused

I want to use Kafka to publish MSSQL CDC events.
I am using Docker containers for:
debezium/zookeeper
debezium/kafka
debezium/connect
Microsoft SQL Server
Containers started as follows:
docker run -it --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper
docker run -it --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka
docker run -it --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME="localhost" --link zookeeper:zookeeper --link kafka:kafka debezium/connect
docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=xxxxxxxxxxxxx" -p 1433:1433 --name sql1 -d mcr.microsoft.com/mssql/server:2017-CU8-ubuntu
All containers start running successfully.
Then I created new MSSQL db in SQL Server container. Created 1 table in new db and turned on CDC for that table. CDC is working fine.
Then I send connector configuration below to Kafka Connect REST API as follows:
curl -X POST -H "Content-Type: application/json" -d #test-mssql-connector.json http://localhost:8083/connectors
using test-mssql-connector.json
{
"name": "test-mssql-connector5",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"database.hostname": "localhost",
"database.port": "1433",
"database.user": "SA",
"database.password": "xxxxxxxxxxxxx",
"database.dbname": "test",
"database.server.name": "sql1",
"table.whitelist": "dbo.Persons",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "dbhistory.sql1"
}
}
However, Kafka connector cannot connect to the MSSQL db giving error message below:
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection
to the host localhost, port 1433 has failed. Error: \"Connection
refused. Verify the connection properties. Make sure that an instance
of SQL Server is running on the host and accepting TCP/IP connections
at the port. Make sure that TCP connections to the port are not
blocked by a firewall.
Most troubleshooting are if database actually running, or port is blocked, but there is no problem with new MSSQL db. It's container is active, and the db is successfully running. The port is not blocked. I can successfully connect to it from host machine using DbVisualizer or other query tools with following configuration:
database server = localhost
database port = 1433
user = SA
pw = xxxxxxxxxxxxx
database name = test
I can successfully use telnet localhost 1433 to connect to server.
Is there something missing in the connector configuration above?
IMHO the localhost is not correct as localhost is something else in Connect container and in SQL Server container. You should link the database container into Connect container and use the appropriate hostname.
You need to first set up your sql container and THEN only start the connect service specifying the sql server as an additional link:
docker run -it --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME="localhost" --link zookeeper:zookeeper --link kafka:kafka --link kafka:kafka debezium/connect

Resources