I changed the port in the /etc/mongod.conf file ie
net:
port: 12345
bindIp: 127.0.0.1,123.123.23.255 #localhost and server pub ip
Then I restarted mongo
sudo systemctl restart mongod.service
and checked all ok:
sudo systemctl status mongod
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2018-07-10 13:35:14 UTC; 16s ago
Docs: https://docs.mongodb.org/manual
Main PID: 1927 (mongod)
Tasks: 23
Memory: 136.3M
CPU: 1.240s
CGroup: /system.slice/mongod.service
└─1927 /usr/bin/mongod --config /etc/mongod.conf
Then I tried to login to mongo:
mongo --username me --password my_password --authenticationDatabase authdb
And got the following error:
MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-07-10T13:29:14.716+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-07-10T13:29:14.717+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
I then changed the port back to 27017 in mongod.conf and could login to the db OK.
Any idea why the mongo shell tries 27017 even though mongo port is changed?
You're confusing between mongo server (the server you configure when you change .conf file) and mongo client you used to connect to mongo server. If you changed your port to 12345, then you must precise it when you try to connect, providing it in the connecting command :
mongo --username me --password my_password --authenticationDatabase authdb --port 12345
or
mongo --username me --password my_password --authenticationDatabase authdb --host 127.0.0.1:12345
EDIT : without any port specification in mongo command, default one will be used, so 27017.
Related
This is my docker-compose.yml file:
version: "3.9"
services:
api:
# configuration of API here
db:
image: mcr.microsoft.com/mssql/server
restart: always
container_name: Database
volumes:
- ./Data:/var/opt/mssql
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=lksU2o412f7tBj58t07B
- MSSQL_PID=Express
ports:
- 1433:1433
command: >
sh -c
"
tail -f /dev/null
"
As you can see, I'm using the official docker image for MS SQL Server, mcr.microsoft.com/mssql/server.
My laptop is Ubuntu 20.04 LTS.
I run docker-compose up -d and then docker exec -it db sh to get interactive shell.
Then I write this command:
/opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P 'lksU2o412f7tBj58t07B'
And I get this response:
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : TCP Provider: Error code 0x2749.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online..
This is the official image and I have not changed anything.
Also in my api container, I can ping db or Database. But I can't telnet port 1433.
Also from my host machine I can telnet localhost 1433, but I get this response:
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
This is a terrible experience in working with containers from Microsoft. Aren't containers supposed to work out of the box? Isn't container technology there only to solve these stupid problems?
What should I do?
I have uploaded some code on github: https://github.com/darkcloudi/helm-camunda-postgres
Running the following commands deploys the two charts (Note the set is required to allow the postgres db to be deployed, i've disabled it by default, as camunda comes with its own DB, i'm trying to configure it to use postgres):
helm install dev ./camunda-install --set tags.postgres=true
You will see that its all looking good:
NAME READY STATUS RESTARTS AGE
pod/dev-camunda-67f487dcd-wjdfr 1/1 Running 0 36m
pod/dev-camunda-test-connection 0/1 Completed 0 45h
pod/postgres-86c565898d-h5tf2 1/1 Running 0 36m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dev-camunda NodePort 10.106.239.96 <none> 8080:30000/TCP 36m
service/dev-postgres NodePort 10.108.235.106 <none> 5432:30001/TCP 36m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d19h
If I either use the 10.108.x.x ip or the minikube ip 192.168.64.2 I get the same error below, I can connect to tomcat using http://camunda.minikube.local/ or http://192.168.64.2:30000/ so was wondering where I might be going wrong when attempting to connect to postgres.
kubectl exec -it postgres-86c565898d-h5tf2 -- psql -h 10.108.235.106 -U admin --password -p 30001 camunda
Password:
psql: error: could not connect to server: could not connect to server: Connection timed out
Is the server running on host "10.108.235.106" and accepting
TCP/IP connections on port 30
kubectl describe svc dev-postgres
Name: dev-postgres
Namespace: default
Labels: app.kubernetes.io/managed-by=Helm
name=dev-postgres
Annotations: meta.helm.sh/release-name: dev
meta.helm.sh/release-namespace: default
Selector: app=dev-postgres,name=dev-postgres
Type: NodePort
IP: 10.108.235.106
Port: postgres-http 5432/TCP
TargetPort: 5432/TCP
NodePort: postgres-http 30001/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
https://github.com/darkcloudi/helm-camunda-postgres/blob/master/camunda-install/charts/postgres/templates/postgres.yaml
Since you are accessing it from within the cluster you should use the ClusterIP 10.108.235.106 and port 5432.
If you wan to access it from outside the cluster then you can use Node IP 192.168.64.2 and NodePort 30001
Port 30001 is listening on the node VM and container is listening on port 5432.So you can not access it via port 30001 from within the cluster.
Edit:
The Endpoints is empty on the service. This is because the label selector on the service is selecting pods with labels app=dev-postgres,name=dev-postgres but the pods don't have that label.
I am trying to connect kafka-connect to my local mssql with localhost:3030
I am receiving this error when I try to make a new connection for mssql. in centos 7(linux). Mssql data is from an external IP(windows), my consumer is inside of linux environment.
"No suitable driver found for configuration".
connect-distributed.properties is shown below;
plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors,
I added "ojdbc7-12.1.0.2.jar" file under /opt/connectors/kafka-connect-jdbc/ but still I am receiving error messages. I dont know what is wrong.
Also my connect-console-source.properties
name=source-sqlserver-user
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
topic.prefix=my-timestamp
connection.url=jdbc:sqlserver://externalIP;database=database;username=username;password=password
version: '2'
services:
kafka-cluster:
image: landoop/fast-data-dev:cp3.3.0
environment:
ADV_HOST: 127.0.0.1
RUNTESTS: 0
ports:
- 2181:2181 # Zookeeper
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect ports
- 9581-9585:9581-9585 # JMX Ports
- 9092:9092 # Kafka Broker
ojdbc7-12.1.0.2.jar is the JDBC driver for Oracle.
For MS SQL you need the MS SQL JDBC driver
Edit: Since you're using Docker to run Kafka Connect, you need to make the JDBC JAR file available to the Kafka Connect worker before it runs. You can't just run the Docker container and copy the JDBC driver into it, because you need to restart Kafka Connect afterwards.
To workaround this, you can mount the JAR from your local machine to the relevant path in the container. The relevant path is whereever the Kafka Connect JDBC jar is. Looking at the fast-data-dev image it's in
root#fast-data-dev / $ ls -l /opt/confluent-3.3.0/share/java/kafka-connect-jdbc
total 6544
-rw-r--r-- 1 root root 133842 Jul 28 2017 kafka-connect-jdbc-3.3.0.jar
-rw-r--r-- 1 root root 658466 Jul 28 2017 postgresql-9.4-1206-jdbc41.jar
-rw-r--r-- 1 root root 5575351 Jul 28 2017 sqlite-jdbc-3.8.11.2.jar
So you can run
docker run --rm --net=host --volume ~/Downloads/mssql-jdbc-7.4.1.jre8.jar:/opt/confluent-3.3.0/share/java/kafka-connect-jdbc/mssql-jdbc-7.4.1.jre8.jar landoop/fast-data-dev:cp3.3.0
or mount it in your Docker Compose with a volumes config:
version: '2'
services:
kafka-cluster:
image: landoop/fast-data-dev:cp3.3.0
environment:
ADV_HOST: 127.0.0.1
RUNTESTS: 0
ports:
- 2181:2181 # Zookeeper
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect ports
- 9581-9585:9581-9585 # JMX Ports
- 9092:9092 # Kafka Broker
volumes:
- ~/Downloads/mssql-jdbc-7.4.1.jre8.jar:/opt/confluent-3.3.0/share/java/kafka-connect-jdbc/mssql-jdbc-7.4.1.jre8.jar
It's worth noting that Confluent Platform 3.3.0 is really old - the latest is 5.3.1. If you want to see an up-to-date example of running Kafka, Kafka Connect, SQL Server etc with JDBC driver automatically set up see this example here.
I want to use Kafka to publish MSSQL CDC events.
I am using Docker containers for:
debezium/zookeeper
debezium/kafka
debezium/connect
Microsoft SQL Server
Containers started as follows:
docker run -it --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper
docker run -it --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka
docker run -it --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME="localhost" --link zookeeper:zookeeper --link kafka:kafka debezium/connect
docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=xxxxxxxxxxxxx" -p 1433:1433 --name sql1 -d mcr.microsoft.com/mssql/server:2017-CU8-ubuntu
All containers start running successfully.
Then I created new MSSQL db in SQL Server container. Created 1 table in new db and turned on CDC for that table. CDC is working fine.
Then I send connector configuration below to Kafka Connect REST API as follows:
curl -X POST -H "Content-Type: application/json" -d #test-mssql-connector.json http://localhost:8083/connectors
using test-mssql-connector.json
{
"name": "test-mssql-connector5",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"database.hostname": "localhost",
"database.port": "1433",
"database.user": "SA",
"database.password": "xxxxxxxxxxxxx",
"database.dbname": "test",
"database.server.name": "sql1",
"table.whitelist": "dbo.Persons",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "dbhistory.sql1"
}
}
However, Kafka connector cannot connect to the MSSQL db giving error message below:
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection
to the host localhost, port 1433 has failed. Error: \"Connection
refused. Verify the connection properties. Make sure that an instance
of SQL Server is running on the host and accepting TCP/IP connections
at the port. Make sure that TCP connections to the port are not
blocked by a firewall.
Most troubleshooting are if database actually running, or port is blocked, but there is no problem with new MSSQL db. It's container is active, and the db is successfully running. The port is not blocked. I can successfully connect to it from host machine using DbVisualizer or other query tools with following configuration:
database server = localhost
database port = 1433
user = SA
pw = xxxxxxxxxxxxx
database name = test
I can successfully use telnet localhost 1433 to connect to server.
Is there something missing in the connector configuration above?
IMHO the localhost is not correct as localhost is something else in Connect container and in SQL Server container. You should link the database container into Connect container and use the appropriate hostname.
You need to first set up your sql container and THEN only start the connect service specifying the sql server as an additional link:
docker run -it --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME="localhost" --link zookeeper:zookeeper --link kafka:kafka --link kafka:kafka debezium/connect
I am attempting to connect to Microsoft SQL Server using Sqoop. I have installed the JDBC driver from Microsoft by following the instructions for the Sqoop Connector and the JDBC Driver. Next I attempt to list the databases on the server. I have tried the following commands:
sqoop list-databases --connect 'jdbc:sqlserver://<HOST>' --username <USER> --password <PASS>
sqoop list-databases --connect 'jdbc:sqlserver://<HOST>;username=<USER>;password=<PASS>'
sqoop list-databases --connect 'jdbc:sqlserver://<HOST>;username=<USER>;password=<PASS>' --username <USER> --password <PASS>
Each of these commands produces the same error messages.
13/01/02 10:44:52 ERROR sqoop.ConnFactory: Error loading ManagerFactory information from file <MY SQOOP DIRECTORY>/conf/managers.d/mssqoop-sqlserver: java.io.IOException: the content of connector file must be in form of key=value
at org.apache.sqoop.ConnFactory.addManagersFromFile(ConnFactory.java:219)
at org.apache.sqoop.ConnFactory.loadManagersFromConfDir(ConnFactory.java:294)
at org.apache.sqoop.ConnFactory.instantiateFactories(ConnFactory.java:85)
at org.apache.sqoop.ConnFactory.<init>(ConnFactory.java:62)
at com.cloudera.sqoop.ConnFactory.<init>(ConnFactory.java:36)
at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:200)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:44)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
13/01/02 10:45:08 ERROR manager.CatalogQueryManager: Failed to list databases
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host <HOST>, port 1433 has failed. Error: "connect timed out. Verify the connection properties, check that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port, and that no firewall is blocking TCP connections to the port.".
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:171)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1033)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:817)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:700)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:842)
at java.sql.DriverManager.getConnection(DriverManager.java:579)
at java.sql.DriverManager.getConnection(DriverManager.java:221)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:665)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:56)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
I have connected to the database using Microsoft SQL Server Management Studio to ensure the database is operation and that that the host/username/password are all correct. Additionally I have ensured that the port is open and that MSSQL is on the other side with the following.
sudo nmap -sS -p 1433 <HOST>
Starting Nmap 5.21 ( http://nmap.org ) at 2013-01-02 11:04 PST
Nmap scan report for <HOST> (<HOST IP>)
Host is up (0.00070s latency).
rDNS record for <HOST IP: <HOST FQDN>
PORT STATE SERVICE
1433/tcp filtered ms-sql-s
Nmap done: 1 IP address (1 host up) scanned in 0.33 seconds
Any suggestions on where I should go from here? I have not been able to find any documentation on this error. Thanks
I am currently attempting to verify that the SQL server is reachable using OSQL from FreeTDS. Will update this post with my findings.
After much searching and talking to many people I have determined that this issue is being caused by the port being blocked. I am still not 100% certain why this error in particular occurred. If I try with an invalid username or password it will successfully tell me that they are invalid. It is just when it comes to making the actual query that the port is restricted. Most likely different ports are used.