I am new to Debezium, Kafka, and Docker. I have successfully installed Docker and it is running on my locahost.
I am attempting to go through the Debezium tutorial at: https://github.com/debezium/debezium-examples/blob/master/tutorial/README.md#debezium-tutorial
I went to the section for SQL Server: and the first step says to # Start the topology as defined in https://debezium.io/docs/tutorial/. I successfully ran through that tutorial. But, it is for MySQL and not MSSQL Server. Anyways, I went back to the ../debezium-tutorial and the first line tells me to run:
export DEBEZIUM_VERSION=1.1
docker-compose -f docker-compose-sqlserver.yaml up
The tutorial does not discuss how to create the docker-compose-sqlserver.yaml. I checked Debezium's github site for this file and it is not there. Am I supposed to create this file manually or am I missing something in the steps?
In order to get Debezium to work, am I supposed to create and run a SQL Server instance in Docker, or can I use the instance that is running on my localhost?
The Docker Compose is included in the tutorials repository.
git clone https://github.com/debezium/debezium-examples.git
cd debezium-examples/tutorial
export DEBEZIUM_VERSION=1.1
docker-compose -f docker-compose-sqlserver.yaml up
Related
I am new in elasticsearch. I have to set up wazuh with elasticsearch cluster. I did all the thing. I have also installed wazuh plugin on the Kibana . Once, I opened the app and clicked on the agent section It is saying =>
Error getting alerts from compliances
Your environment may not have any index with Wazuh's alerts
Please help me.
you could try to uninstall and install it again. Here you have the official uninstallation guide Uninstalling Wazuh with Elastic Stack, and after installing Elastic again with this guide Unattended installation
Remember if you want to preserve your configuration you can backup the files:
cp -p /var/ossec/etc/ossec.conf.orig /var/ossec_backup/etc/ossec.conf
cp -p /var/ossec/etc/local_internal_options.conf /var/ossec_backup /etc/local_internal_options.conf
cp -p /var/ossec/etc/ /var/ossec_backup/etc/client.keys
cp -p /var/ossec/queue/rids/ /var/ossec_backup/queue/rids/*
Here you have more information about that Migrating OSSEC agent
After that, you have to put the files in their original path again
At our company we use a couple methods of updating database models. we use sqlproj and dacpacs as well as we have a .net core app using DbUp. I have successfully containerized using dacpac based of this article automating-sql-server-2019-docker-deployments
Now I am working on Db Up. There are a few challenges that I've tried to sort out but I am not able to sort through them.
My first though was to gen a dockerfile based off mcr.microsoft.com/mssql/server:2019-latest then install the .net core runtime and run the built dbup inside sql container. It has been hard to get the .net core runtime running I just can't seem to get it to work.
I was trying to use docker-compose to buildjust a norm sql db and also run the dbUp using a base image mcr.microsoft.com/dotnet/core/sdk:3.1 AS build and connect to the sql db this way. this is also not working
here is my docker-compose.yml file(sorry for some reason i can't coy the code correctly here:
version: "3.7" services: ms-sql-server:
image: mcr.microsoft.com/mssql/server:2019-latest
ports:
- "1477:1433"
environment:
SA_PASSWORD: "SuperFun!23"
ACCEPT_EULA: "Y" dbup-exe:
build: .
depends_on:
- ms-sql-server
In my Dockerfile I am calling the db but it won't connect to it
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /src
COPY ["/Sentinel.DbUp.csproj", "Sentinel.DbUp/"]
RUN dotnet restore "Sentinel.DbUp/Sentinel.DbUp.csproj"
WORKDIR "/src/Sentinel.DbUp"
COPY . .
RUN dotnet build "Sentinel.DbUp.csproj" -c Release -o /app
FROM build AS publish
RUN dotnet publish "Sentinel.DbUp.csproj" -c Release -o /app
RUN dotnet run Sentinel.DbUp --ConnectionString=Server=ms-sql-server,1477;Database=Sentinel_Local;User Id=sa;Password=SuperFun!23; --WithSeedOnce --EnsureDatabase --PerformUpgrade
I have two errors
System.Data.SqlClient.SqlException (0x80131904): A network-related or
instance-specific error occurred while establishing a connection to
SQL Server. The server was not found or was not accessible
/bin/sh: 1: User: not found /bin/sh: 1: --WithSeedOnce: not found
Based on my research, I though when using docker compose between docker images you can reference ms-sql-server instead of local host.
also when executing the last command how do I append multiple arguments? it seems --WithSeedOnce is not considered and argument for the app
So, from what I gather, there are two things that jump off the page:
In a dockerfile, RUN instructions are executed when the image is built. Entrypoint or command instructions are executed when the container is started. As such, the command to start the DbUp (which I assume is the last line in your current dockerfile) should be changed from RUN to entrypoint.
using the depends on instruction does not mean that the docker will wait for your database to be ready before starting the DbUp container. It just means that the database container will be launch before your DbUp is launched. Instead you have to wait for the DB in you code or using a wrapper script like wait-for-it. Check the documentation here
I am using the cloned dspace 6-x branch and installed it via docker. Can someone help me with the backup of my local database (Communities, collections, items)to a remote database?
According to the documentation we need to use the command:
dspace packager -s -t AIP -e eperson -p parent-handle file-path
But it returns an error: dspace is not a command
Anyone could help me transfer my local database to my remote repo?
Thanks!
Moving publications to a new repository will be a more substantial undertaking!
But your recent problem seems just that you are either not on the right container or in the right directory for executing the dspace command. Thus it is "not found". Make sure to execute dspace on the dspace container and specify the right/complete path. The dspace command is located in
/path/to/your/dspace-deployement-directory/bin.
I install docker container on mac(OS X) and install Microsoft SQL 2017 image file on docker.So, I try to connect docker with Azure Data Studio but didn't connect it. Can I connect docker with Azure Data Studio and How to configure it? Please help me, thank a lot.
Use 127.0.0.1,1433 instead of 127.0.0.1:1433
This syntax is what my ASP.NET Core app uses as syntax so I figured MS liked that format for connection strings and such.
This worked for me. Hope it helps.
I was able to run SQL server on MAC using Docker by running it along with the Azure Data Studio.
In order to connect to a server, you need to go to preferences of your Docker settings and increase the Memory allocation from the default of 2GB to minimum 4GB (as SQL server needs min 3.25GB space). Save and restart the docker.
Once restarted, all you need to do is pull the docker image of the sql server and download it. this can be done by below commands on your terminal . FYI, I am using bash commands below:
Command 1:
sudo docker pull mcr.microsoft.com/mssql/server:2017-latest
This will pull the latest vesion docker image and download. Once done, you need to set your SQL authentication on the server for your database. Follow below commands:
Command 2:
sudo docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<SetYourPasswordHere>' \
-p 1433:1433 --name sql1 \
-d mcr.microsoft.com/mssql/server:2017-latest
This sets your password and uses the port 1433 for SQL server (which is the default port). To confirm if the image has been created and the SQL server is running on docker, execute the below command to check log(s).
Command 3:
docker ps
To check all instances in your history of dockers( i.e. if you already had dockers installed before you are attempting this SQL connection/execution), run the below command and it will give you all the logs of all instances you have created
Command 4:
docker ps -a
or
docker ps -all
Once, you have completed above steps and see that the docker has created SQL instance, you need to go to Azure Data Studio and set the below credentials to access the server that you just created above using Docker.
Server: localhost
Authentication Type: SQL Authentication
Username: sa
Password: <Check Command 2 to see what you entered in the password where it says SetYourPasswordHere>
Hope this helps in your tryst with running SQL server on your MAC. All the Best!
You certainly can connect to a sql server image running in a docker container through azure data studio,
Based on the details mentioned in the question, I'm assuming that you have followed the steps on Microsoft docs for configuring sql server with docker,
The following command is needed to configure and run the SQL Server image docker container:
sudo docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=your-strong-password’ -p 1433:1433 -d mcr.microsoft.com/mssql/server:2017-latest;
To quickly verify
check that the image is running by running:
docker ps -a
And checking the status column (with the correct instance name) to be 'UP',
Then launch Azure Data Studio and fill the connection details:
If you have followed all the default settings in setting up the image, this should work for you,
Hope this helps,
I hope first you have installed sql-cli(make sure you have node.js installed in your system),
Then connect to Mssql with command -> mssql -u -p
try to connect/create a database with docker first then connect from Azure Data Studio
I'm new to nifi and i want to connect SQL server database to nifi and create a data flow with the processors. how can I do this, can any one Help me with this clearly.
Thanks in Advance
sam
Here are two great articles on getting information in and out of databases with NiFi:
http://www.batchiq.com/database-injest-with-nifi.html
http://www.batchiq.com/database-extract-with-nifi.html
They describe/illustrate how to configure a DBCPConnectionPool service to provide connection(s) to an RDBMS, and example flows to extract data and ingest data.
Expanding on mattyb answer
If you are using the latest Hortonworks sandbox, or other setup that uses docker containers, read below.
You have to install the JDBC jar file inside the docker. For SQL Server, it should be 6.2 or above.
docker ps
docker exec -it <mycontainer uuid> bash
How do I get into a Docker container's shell?
will help you log into the container.
cd file:///usr/lib/jvm/jre/lib/
mkdir jdbc
cd ./jdbc
wget https://download.microsoft.com/download/3/F/7/3F74A9B9-C5F0-43EA-A721-07DA590FD186/sqljdbc_6.2.2.0_enu.tar.gz
tar xvzf sqljdbc_6.2.2.0_enu.tar.gz
cp ./sqljdbc_6.2/enu/mssql-jdbc-6.2.2.jre8.jar ./
jdbc:sqlserver://192.168.1.201:1433;databaseName=[your database]
com.microsoft.sqlserver.jdbc.SQLServerDriver
You might need to replace the ip address with IPv4 address of your host found under ipconfig in Windows or ifconfig in Mac/Linux.
You may change file:///usr/lib/jvm/jre/lib/ to any path you desire.
Expanding on TamusJRoyce's answer
If you are running nifi via a docker image like apache/nifi or the aforementioned Hortonworks sandbox, the following should help you get the required driver on the image so that you don't need to exec into the container to do it manually.
See the comments below the docker file
FROM apache/nifi
USER root
RUN mkdir /lib/jdbc
WORKDIR /lib/jdbc
RUN wget https://download.microsoft.com/download/3/F/7/3F74A9B9-C5F0-43EA-A721-07DA590FD186/sqljdbc_6.2.2.0_enu.tar.gz
RUN tar xvzf sqljdbc_6.2.2.0_enu.tar.gz
RUN cp ./sqljdbc_6.2/enu/mssql-jdbc-6.2.2.jre8.jar ./
USER nifi
EXPOSE 8080 8443 10000 8000
WORKDIR ${NIFI_HOME}
ENTRYPOINT ["../scripts/start.sh"]
The above image uses apache/nifi as the base image. You can use any nifi docker image has a base if you would like.
You can specify any location for lib/jdbc, just remember that you need to use this as the reference for the file location so that it is referenced as file:///lib/jdbc/mssql-jdbc-6.2.2.jre8.jar
Lastly, switch back to the nifi user and finish off with the standard nifi image details. This will allow the image to run correctly.