Gitlab with SQL Server as database source configuration handling - sql-server

I have a self hosted gitlab on ubuntu machine. I configure a linux container for it to run runner. Now, I am trying to write a configuration for my dotnet project to run unit test on this setup.
I get configuration to run dotnet application without database, and only part I got stuck is that I cannot get Database to load or connect through my test environment.
I get SQL Server linux container to run as service (I am guessing it is running). But I am not sure how I can load my database to it. I know I can do that using Docker Run. But I cannot figure it out how to run it here.
When I try to run "mssql-tools" as service I cannot get it's command to run as it is not install by default in dotnet image.
Here is my file.
image: microsoft/dotnet:latest
variables:
ACCEPT_EULA: Y
SA_PASSWORD: my_secure_password
MSSQL_PID: Developer
stages:
- test
before_script:
- "cd Source"
- "dotnet restore"
test:
stage: test
services:
- mcr.microsoft.com/mssql/server:2017-latest
- mcr.microsoft.com/mssql-tools
script:
- "cd ../Database"
- "docker run -it mcr.microsoft.com/mssql-tools"
- "sqlcmd -S . -U SA -P my_secure_password -i testdata_structure.sql"
- "exit"
- "cd ../Source"
- "dotnet build"
- "dotnet test"
"sqlcmd -S . -U SA -P my_secure_password -i testdata_structure.sql this command won't work in this setup as sqlcmd is not installed, but is one of service. I don't want to make a new image that has all pre-install. But use available stuff to work.
Not, sure if I am able to explain my issue and knowledge here. I am new, but I am reading and changing configuration from 2 days. I can get Linux based SQL Server to run with my app from local docker commands and stuff, but on Gitlab to run Unit Test I cannot get database to restore/get running and connect to application.

GitLab Services does not install commands or apps inside your container job, instead a Service is another container that is usually run in parallel to offer infrastructure services such as databases, cache, queues, etc.
if you want to have sqlcmd inside your container you must install it:
This is an extract from my pipeline, in this case my container is based on Alpine but you can find more ways here: https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15
before_script:
- apk add curl
- apk add --no-cache gnupg
- curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.7.2.1-1_amd64.sig
- curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/mssql-tools_17.7.1.1-1_amd64.sig
- curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.7.2.1-1_amd64.apk
- curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/mssql-tools_17.7.1.1-1_amd64.apk
- curl https://packages.microsoft.com/keys/microsoft.asc | gpg --import -
- gpg --verify msodbcsql17_17.7.2.1-1_amd64.sig msodbcsql17_17.7.2.1-1_amd64.apk
- gpg --verify mssql-tools_17.7.1.1-1_amd64.sig mssql-tools_17.7.1.1-1_amd64.apk
- apk add --allow-untrusted msodbcsql17_17.7.2.1-1_amd64.apk
- apk add --allow-untrusted mssql-tools_17.7.1.1-1_amd64.apk
script:
- /opt/mssql-tools/bin/sqlcmd -S $DBC_SERVER -U $DBC_USER -P $DBC_PASSWORD -q "USE myTestDb; CREATE TABLE testGitlab (id int); SELECT * FROM testGitLab"

I end up using my custom Docker Image that has dotnetcore and Sqlcmd installed in it, I can use MsSQL Server as Service in gitlab configuration. (have to define SQL Server' hostname, as IP in same range as my server).
Not an idle answer, but workaround for me.

Related

Bitbucket pipeline sql server database set port

I have a bitbucket pipeline that must execute django unittests. Therefore, I need a test database which should be a SQL SERVER datbase.
The pipeline looks like this:
# This is a sample build configuration for Python.
# Check our guides at https://confluence.atlassian.com/x/x4UWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: python:3.7.3
pipelines:
branches:
master:
- step:
name: Setup sql
image: fabiang/sqlcmd
script:
- sqlcmd -S localhost -U sa -P $DB_PASSWORD
services:
- sqlserver
- step:
name: Run tests
caches:
- pip
script: # Modify the commands below to build your repository.
- python3 -m venv my_env
- source my_env/bin/activate
- apt-get update && apt-get install
- pip3 install -r req-dev.txt
- python3 manage.py test
- step:
name: Linter
script: # Modify the commands below to build your repository.
- pip3 install flake8
- flake8 --exclude=__init__.py migrations/
definitions:
services:
sqlserver:
image: mcr.microsoft.com/mssql/server:2017-latest
variables:
ACCEPT_EULA: Y
SA_PASSWORD: $DB_PASSWORD
And everytime when I run the pipeline I get:
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : TCP Provider: Error code 0x2726.
I tried to do it locally but then it only work when I defined a port with the -p tag:
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong!' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2017-latest
How can I make the pipeline work? (probably defining a port but how?)
UPDATE:
On the sqlserver tab in the result section is the following error shown:
I think the problem is when you call the script - sqlcmd -S localhost -U sa -P $DB_PASSWORD because your sqlserver is not yet completly initialized.
Try to put a sleep 10 before the command and the best is to add an error case if command fail sleep 5 and retry again.

Docker - SQL service wont run when cloned

I'm trying to add a volume to a docker container but when I commit it and run with the new volume none of the sql services run on this copy?? Why would that be.
https://learn.microsoft.com/en-us/sql/linux/quickstart-install-connect-docker?view=sql-server-ver15&pivots=cs1-powershell
I am adding the initial one as above and it works.All fine. Services running. I can connect to it, run SQL but I need to share a drive.
Seems I cant add one directly to an existing instance??
docker commit 5a8f89adeead newimagename
docker run -ti -v "C:/dir1":/dir1 newimagename /bin/bash
I do the above to clone it and add a volume. WORKS. But the sql services just arent running on this new one. Ill accept it either way I just want SQL running and a share in there.
Can anyone help.
Manged it:
docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Pa55word1" `
-v C:/db:/dir1 `
-p 1433:1433 --name sql3 `
-d mcr.microsoft.com/mssql/server:2019-CU3-ubuntu-18.04
Had an issue with having no drive or no services but this has done it.

Unable to connect to remote SQL server from container

I'm trying to connect to my remote SQL server from a docker container hosted on my computer.
But are just reciving the following error:
A network-related or in stance-specific error has occurred while establishing a connection to SQL Server . Server is not found or not accessible. Check if instance name is correct and i f SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.. Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : Login timeout expired.
But if I try to connect to my SQL server from SQL server management studio on the host machine everything works properly.
Also note that 2 weeks ago everything also worked inside the docker container.
Here is my docker compose file and the docker file which has the SQL driver installed:
Compose:
version: '3'
services:
nginx:
image: nginx:1.10
volumes:
- ./:/var/www
- .docker/nginx/vhost.conf:/etc/nginx/conf.d/default.conf
ports:
- ${DOCKER_IP}80:80
links:
- php
networks:
- app-net
php:
build:
context: ./
dockerfile: .docker/php/DockerFile
volumes:
- ./:/var/www
networks:
- app-net
networks:
app-net:
driver: bridge
Docker file
FROM phpdockerio/php71-fpm:latest
# Install selected extensions and other stuff
RUN export DEBIAN_FRONTEND=noninteractive && \
apt-get update && apt-get -y --no-install-recommends install \
php7.1-mysql \
php7.1-mbstring \
php7.1-gd \
php7.1-soap \
php7.1-dev \
apt-transport-https \
git \
ssh \
curl \
php-pear \
&& apt-get clean; rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /usr/share/doc/*
# Install Composer
RUN cd /usr/src
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
# Install MSSQL extention
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get -y install msodbcsql mssql-tools g++ unixodbc-dev make
RUN pear config-set php_ini `php --ini | grep "Loaded Configuration" | sed -e "s|.*:\s*||"` system
RUN pecl install sqlsrv
RUN pecl install pdo_sqlsrv
RUN echo "extension=sqlsrv.so" >> /etc/php/7.1/fpm/php.ini
RUN echo "extension=pdo_sqlsrv.so" >> /etc/php/7.1/fpm/php.ini
# Fixed locals for MSSQL extention
RUN apt-get install -y locales
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen
RUN locale-gen
WORKDIR /var/www
Inside the docker container I can't ping the SQL server. So for me i sounds like a network issue, but I'm unable find a solution.
Please note the SQL server is hosted locally on a server in the office.
UPDATE/Solved for now
After downgrading the dokcer for windows to 18.03.0-CE everything worked as expected.
Docker Bridge networks don't connect to the world outside of Docker at all by default; they only allow containers to talk to each other. The documentation for Docker Bridge Networks does offer some advice for allowing Bridge network traffic to talk to the outside world by making changes on the Docker host:
First enable IP forwarding in the kernel:
$ sysctl net.ipv4.conf.all.forwarding=1
Then change the host firewall to allow the forwarding
$ sudo iptables -P FORWARD ACCEPT
This is a fairly permissive firewall configuration, so you may want to look at keeping that a bit more locked down.
The other way would be to attach the container to two different networks: your current bridge network for communication between container, and a second Host network for talking to MySQL. Since this bypasses all of Docker's NAT configuration, the scalability of your service may be impacted, although I think outgoing connections might be OK.
For my case:
I ran these below:
To see all containers:
docker ps -a
Then restart container:
docker container restart yourIdContainer
If there is error: Error response from daemon: Cannot restart container ...
Please restart Docker then restart container again.
Connect to Sql Server by MSSM, server name: localhost,1433 or IP
Hope this is helpful.
If you are using Windows 10 (HOME VERSION ) may you don't have the virtualization and you are using (docker toolbox ), they have solved the problem using a static ip. Check it.
I mean localhost won't work if you are using docker toolbox, search the ip where is virtualized docker toolbox.
My case ip: 192.168.99.100

How to import .bacpac into docker Sqlserver?

I installed Sqlserver on my Mac in a docker container, following the instructions from this article.
I run the container with Kitematic and managed to connect to the server using Navicat Essentials for SQl Server.
The server has four databases and I can create new ones, but, ideally, I would like to import an existing database as .bacpac.
The instructions from this answer have been of use to me in the past. Can I run something similar within the container? Or, more generally, is there a way to import a database in the container?
Hi all! We finally have a preview ready for sqlpackage that is built on dotnet core and is cross-platform! Below are the links to download from. They are evergreen links, i.e. each day a new build is uploaded. This way any checked in bug fix is available the next day. Included in the .zip file is the preview EULA.
linux
https://go.microsoft.com/fwlink/?linkid=873926
osx
https://go.microsoft.com/fwlink/?linkid=873927
windows
https://go.microsoft.com/fwlink/?linkid=873928
Release notes:
The /p:CommandTimeout parameter is hardcoded to 120
Build and deployment contributors are not supported
a. Need to move to .NET Core 2.1 where System.ComponentModel.Composition.dll is supported
b. Need to handle case-sensitive paths
SQL CLR UDT types are not supported.
a. This includes SQL Server Types SqlGeography, SqlGeometry, & SqlHierarchyId
Older .dacpac and .bacpac files that use Json serialization are not supported
Referenced .dacpacs (e.g. master.dacpac) may not resolve due to issues with case-sensitive file systems
For lack of a better method, please provide any feedback you have here on this GitHub issue.
Thanks for giving it a try and letting us know how it goes!
https://github.com/Microsoft/mssql-docker/issues/135#issuecomment-389245587
EDIT: I've made you a Docker image for this
https://hub.docker.com/r/samuelmarks/mssql-server-fts-sqlpackage-linux/
Example of setting up a container, creating a database, copying a .bacpac file over, and importing it into aforementioned database:
docker run -d -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<YourStrong!Passw0rd>' -p 1433:1433 --name sqlfts0 samuelmarks/mssql-server-fts-sqlpackage-linux
docker exec -it sqlfts0 /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P '<YourStrong!Passw0rd>' -Q 'CREATE DATABASE MyDb0'
docker cp ~/Downloads/foo.bacpac sqlfts0:/opt/downloads/foo.bacpac
docker exec -it sqlfts0 dotnet /opt/sqlpackage/sqlpackage.dll /tsn:localhost /tu:SA /tp:'<YourStrong!Passw0rd>' /A:Import /tdn:MyDb0 /sf:foo.bacpac
It looks like Microsoft has implemented support of this on sqlpackage, with documentation!
You will have to add sqlpackage to your container.
You can download it here. (optionally, direct link to linux package here, hopefully doesn't change)
The following are instructions for running this from a windows machine -- obviously it's the bare minimum to get it working. Please change passwords, and probably put this in a docker-compose.yml for re-use.
I unzip the above package into a folder 'c:\sqlpackage' (my windows docker run doesn't allow relative paths), and then mount that into the container with the bacpac, like such:
docker run -d -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Asdf1234" -v c:\sqlpackage:/opt/sqlpackage -v c:\yourdb.bacpac:/tmp/yourdb.bacpac -p 1433:1433 --name mssql-server-example microsoft/mssql-server-linux:2017-latest
here is what a *nix user could run alternatively:
docker run -d -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Asdf1234' -v ./sqlpackage:/opt/sqlpackage -v ./yourdb.bacpac:/tmp/yourdb.bacpac -p 1433:1433 --name mssql-server-example microsoft/mssql-server-linux:2017-latest
and finally, attach to your container and run:
/opt/sqlpackage/sqlpackage /a:Import /tsn:. /tdn:targetdbname /tu:sa /tp:Asdf1234 /sf:/tmp/yourdb.bacpac
After this, you should be able to connect with SSMS to localhost, username and password as you provide them above, and see 'targetdbname'! These are mostly notes I wrote for myself but I'm sure others could use them too.
You can use free Azure Data Studio from Microsoft. Once you have it installed, install the extension "Admin Pack for SQL Server" from Microsoft. Then you can import bacpac files with ease.
This is not a supported feature with a LINUX implementation it seems.
See this link.

Database migrations in docker swarm mode

I have an application that consists of simple Node app and Mongo db. I wonder, how could I run database migrations in docker swarm mode?
Without swarm mode I run migrations by stopping first the old version of application, running one-off migration command with new version of application and then finally starting a new version of app:
# Setup is roughly the following
$ docker network create appnet
$ docker run -d --name db --net appnet db:1
$ docker run -d --name app --net appnet -p 80:80 app:1
# Update process
$ docker stop app && docker rm app
$ docker run --rm --net appnet app:2 npm run migrate
$ docker run -d --name app --net appnet -p 80:80 app:2
Now I'm testing the setup in docker swarm mode so that I could easily scale app. The problem is that in swarm mode one can't start containers in swarm network and thus I can't reach the db to run migrations:
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
6jtmtihmrcjl appnet overlay swarm
# Trying to replicate the manual migration process in swarm mode
$ docker service scale app=0
$ docker run --rm --net appnet app:2 npm run migrate
docker: Error response from daemon: swarm-scoped network (appnet) is not compatible with `docker create` or `docker run`. This network can only be used by a docker service.
I don't want to run the migration command during app startup either, as there might be several instances launching and that would potentially screw the database. Automatic migrations are scary, so I want to avoid them at all costs.
Do you have any idea how to implement manual migration step in docker swarm mode?
Edit
I found out a dirty hack that allows to replicate the original workflow. Idea is to create a new service with custom command and remove it when one of its tasks is finished. This is far from pleasant usage, better alternatives are more than welcome!
$ docker service scale app=0
$ docker service create --name app-migrator --network appnet app:2 npm run migrate
# Check when the first app-migrator task is finished and check its output
$ docker service ps app-migrator
$ docker logs <container id from app-migrator>
$ docker service rm app-migrator
# Ready to update the app
$ docker service update --image app:2 --replicas 2 app
I believe you can fix this problem by making your overlay network, appnet, attachable. This can be accomplished with the following command:
docker network create --driver overlay --attachable appnet
This should fix the swarm-scoped network error and and allow you to run migrations
This is indeed tricky situation, though I think running the migration during startup might be the the final piece of the puzzle.
The way I do it right now (though not very elegant, it works), is using a message queue (I'm using redis), on app startup, it will send a message to a the queue, informing it that the migration task needs to be run. At the other end of the queue, I have a listener app that will process the queue and run the migration task. The migration task would only run once, since there's only a single instance of the listener running it sequentially. So essentially I'm just using the queue & the listener app to make sure that the migration task runs only once.

Resources