Deploying SQL Server database with dbup on Docker - sql-server

At our company we use a couple methods of updating database models. we use sqlproj and dacpacs as well as we have a .net core app using DbUp. I have successfully containerized using dacpac based of this article automating-sql-server-2019-docker-deployments
Now I am working on Db Up. There are a few challenges that I've tried to sort out but I am not able to sort through them.
My first though was to gen a dockerfile based off mcr.microsoft.com/mssql/server:2019-latest then install the .net core runtime and run the built dbup inside sql container. It has been hard to get the .net core runtime running I just can't seem to get it to work.
I was trying to use docker-compose to buildjust a norm sql db and also run the dbUp using a base image mcr.microsoft.com/dotnet/core/sdk:3.1 AS build and connect to the sql db this way. this is also not working
here is my docker-compose.yml file(sorry for some reason i can't coy the code correctly here:
version: "3.7" services: ms-sql-server:
image: mcr.microsoft.com/mssql/server:2019-latest
ports:
- "1477:1433"
environment:
SA_PASSWORD: "SuperFun!23"
ACCEPT_EULA: "Y" dbup-exe:
build: .
depends_on:
- ms-sql-server
In my Dockerfile I am calling the db but it won't connect to it
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /src
COPY ["/Sentinel.DbUp.csproj", "Sentinel.DbUp/"]
RUN dotnet restore "Sentinel.DbUp/Sentinel.DbUp.csproj"
WORKDIR "/src/Sentinel.DbUp"
COPY . .
RUN dotnet build "Sentinel.DbUp.csproj" -c Release -o /app
FROM build AS publish
RUN dotnet publish "Sentinel.DbUp.csproj" -c Release -o /app
RUN dotnet run Sentinel.DbUp --ConnectionString=Server=ms-sql-server,1477;Database=Sentinel_Local;User Id=sa;Password=SuperFun!23; --WithSeedOnce --EnsureDatabase --PerformUpgrade
I have two errors
System.Data.SqlClient.SqlException (0x80131904): A network-related or
instance-specific error occurred while establishing a connection to
SQL Server. The server was not found or was not accessible
/bin/sh: 1: User: not found /bin/sh: 1: --WithSeedOnce: not found
Based on my research, I though when using docker compose between docker images you can reference ms-sql-server instead of local host.
also when executing the last command how do I append multiple arguments? it seems --WithSeedOnce is not considered and argument for the app

So, from what I gather, there are two things that jump off the page:
In a dockerfile, RUN instructions are executed when the image is built. Entrypoint or command instructions are executed when the container is started. As such, the command to start the DbUp (which I assume is the last line in your current dockerfile) should be changed from RUN to entrypoint.
using the depends on instruction does not mean that the docker will wait for your database to be ready before starting the DbUp container. It just means that the database container will be launch before your DbUp is launched. Instead you have to wait for the DB in you code or using a wrapper script like wait-for-it. Check the documentation here

Related

How to use .NET Core secrets in .sh file that is called from docker-compose

Background
I am writing a .NET 5 application and using .net user secrets for my secret keys (database connections & passwords).
Recently I decided to learn Dockers and update my application to work with it so that using Visual Studio I generated a docker file for my API project and then created a docker-compose file that includes the API project & database (and some more irrelevant things for this question).
Almost everything works well. Technically, I can hard-code the secrets, and then the application will work well.
I have some secrets and most of them work fine, e.g: the database connection secrets works well, in the C# code I do the following code and it gets the value from the .net user-secrets:
config.GetConnectionString("Default");
Code Details
I have a secret key that contains a SQL password for the sa user.
dotnet user-secrets set "SA_PASSWORD" "<MySecretPassword>"
Then I have the docker-compose file which is of Linux system and this is part of the code:
sql_in_dc:
build:
context: .
dockerfile: items/sql/sql.Dockerfile
restart: always
ports:
- "1440:1433"
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=$SA_PASSWORD
- ASPNETCORE_ENVIRONMENT=Development
- USER_SECRETS_ID=80a155b1-fb7a-44de-8788-4f5759c60ff6
volumes:
- $APPDATA/Microsoft/UserSecrets/$USER_SECRETS_ID:/root/.microsoft/usersecrets/$USER_SECRETS_ID
- $HOME/.microsoft/usersecrets/$USER_SECRETS_ID:/root/.microsoft/usersecrets/$USER_SECRETS_ID
As you can see it calls the sql.Dockerfile which is:
FROM mcr.microsoft.com/mssql/server
ARG PROJECT_DIR=/tmp/devdatabase
RUN mkdir -p $PROJECT_DIR
WORKDIR $PROJECT_DIR
COPY items/sql/InitializeDatabase.sql ./
COPY items/sql/wait-for-it.sh ./
COPY items/sql/entrypoint.sh ./
COPY items/sql/setup.sh ./
CMD ["/bin/bash", "entrypoint.sh"]
Then the setup.sh is:
# Wait for SQL Server to be started and then run the sql script
./wait-for-it.sh sql_in_dc:1433 --timeout=0 --strict -- sleep 5s && \
/opt/mssql-tools/bin/sqlcmd -S localhost -i InitializeDatabase.sql -U sa -P "$SA_PASSWORD"
The Problem
The file setup.sh doesn't recognizes the $SA_PASSWORD environment variable when it comes from the secrets file.
It works well if I change the docker-compose.yml file to:
- SA_PASSWORD=SomePassword
Notes
I searched for an answer in Google and tried many things but couldn't find exactly my case.
I know it is possible to use Docker Swarm for the secrets but for now I want to do it without it. I am still learning and prefer that the code will work good and the next step will be to use Docker Swarm / Kubernetes / etc...
I would be happy to know if there is a fast solution even if it is not the ideal one. Later I will improve it and use better techniques.
I wrote the code I think that should be enough for the case and the relevant code, but if you need any more data, let me know and I will add it.
I have it in GitHub in a public repository in a pushed branch. If you want I can share with you the code.
Really big thanks in advance!
The docker-compose.yml is executed on your host OS (so it can use OS environment variables or vars from .env file, or from compose file, ...).
The running image - container has it's own set of env variables, in your case that means the running container has no SA_PASSWORD variable.
Your usecase would work if you had set the SA_PASSWORD Variable on your host OS.
You can check which variables are set in your container with (if your image comes with bash):
docker exec -it [container id] bash
printenv
The dotnet-secrets environment variables are created implicit during execution/runtime from Visual Studio (see entry in project file).
So as you mentioned "the compose file can't recognize dotnet-secrets".
You can use:
*.env File
pass it to compose command: with -e
Plain text in compose yml: - SA_PASSWORD=thepassword
as host OS Variable
Keep in mind that Visual Studio adds some magic when running or debugging your docker container. See Visual Studio container volume mapping: For ASP.NET core web apps, there might be two additional folders for the SSL certificate and the user secrets, which is explained in more detail in the next section.

Transferring a data from a mongodb on my local machine to a docker container

I'm trying to deploy a site and I am stuck trying to get my MongoDB data into my docker container. My API seems to work just fine without the docker container but when it is run using the docker container, it throws an error. The errors are due to the database being empty. I'm looking for a way to transfer previously stored data from my local MongoDB to the MongoDB on my container. Any solutions for this.
Below is my docker-compose.yml file:
version: "2"
services:
web:
build: .
ports:
- "3030:3030"
depends_on:
- mongo
mongo:
image: mongo
ports:
- "27018:27017"
I was told using mongodump and mongorestore could be helpful but haven't had much luck with mongorestore.
Currently, I have a dump folder with the db that I'm trying to transfer on my local machine. What steps should I take next to get it into docker?
Found the issue for anyone attempting to populate their mongo database in docker.
Here are the steps I took:
First used the mongodump to copy my database into a dump file
mongodump --db
Used docker cp to copy that dump file into a docker container
docker cp ~/dump/ :/usr/
Used mongorestore inside of the docker container
Open docker mongo shell in Docker Desktop or docker exec -it bash
cd into the usr directory
mongorestore --db= --collection= ./dump//.bson

Trouble connecting from docker container with ASP.NET core to SQL Server container

I have a container deploying a front end in ASP.NET Core trying to connect to the backend SQL Server database. I am running windows 10 with Docker desktop v19.03.13.
The website container is built on
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
WORKDIR /app
EXPOSE 80
# Copy csproj and restore as distinct layers
COPY ./FOOBAR/*.csproj ./
RUN dotnet restore
# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out
# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "FOOBAR.dll"]
The database is built on
FROM mcr.microsoft.com/mssql/server:2017-latest
USER root
COPY setup.sql setup.sql
COPY import-data.sh import-data.sh
COPY entrypoint.sh entrypoint.sh
RUN chmod +x entrypoint.sh
CMD /bin/bash ./entrypoint.sh
Everything works marvelous when running outside docker, .NET, Python, SQL Server Management Studio.
In .NET, my connection string is:
Server=localhost;Database=FOOBAR;Integrated Security=True
In Python:
DRIVER={ODBC Driver 17 for SQL Server};server=localhost;database=FOOBAR;Trusted_Connection=yes;
So I need to deploy this to a network that does not have a domain controller so I need to handle all database authentication.
When I build my containers, I change my .NET connection string to
dbConnection="Server=host.docker.internal;Database=FOOBAR;User Id=sa;Password=Password1!;"
I spawn my containers with
docker run -e ACCEPT_EULA=Y -e SA_PASSWORD=Password1! -p 1433:1433 -v c:\temp\:/var/opt/mssql/data --name foobar_db -d foobar_db:1.0
docker run -p 8080:80 --name foobar --link foobar_db:foobar_db -d foobar:1.0
My containers spin up, my database is deployed just fine. From the host, I can use SQL Server Management Studio and Python, and connect to my database container using the credentials above, and connect and perform read/writes perfectly.
When I connect from the .NET using
Server=host.docker.internal;Database=FOOBAR;User Id=sa;Password=Password1!;
I can see my SQL Server container complain about an invalid login,
Login failed for user '6794cfd81d48\Guest'
where I can confirm that 6794cfd81d48 is the hash of my SQL Server container, foobar_db.
IIS serves up webpages just fine, the problem is connecting to the database. Even though I am providing the correct username and password, I am unable to connect from another container to the SQL Server container because it thinks that I am a guest to that container. Depending on the deployment environment, normally I would create SQL Server logins, either for a machine or for a user, but not in this case.
There was an offending IT security software application that was identified as the cause. IT set a passthrough so the docker applications could by the security and everything worked as designed.

How to access SQL server from docker ASP.net core API image

I have generated a docker file for asp.net core API with a single page application thanks to Visual studio. After some research on the web I correct differents trouble about SPA in this docker file.
Finnaly my trouble is the connexion with our database server.
When I tried to connect, I've got a
Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server.
It seems that it appears because my container could not acces to the server, after hour and hour of google search, I only found solution with a SQL hosted in docker image.
How to all my docker image of wab app accessing the entire company network to access different server ? I use computer name ant not IP to match company requirement.
Thanks for all
Versions :
.net core api : 3.1
I'm using docker for Windows
docker use linux container
Here is my docker file
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
RUN apt-get update -yq \
&& apt-get install curl gnupg -yq \
&& curl -sL https://deb.nodesource.com/setup_10.x | bash \
&& apt-get install nodejs -yq
WORKDIR /src
COPY ["Company.Dtm.WebApi.AppWebApi/Company.Dtm.WebApi.AppWebApi.csproj", "Company.Dtm.WebApi.AppWebApi/"]
COPY ["CompanyFramework/Company.Framework.WebApi/Company.Framework.WebApi.csproj", "CompanyFramework/Company.Framework.WebApi/"]
COPY ["CompanyFramework/Company.Framework.Model/Company.Framework.Model.csproj", "CompanyFramework/Company.Framework.Model/"]
COPY ["CompanyFramework/Company.Framework.Tools/Company.Framework.Tools.csproj", "CompanyFramework/Company.Framework.Tools/"]
COPY ["AppLib/Company.Dtm.Lib.AppLib/Company.Dtm.Lib.AppLib.csproj", "AppLib/Company.Dtm.Lib.AppLib/"]
RUN dotnet restore "Company.Dtm.WebApi.AppWebApi/Company.Dtm.WebApi.AppWebApi.csproj"
COPY . .
WORKDIR "/src/Company.Dtm.WebApi.AppWebApi"
RUN dotnet build "Company.Dtm.WebApi.AppWebApi.csproj" -c Debug -o /app/build
FROM build AS publish
RUN dotnet publish "Company.Dtm.WebApi.AppWebApi.csproj" -c Debug -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Company.Dtm.WebApi.AppWebApi.dll"]
Here is my docker-compose
version: '3'
services:
webapp:
build: .
network_mode: "bridge"
ports:
- "8880:80"
I also had this issue, just trying to connect to my localhost development SQL Server.
What ended up working was to add the normal SQL Server ports to my Dockerfile:
EXPOSE 1433
EXPOSE 5000 [or whatever other ports you may be using.]
Then set up a firewall Inbound Rule to allow those ports.
You cannot use 'localhost' obviously since the 'localhost' is the host the container it is running in, but I did find with Windows at least, that I can simply use my dev machine name as the server, so it seems that DNS works across the nat. I would think you should be able to access any network resource at that point, but I would say your firewall[s] might be a place to start. Your Docker container acts like an external network and therefore generally un-trusted.
I also found that I did not have a 'bridge' network. Maybe you get that with the Linux container.
My >docker network ls command revealed a "Default Switch" network, but no "bridge". Because this is Docker for Windows, there is no 'host' option.
That was all there was to it for me. I see a lot of other posts talking about a lot of other things, but honestly, just opening up the firewall is what did the trick. Good luck!
You need to add another service for your db in your compose file.
Something like this:
version: "3"
services:
web:
build: .
ports:
- "8000:80"
depends_on:
- db
db:
image: "mcr.microsoft.com/mssql/server"
environment:
SA_PASSWORD: "Your_password123"
ACCEPT_EULA: "Y"
make sure to replace the password in the SA_PASSWORD environment variable under db.

Is it possible to create custom Linux-based Docker image on Azure Windows Server DSVM

I am using an Azure DSVM in a DevTest Lab running Windows Server 2019. I am trying to get Docker installed and working to allow me to run local experiments from Azure ML Service environments.
I want to build a custom Linux container on Docker - which I believe is possible on Windows from reading some other online posts (I can't use a Linux host for various reasons). When I try to create such an image that contains a WORKDIR ... step, I get a "container ***** encountered an error during CreateProcess: failure in a Windows system call" error.
I installed Docker on the DSVM (which is a Standard D2s_v3) by adding the "Docker" artifact at creation and then running the following commands to enable Linux containers:
$> Install-WindowsFeature -Name Hyper-V -IncludeManagementTools -Restart
$> [Environment]::SetEnvironmentVariable("LCOW_SUPPORTED", "1", "Machine")
Running a simple Linux container works fine:
$> docker run --rm -it alpine:latest
/ # ls
bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var
/ #
To build a custom image, I'm using a simple Dockerfile as follows:
FROM alpine:latest
WORKDIR /abm
The image appears to build successfully:
$> docker build --no-cache -t abm-alpine:workdir -f .\abm-alpine.Dockerfile .
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM alpine:latest
---> a187dde48cd2
Step 2/2 : WORKDIR /abm
---> 495f8ecb3a0e
Removing intermediate container 219e91296e47
Successfully built 495f8ecb3a0e
Successfully tagged abm-alpine:workdir
When I run the image, I get the following error:
$> docker run --rm -it abm-alpine:workdir
C:\Program Files\Docker\docker.exe: Error response from daemon: container 01fad57c971d672d91238a6c6ec21376e033006ec4c26563e91e7288cfb3bfeb encountered an error during CreateProcess: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106) extra info: {"CommandArgs":["/bin/sh"],"WorkingDirectory":"/abm","Environment":{"HOSTNAME":"01fad57c971d","PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","TERM":"xterm"},"EmulateConsole":true,"CreateStdInPipe":true,"CreateStdOutPipe":true,"ConsoleSize":[50,120],"OCISpecification":{"ociVersion":"1.0.0","process":{"terminal":true,"consoleSize":{"height":50,"width":120},"user":{"uid":0,"gid":0},"args":["/bin/sh"],"env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME=01fad57c971d","TERM=xterm"],"cwd":"/abm","capabilities":{"bounding":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"effective":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"inheritable":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"permitted":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"]}},"root":{"path":"rootfs"},"hostname":"01fad57c971d","mounts":[{"destination":"/proc","type":"proc","source":"proc","options":["nosuid","noexec","nodev"]},{"destination":"/dev","type":"tmpfs","source":"tmpfs","options":["nosuid","strictatime","mode=755","size=65536k"]},{"destination":"/dev/pts","type":"devpts","source":"devpts","options":["nosuid","noexec","newinstance","ptmxmode=0666","mode=0620","gid=5"]},{"destination":"/sys","type":"sysfs","source":"sysfs","options":["nosuid","noexec","nodev","ro"]},{"destination":"/sys/fs/cgroup","type":"cgroup","source":"cgroup","options":["ro","nosuid","noexec","nodev"]},{"destination":"/dev/mqueue","type":"mqueue","source":"mqueue","options":["nosuid","noexec","nodev"]},{"destination":"/dev/shm","type":"tmpfs","source":"shm","options":["nosuid","noexec","nodev","mode=1777"]}],"linux":{"resources":{"devices":[{"allow":false,"access":"rwm"},{"allow":true,"type":"c","major":1,"minor":5,"access":"rwm"},{"allow":true,"type":"c","major":1,"minor":3,"access":"rwm"},{"allow":true,"type":"c","major":1,"minor":9,"access":"rwm"},{"allow":true,"type":"c","major":1,"minor":8,"access":"rwm"},{"allow":true,"type":"c","major":5,"minor":0,"access":"rwm"},{"allow":true,"type":"c","major":5,"minor":1,"access":"rwm"},{"allow":false,"type":"c","major":10,"minor":229,"access":"rwm"}]},"namespaces":[{"type":"mount"},{"type":"network"},{"type":"uts"},{"type":"pid"},{"type":"ipc"}],"maskedPaths":["/proc/kcore","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug"],"readonlyPaths":["/proc/asound","/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]},"windows":{"layerFolders":["C:\\ProgramData\\docker\\lcow\\5ba6a7b4fbdf9748ec89898be9bdaa911ee614436a475945638ab296b1155966","C:\\ProgramData\\docker\\lcow\\01fad57c971d672d91238a6c6ec21376e033006ec4c26563e91e7288cfb3bfeb"],"hyperv":{},"network":{"endpointList":["D615E3D5-B6AA-401E-A0A0-72581FA47059"],"allowUnqualifiedDNSQuery":true}}}}.
I've tried various logs (e.g. Get-WinEvent -LogName Microsoft-Windows-Hyper-V-Compute-Operational and Get-EventLog -LogName Application -Source Docker) but cannot see any additional information about the error.
Can anyone advise if it is possible to create custom Linux-based images on a Windows DSVM? If it is, can anyone advise what the problem may be or any additional troubleshooting steps I could take?
Thanks!
It is possible to create Linux container on Windows Server.
Although this is currently in experimental stage.
This article might help : https://www.b2-4ac.com/lcow-linux-containers-on-windows-server/

Resources