Background
I am writing a .NET 5 application and using .net user secrets for my secret keys (database connections & passwords).
Recently I decided to learn Dockers and update my application to work with it so that using Visual Studio I generated a docker file for my API project and then created a docker-compose file that includes the API project & database (and some more irrelevant things for this question).
Almost everything works well. Technically, I can hard-code the secrets, and then the application will work well.
I have some secrets and most of them work fine, e.g: the database connection secrets works well, in the C# code I do the following code and it gets the value from the .net user-secrets:
config.GetConnectionString("Default");
Code Details
I have a secret key that contains a SQL password for the sa user.
dotnet user-secrets set "SA_PASSWORD" "<MySecretPassword>"
Then I have the docker-compose file which is of Linux system and this is part of the code:
sql_in_dc:
build:
context: .
dockerfile: items/sql/sql.Dockerfile
restart: always
ports:
- "1440:1433"
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=$SA_PASSWORD
- ASPNETCORE_ENVIRONMENT=Development
- USER_SECRETS_ID=80a155b1-fb7a-44de-8788-4f5759c60ff6
volumes:
- $APPDATA/Microsoft/UserSecrets/$USER_SECRETS_ID:/root/.microsoft/usersecrets/$USER_SECRETS_ID
- $HOME/.microsoft/usersecrets/$USER_SECRETS_ID:/root/.microsoft/usersecrets/$USER_SECRETS_ID
As you can see it calls the sql.Dockerfile which is:
FROM mcr.microsoft.com/mssql/server
ARG PROJECT_DIR=/tmp/devdatabase
RUN mkdir -p $PROJECT_DIR
WORKDIR $PROJECT_DIR
COPY items/sql/InitializeDatabase.sql ./
COPY items/sql/wait-for-it.sh ./
COPY items/sql/entrypoint.sh ./
COPY items/sql/setup.sh ./
CMD ["/bin/bash", "entrypoint.sh"]
Then the setup.sh is:
# Wait for SQL Server to be started and then run the sql script
./wait-for-it.sh sql_in_dc:1433 --timeout=0 --strict -- sleep 5s && \
/opt/mssql-tools/bin/sqlcmd -S localhost -i InitializeDatabase.sql -U sa -P "$SA_PASSWORD"
The Problem
The file setup.sh doesn't recognizes the $SA_PASSWORD environment variable when it comes from the secrets file.
It works well if I change the docker-compose.yml file to:
- SA_PASSWORD=SomePassword
Notes
I searched for an answer in Google and tried many things but couldn't find exactly my case.
I know it is possible to use Docker Swarm for the secrets but for now I want to do it without it. I am still learning and prefer that the code will work good and the next step will be to use Docker Swarm / Kubernetes / etc...
I would be happy to know if there is a fast solution even if it is not the ideal one. Later I will improve it and use better techniques.
I wrote the code I think that should be enough for the case and the relevant code, but if you need any more data, let me know and I will add it.
I have it in GitHub in a public repository in a pushed branch. If you want I can share with you the code.
Really big thanks in advance!
The docker-compose.yml is executed on your host OS (so it can use OS environment variables or vars from .env file, or from compose file, ...).
The running image - container has it's own set of env variables, in your case that means the running container has no SA_PASSWORD variable.
Your usecase would work if you had set the SA_PASSWORD Variable on your host OS.
You can check which variables are set in your container with (if your image comes with bash):
docker exec -it [container id] bash
printenv
The dotnet-secrets environment variables are created implicit during execution/runtime from Visual Studio (see entry in project file).
So as you mentioned "the compose file can't recognize dotnet-secrets".
You can use:
*.env File
pass it to compose command: with -e
Plain text in compose yml: - SA_PASSWORD=thepassword
as host OS Variable
Keep in mind that Visual Studio adds some magic when running or debugging your docker container. See Visual Studio container volume mapping: For ASP.NET core web apps, there might be two additional folders for the SSL certificate and the user secrets, which is explained in more detail in the next section.
I have a container deploying a front end in ASP.NET Core trying to connect to the backend SQL Server database. I am running windows 10 with Docker desktop v19.03.13.
The website container is built on
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
WORKDIR /app
EXPOSE 80
# Copy csproj and restore as distinct layers
COPY ./FOOBAR/*.csproj ./
RUN dotnet restore
# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out
# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "FOOBAR.dll"]
The database is built on
FROM mcr.microsoft.com/mssql/server:2017-latest
USER root
COPY setup.sql setup.sql
COPY import-data.sh import-data.sh
COPY entrypoint.sh entrypoint.sh
RUN chmod +x entrypoint.sh
CMD /bin/bash ./entrypoint.sh
Everything works marvelous when running outside docker, .NET, Python, SQL Server Management Studio.
In .NET, my connection string is:
Server=localhost;Database=FOOBAR;Integrated Security=True
In Python:
DRIVER={ODBC Driver 17 for SQL Server};server=localhost;database=FOOBAR;Trusted_Connection=yes;
So I need to deploy this to a network that does not have a domain controller so I need to handle all database authentication.
When I build my containers, I change my .NET connection string to
dbConnection="Server=host.docker.internal;Database=FOOBAR;User Id=sa;Password=Password1!;"
I spawn my containers with
docker run -e ACCEPT_EULA=Y -e SA_PASSWORD=Password1! -p 1433:1433 -v c:\temp\:/var/opt/mssql/data --name foobar_db -d foobar_db:1.0
docker run -p 8080:80 --name foobar --link foobar_db:foobar_db -d foobar:1.0
My containers spin up, my database is deployed just fine. From the host, I can use SQL Server Management Studio and Python, and connect to my database container using the credentials above, and connect and perform read/writes perfectly.
When I connect from the .NET using
Server=host.docker.internal;Database=FOOBAR;User Id=sa;Password=Password1!;
I can see my SQL Server container complain about an invalid login,
Login failed for user '6794cfd81d48\Guest'
where I can confirm that 6794cfd81d48 is the hash of my SQL Server container, foobar_db.
IIS serves up webpages just fine, the problem is connecting to the database. Even though I am providing the correct username and password, I am unable to connect from another container to the SQL Server container because it thinks that I am a guest to that container. Depending on the deployment environment, normally I would create SQL Server logins, either for a machine or for a user, but not in this case.
There was an offending IT security software application that was identified as the cause. IT set a passthrough so the docker applications could by the security and everything worked as designed.
I have generated a docker file for asp.net core API with a single page application thanks to Visual studio. After some research on the web I correct differents trouble about SPA in this docker file.
Finnaly my trouble is the connexion with our database server.
When I tried to connect, I've got a
Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server.
It seems that it appears because my container could not acces to the server, after hour and hour of google search, I only found solution with a SQL hosted in docker image.
How to all my docker image of wab app accessing the entire company network to access different server ? I use computer name ant not IP to match company requirement.
Thanks for all
Versions :
.net core api : 3.1
I'm using docker for Windows
docker use linux container
Here is my docker file
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
RUN apt-get update -yq \
&& apt-get install curl gnupg -yq \
&& curl -sL https://deb.nodesource.com/setup_10.x | bash \
&& apt-get install nodejs -yq
WORKDIR /src
COPY ["Company.Dtm.WebApi.AppWebApi/Company.Dtm.WebApi.AppWebApi.csproj", "Company.Dtm.WebApi.AppWebApi/"]
COPY ["CompanyFramework/Company.Framework.WebApi/Company.Framework.WebApi.csproj", "CompanyFramework/Company.Framework.WebApi/"]
COPY ["CompanyFramework/Company.Framework.Model/Company.Framework.Model.csproj", "CompanyFramework/Company.Framework.Model/"]
COPY ["CompanyFramework/Company.Framework.Tools/Company.Framework.Tools.csproj", "CompanyFramework/Company.Framework.Tools/"]
COPY ["AppLib/Company.Dtm.Lib.AppLib/Company.Dtm.Lib.AppLib.csproj", "AppLib/Company.Dtm.Lib.AppLib/"]
RUN dotnet restore "Company.Dtm.WebApi.AppWebApi/Company.Dtm.WebApi.AppWebApi.csproj"
COPY . .
WORKDIR "/src/Company.Dtm.WebApi.AppWebApi"
RUN dotnet build "Company.Dtm.WebApi.AppWebApi.csproj" -c Debug -o /app/build
FROM build AS publish
RUN dotnet publish "Company.Dtm.WebApi.AppWebApi.csproj" -c Debug -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Company.Dtm.WebApi.AppWebApi.dll"]
Here is my docker-compose
version: '3'
services:
webapp:
build: .
network_mode: "bridge"
ports:
- "8880:80"
I also had this issue, just trying to connect to my localhost development SQL Server.
What ended up working was to add the normal SQL Server ports to my Dockerfile:
EXPOSE 1433
EXPOSE 5000 [or whatever other ports you may be using.]
Then set up a firewall Inbound Rule to allow those ports.
You cannot use 'localhost' obviously since the 'localhost' is the host the container it is running in, but I did find with Windows at least, that I can simply use my dev machine name as the server, so it seems that DNS works across the nat. I would think you should be able to access any network resource at that point, but I would say your firewall[s] might be a place to start. Your Docker container acts like an external network and therefore generally un-trusted.
I also found that I did not have a 'bridge' network. Maybe you get that with the Linux container.
My >docker network ls command revealed a "Default Switch" network, but no "bridge". Because this is Docker for Windows, there is no 'host' option.
That was all there was to it for me. I see a lot of other posts talking about a lot of other things, but honestly, just opening up the firewall is what did the trick. Good luck!
You need to add another service for your db in your compose file.
Something like this:
version: "3"
services:
web:
build: .
ports:
- "8000:80"
depends_on:
- db
db:
image: "mcr.microsoft.com/mssql/server"
environment:
SA_PASSWORD: "Your_password123"
ACCEPT_EULA: "Y"
make sure to replace the password in the SA_PASSWORD environment variable under db.
I am using an Azure DSVM in a DevTest Lab running Windows Server 2019. I am trying to get Docker installed and working to allow me to run local experiments from Azure ML Service environments.
I want to build a custom Linux container on Docker - which I believe is possible on Windows from reading some other online posts (I can't use a Linux host for various reasons). When I try to create such an image that contains a WORKDIR ... step, I get a "container ***** encountered an error during CreateProcess: failure in a Windows system call" error.
I installed Docker on the DSVM (which is a Standard D2s_v3) by adding the "Docker" artifact at creation and then running the following commands to enable Linux containers:
$> Install-WindowsFeature -Name Hyper-V -IncludeManagementTools -Restart
$> [Environment]::SetEnvironmentVariable("LCOW_SUPPORTED", "1", "Machine")
Running a simple Linux container works fine:
$> docker run --rm -it alpine:latest
/ # ls
bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var
/ #
To build a custom image, I'm using a simple Dockerfile as follows:
FROM alpine:latest
WORKDIR /abm
The image appears to build successfully:
$> docker build --no-cache -t abm-alpine:workdir -f .\abm-alpine.Dockerfile .
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM alpine:latest
---> a187dde48cd2
Step 2/2 : WORKDIR /abm
---> 495f8ecb3a0e
Removing intermediate container 219e91296e47
Successfully built 495f8ecb3a0e
Successfully tagged abm-alpine:workdir
When I run the image, I get the following error:
$> docker run --rm -it abm-alpine:workdir
C:\Program Files\Docker\docker.exe: Error response from daemon: container 01fad57c971d672d91238a6c6ec21376e033006ec4c26563e91e7288cfb3bfeb encountered an error during CreateProcess: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106) extra info: {"CommandArgs":["/bin/sh"],"WorkingDirectory":"/abm","Environment":{"HOSTNAME":"01fad57c971d","PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","TERM":"xterm"},"EmulateConsole":true,"CreateStdInPipe":true,"CreateStdOutPipe":true,"ConsoleSize":[50,120],"OCISpecification":{"ociVersion":"1.0.0","process":{"terminal":true,"consoleSize":{"height":50,"width":120},"user":{"uid":0,"gid":0},"args":["/bin/sh"],"env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME=01fad57c971d","TERM=xterm"],"cwd":"/abm","capabilities":{"bounding":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"effective":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"inheritable":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"permitted":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"]}},"root":{"path":"rootfs"},"hostname":"01fad57c971d","mounts":[{"destination":"/proc","type":"proc","source":"proc","options":["nosuid","noexec","nodev"]},{"destination":"/dev","type":"tmpfs","source":"tmpfs","options":["nosuid","strictatime","mode=755","size=65536k"]},{"destination":"/dev/pts","type":"devpts","source":"devpts","options":["nosuid","noexec","newinstance","ptmxmode=0666","mode=0620","gid=5"]},{"destination":"/sys","type":"sysfs","source":"sysfs","options":["nosuid","noexec","nodev","ro"]},{"destination":"/sys/fs/cgroup","type":"cgroup","source":"cgroup","options":["ro","nosuid","noexec","nodev"]},{"destination":"/dev/mqueue","type":"mqueue","source":"mqueue","options":["nosuid","noexec","nodev"]},{"destination":"/dev/shm","type":"tmpfs","source":"shm","options":["nosuid","noexec","nodev","mode=1777"]}],"linux":{"resources":{"devices":[{"allow":false,"access":"rwm"},{"allow":true,"type":"c","major":1,"minor":5,"access":"rwm"},{"allow":true,"type":"c","major":1,"minor":3,"access":"rwm"},{"allow":true,"type":"c","major":1,"minor":9,"access":"rwm"},{"allow":true,"type":"c","major":1,"minor":8,"access":"rwm"},{"allow":true,"type":"c","major":5,"minor":0,"access":"rwm"},{"allow":true,"type":"c","major":5,"minor":1,"access":"rwm"},{"allow":false,"type":"c","major":10,"minor":229,"access":"rwm"}]},"namespaces":[{"type":"mount"},{"type":"network"},{"type":"uts"},{"type":"pid"},{"type":"ipc"}],"maskedPaths":["/proc/kcore","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug"],"readonlyPaths":["/proc/asound","/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]},"windows":{"layerFolders":["C:\\ProgramData\\docker\\lcow\\5ba6a7b4fbdf9748ec89898be9bdaa911ee614436a475945638ab296b1155966","C:\\ProgramData\\docker\\lcow\\01fad57c971d672d91238a6c6ec21376e033006ec4c26563e91e7288cfb3bfeb"],"hyperv":{},"network":{"endpointList":["D615E3D5-B6AA-401E-A0A0-72581FA47059"],"allowUnqualifiedDNSQuery":true}}}}.
I've tried various logs (e.g. Get-WinEvent -LogName Microsoft-Windows-Hyper-V-Compute-Operational and Get-EventLog -LogName Application -Source Docker) but cannot see any additional information about the error.
Can anyone advise if it is possible to create custom Linux-based images on a Windows DSVM? If it is, can anyone advise what the problem may be or any additional troubleshooting steps I could take?
Thanks!
It is possible to create Linux container on Windows Server.
Although this is currently in experimental stage.
This article might help : https://www.b2-4ac.com/lcow-linux-containers-on-windows-server/