I am trying to build a PostgreSQL image in DockerManually becuase I need to add some certain configuration for the database that doesnot exit in the postgres images provided with docker.
I wrote the following dockerfile
FROM alpine:3.7
#update the OS
RUN apk update
#installing alpine software kit
RUN apk add alpine-sdk
RUN apk update \
postgresql-$PG_MAJOR-postgis-$POSTGISV \
postgresql-$PG_MAJOR-postgis-$POSTGISV-scripts \
postgresql-$PG_MAJOR-pgrouting \
postgresql-$PG_MAJOR-pgrouting-scripts \
postgresql-server-dev-$PG_MAJOR
#installing postgresql
RUN apk add postgresql
RUN apk add openrc --no-cache
RUN apk update && apk upgrade
RUN apk add --no-cache --virtual .base_deps build-base openssl-dev zlib-dev libxml2-dev wget gnupg ca-certificates
RUN apk add --no-cache readline-dev glib-lang libssl1.0 postgresql postgresql-client
RUN apk add --update binutils
RUN apk add --no-cache sudo
RUN apk --purge del .base_deps
RUN echo "postgres ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/postgres
RUN chmod 600 /etc/sudoers.d/postgres
RUN sync
#RUN /etc/init.d/postgresql start
#RUN postgres -D /usr/local/pgsql/data
EXPOSE 5432
then I am trying to lunch the dockerfile with interactive shell with
docker run -it processID
then the follwoing command to instantite database
initdb Database
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locales
COLLATE: C
CTYPE: C.UTF-8
MESSAGES: C
MONETARY: C
NUMERIC: C
TIME: C
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory Database ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... sh: locale: not found
2019-11-18 08:04:06.670 UTC [89] WARNING: no usable system locales were found
ok
syncing data to disk ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success.
Then I am trying to lunch the database using the following command
postgres -D Database
2019-11-18 08:05:27.358 UTC [91] LOG: listening on IPv4 address "127.0.0.1", port 5432
2019-11-18 08:05:27.358 UTC [91] LOG: could not bind IPv6 address "::1": Address not available
2019-11-18 08:05:27.358 UTC [91] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2019-11-18 08:05:27.360 UTC [91] FATAL: could not create lock file "/run/postgresql/.s.PGSQL.5432.lock": No such file or directory
2019-11-18 08:05:27.360 UTC [91] LOG: database system is shut down
I am getting some errors I do not know exactly why? any help would really be appreciated.
I did switch yo docker-compose, and here is an example of the file
version: "3"
services:
# Create a service named db.
db:
# Use the Docker Image postgres. This will pull the newest release.
image: "postgres:12.1-alpine"
# Give the container the name my_postgres. You can changes to something else.
container_name: "theNameOfDockerCompose"
# Setup the username, password, and database name. You can changes these values.
environment:
- POSTGRES_USER=DatabaseName
- POSTGRES_PASSWORD=Password
- POSTGRES_DB=development
# Maps port 54320 (localhost) to port 5432 on the container. You can change the ports to fix your needs.
ports:
- "5432:5432"
# Set a volume some that database is not lost after shutting down the container.
# I used the name postgres-data but you can changed it to something else.
volumes:
- ./postgres-data:/var/lib/postgresql/data
This is file is what I am using now.
But with Dockerfile, I still cannot enable the endpoints to access the dataBase.
Possible Duplicate of postgres-docker-bind-address-ipv6
update listen_addresses = '*' into postgresql/data/postgresql.conf
2019-11-18 08:05:27.360 UTC [91] FATAL: could not create lock file "/run/postgresql/.s.PGSQL.5432.lock": No such file or directory
points to a missing folder. Add
RUN mkdir -p /run/postgresql
to your Dockerfile
In addition, you need to change the listen_addresses = "*" in your postgresql/data/postgresql.conf. Otherwise, your database will be bound to localhost in your container and you will not be able to access it from anywhere except within that particular container. You can see that in your logs here
2019-11-18 08:05:27.358 UTC [91] LOG: listening on IPv4 address "127.0.0.1", port 5432
for the localhost binding. In addition, this will fix the following error message in your log
2019-11-18 08:05:27.358 UTC [91] LOG: could not bind IPv6 address "::1": Address not available
in which postgres failed to the bind for ipv6.
Related
When I run latest sql server image from official documentation on linux host.
docker run -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=asdasdasdsad' -p 1433:1433 -v ./data:/var/opt/mssql/data -d mcr.microsoft.com/mssql/server:2019-latest
I get error:
ERROR: Setup FAILED copying system data file 'C:\templatedata\model_replicatedmaster.mdf' to '/var/opt/mssql/data/model_replicatedmaster.mdf': 5(Access is denied.)
This message occurs only on Linux host and with binded volumes.
I happen because lack of permission. On 2019 mssql docker move from root user images into not-root. It made that docker sql-server containers with binded volumes and run on Linux host has a permission issue (=> has no permission to write into binded volume).
There are few solution for this problem:
1. Run docker as root.
eg. compose:
version: '3.6'
services:
mssql:
image: mcr.microsoft.com/mssql/server:2019-latest
user: root
ports:
- 1433:1433
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=BLAH
volumes:
- ./data:/var/opt/mssql/data
Source: https://github.com/microsoft/mssql-docker/issues/13#issuecomment-641904197
2. Setup proper directory owner (mssql)
Check id for mssql user on docker image
sudo docker run -it mcr.microsoft.com/mssql/server id mssql
gives: uid=10001(mssql) gid=0(root) groups=0(root)
Change folder's owner
sudo chown 10001 VOLUME_DIRECTORY
Source in Spanish: https://www.eiximenis.dev/posts/2020-06-26-sql-server-docker-no-se-ejecuta-en-root/
3. Give a full access (not recommended)
Give full access to db files on host
sudo chmod 777 -R VOLUME_DIRECTORY
Unfortunately, the only way I found to fix this issue involves a few manual steps.
I used the following docker-compose file for this to work
version: '3.9'
services:
mssql:
image: mcr.microsoft.com/mssql/server:2019-latest
platform: linux
ports:
- 1433:1433
environment:
- ACCEPT_EULA=Y
- MSSQL_SA_PASSWORD=<testPASSWORDthatISlongENOUGH_1234>
volumes:
- ./mssql/data:/var/opt/mssql/data
- ./backups:/var/backups
(the data directory has to be mounted directly due to another issue with SQL server containers hosted on Windows machines)
Then you need to perform the following manual steps:
Connect to the database using SSMS
Find and select your .bak database backup file
Open a terminal in the container
In the directory that the .mdf and .ldf files are going to be created, touch files with the database name you are going to use
touch /var/opt/mssql/data/DATABASE_NAME.mdf
touch /var/opt/mssql/data/DATABASE_NAME_log.ldf
Toggle the option to replace any existing database with the restore
Restore your database
I tried to follow the instructions in this https://www.sqlservercentral.com/blogs/using-volumes-in-sql-server-2019-non-root-containers article but I could not get it to work.
This problem was also discussed in this github issue (which the bot un-helpfully closed without a proper solution).
I encoutered the same problem as you trying to run a container based on sql server on DigitalOcean. user: root also solved the issue.
I have generated a docker file for asp.net core API with a single page application thanks to Visual studio. After some research on the web I correct differents trouble about SPA in this docker file.
Finnaly my trouble is the connexion with our database server.
When I tried to connect, I've got a
Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server.
It seems that it appears because my container could not acces to the server, after hour and hour of google search, I only found solution with a SQL hosted in docker image.
How to all my docker image of wab app accessing the entire company network to access different server ? I use computer name ant not IP to match company requirement.
Thanks for all
Versions :
.net core api : 3.1
I'm using docker for Windows
docker use linux container
Here is my docker file
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
RUN apt-get update -yq \
&& apt-get install curl gnupg -yq \
&& curl -sL https://deb.nodesource.com/setup_10.x | bash \
&& apt-get install nodejs -yq
WORKDIR /src
COPY ["Company.Dtm.WebApi.AppWebApi/Company.Dtm.WebApi.AppWebApi.csproj", "Company.Dtm.WebApi.AppWebApi/"]
COPY ["CompanyFramework/Company.Framework.WebApi/Company.Framework.WebApi.csproj", "CompanyFramework/Company.Framework.WebApi/"]
COPY ["CompanyFramework/Company.Framework.Model/Company.Framework.Model.csproj", "CompanyFramework/Company.Framework.Model/"]
COPY ["CompanyFramework/Company.Framework.Tools/Company.Framework.Tools.csproj", "CompanyFramework/Company.Framework.Tools/"]
COPY ["AppLib/Company.Dtm.Lib.AppLib/Company.Dtm.Lib.AppLib.csproj", "AppLib/Company.Dtm.Lib.AppLib/"]
RUN dotnet restore "Company.Dtm.WebApi.AppWebApi/Company.Dtm.WebApi.AppWebApi.csproj"
COPY . .
WORKDIR "/src/Company.Dtm.WebApi.AppWebApi"
RUN dotnet build "Company.Dtm.WebApi.AppWebApi.csproj" -c Debug -o /app/build
FROM build AS publish
RUN dotnet publish "Company.Dtm.WebApi.AppWebApi.csproj" -c Debug -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Company.Dtm.WebApi.AppWebApi.dll"]
Here is my docker-compose
version: '3'
services:
webapp:
build: .
network_mode: "bridge"
ports:
- "8880:80"
I also had this issue, just trying to connect to my localhost development SQL Server.
What ended up working was to add the normal SQL Server ports to my Dockerfile:
EXPOSE 1433
EXPOSE 5000 [or whatever other ports you may be using.]
Then set up a firewall Inbound Rule to allow those ports.
You cannot use 'localhost' obviously since the 'localhost' is the host the container it is running in, but I did find with Windows at least, that I can simply use my dev machine name as the server, so it seems that DNS works across the nat. I would think you should be able to access any network resource at that point, but I would say your firewall[s] might be a place to start. Your Docker container acts like an external network and therefore generally un-trusted.
I also found that I did not have a 'bridge' network. Maybe you get that with the Linux container.
My >docker network ls command revealed a "Default Switch" network, but no "bridge". Because this is Docker for Windows, there is no 'host' option.
That was all there was to it for me. I see a lot of other posts talking about a lot of other things, but honestly, just opening up the firewall is what did the trick. Good luck!
You need to add another service for your db in your compose file.
Something like this:
version: "3"
services:
web:
build: .
ports:
- "8000:80"
depends_on:
- db
db:
image: "mcr.microsoft.com/mssql/server"
environment:
SA_PASSWORD: "Your_password123"
ACCEPT_EULA: "Y"
make sure to replace the password in the SA_PASSWORD environment variable under db.
I'm trying to connect to my remote SQL server from a docker container hosted on my computer.
But are just reciving the following error:
A network-related or in stance-specific error has occurred while establishing a connection to SQL Server . Server is not found or not accessible. Check if instance name is correct and i f SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.. Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : Login timeout expired.
But if I try to connect to my SQL server from SQL server management studio on the host machine everything works properly.
Also note that 2 weeks ago everything also worked inside the docker container.
Here is my docker compose file and the docker file which has the SQL driver installed:
Compose:
version: '3'
services:
nginx:
image: nginx:1.10
volumes:
- ./:/var/www
- .docker/nginx/vhost.conf:/etc/nginx/conf.d/default.conf
ports:
- ${DOCKER_IP}80:80
links:
- php
networks:
- app-net
php:
build:
context: ./
dockerfile: .docker/php/DockerFile
volumes:
- ./:/var/www
networks:
- app-net
networks:
app-net:
driver: bridge
Docker file
FROM phpdockerio/php71-fpm:latest
# Install selected extensions and other stuff
RUN export DEBIAN_FRONTEND=noninteractive && \
apt-get update && apt-get -y --no-install-recommends install \
php7.1-mysql \
php7.1-mbstring \
php7.1-gd \
php7.1-soap \
php7.1-dev \
apt-transport-https \
git \
ssh \
curl \
php-pear \
&& apt-get clean; rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /usr/share/doc/*
# Install Composer
RUN cd /usr/src
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
# Install MSSQL extention
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get -y install msodbcsql mssql-tools g++ unixodbc-dev make
RUN pear config-set php_ini `php --ini | grep "Loaded Configuration" | sed -e "s|.*:\s*||"` system
RUN pecl install sqlsrv
RUN pecl install pdo_sqlsrv
RUN echo "extension=sqlsrv.so" >> /etc/php/7.1/fpm/php.ini
RUN echo "extension=pdo_sqlsrv.so" >> /etc/php/7.1/fpm/php.ini
# Fixed locals for MSSQL extention
RUN apt-get install -y locales
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen
RUN locale-gen
WORKDIR /var/www
Inside the docker container I can't ping the SQL server. So for me i sounds like a network issue, but I'm unable find a solution.
Please note the SQL server is hosted locally on a server in the office.
UPDATE/Solved for now
After downgrading the dokcer for windows to 18.03.0-CE everything worked as expected.
Docker Bridge networks don't connect to the world outside of Docker at all by default; they only allow containers to talk to each other. The documentation for Docker Bridge Networks does offer some advice for allowing Bridge network traffic to talk to the outside world by making changes on the Docker host:
First enable IP forwarding in the kernel:
$ sysctl net.ipv4.conf.all.forwarding=1
Then change the host firewall to allow the forwarding
$ sudo iptables -P FORWARD ACCEPT
This is a fairly permissive firewall configuration, so you may want to look at keeping that a bit more locked down.
The other way would be to attach the container to two different networks: your current bridge network for communication between container, and a second Host network for talking to MySQL. Since this bypasses all of Docker's NAT configuration, the scalability of your service may be impacted, although I think outgoing connections might be OK.
For my case:
I ran these below:
To see all containers:
docker ps -a
Then restart container:
docker container restart yourIdContainer
If there is error: Error response from daemon: Cannot restart container ...
Please restart Docker then restart container again.
Connect to Sql Server by MSSM, server name: localhost,1433 or IP
Hope this is helpful.
If you are using Windows 10 (HOME VERSION ) may you don't have the virtualization and you are using (docker toolbox ), they have solved the problem using a static ip. Check it.
I mean localhost won't work if you are using docker toolbox, search the ip where is virtualized docker toolbox.
My case ip: 192.168.99.100
EDIT-2
I found out that the database doesn't even start after making the file location change.
This is with the default file location:
$pg_isready
/var/run/postgresql:5432 - accepting connections
$pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
9.5 main 5432 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
pg_lsclusters output is green.
After the file location has changed on postgresql.conf:
$pg_isready
/var/run/postgresql:5432 - no response
$pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
9.5 main 5432 down root /mnt/Data/postgresdb/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
Here the output is red.
Following this post here, I tried to start the cluster manually:
$pg_ctlcluster 9.5 main start
Warning: the cluster will not be running as a systemd service. Consider using systemctl:
sudo systemctl start postgresql#9.5-main
Error: You must run this program as the cluster owner (root) or root
I tried the same command with sudo:
Error: Config owner (postgres:124) and data owner (root:0) do not match, and config owner is not root
Which again makes me think the problem might lie with permissions of the directory. The directory is owned by root whose ownership I am unable to change.
EDIT-1
I've been working on this and I'd like to distill this post further to give more specifics. This is my current situation:
I installed postgres: sudo apt-get install postgresql and postgresql-contrib
I used sudo -U postgres psql to get into the postgres shell (I'm not sure if this is what I need to do)
show data_directory returns: /var/lib/postgresql/9.5/main
The data directory is located in Ubuntu ext4 formatted hard drive. I also have a 1 TB NTFS formatted hard disk mounted on /mnt/Data (which is mounted automatically on boot). What I tried:
Stop the postgres service: sudo systemctl stop postgresql
Create a new directory /mnt/Data/postgresdb and copy contents of the previous main to this which gives me a full path of /mnt/Data/postgresdb/postgresql/9.5/main using: sudo rsync -av /var/lib/postgresql/ /mnt/Data/postgresdb/postgresql/
Edit /etc/postgresql/9.5/main/postgresql.conf to change data_directory from the path mentioned above to /mnt/Data/postgresdb/postgresql/9.5/main
Start the postgres service: sudo systemctl start postgresl
Run sudo -U postgres psql but get the error that was mentioned in the original post.
These are the permissions on the respective main directories:
ls -l /var/lib/postgresql/9.5/
total 4.0K drwx------ 19 postgres postgres 4.0K Jan 16 12:40 main
ls -l /mnt/Data/postgresdb/postgresql/9.5/
total 4.0K drwxrwxrwx 1 root root 4.0K Jan 16 12:13 main
From the looks of it, the default directory is owned by "postgres" and the new directory is owned by root. However, when I try to change ownership to postgres: chown -R postgres main, it doesn't output any error, but the ownership doesn't change. I'm curious whether this is because this drive is NTFS formatted and is mounted.
Here is my /etc/fstab:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda5 during installation
UUID=3f5a9875-89a3-4ce5-b778-9d7aaf148ed6 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda6 during installation
UUID=85c3f4d4-e450-435b-8dd6-cf1b2cbd8fc2 none swap sw 0 0
/dev/disk/by-label/Data /mnt/Data auto nosuid,nodev,nofail,x-gvfs-show 0 0
Any ideas on how I can go about fixing this?
ORIGINAL POST
Recently, I installed Postgresql for storing some data for my research. The dataset came with instructions on how to setup the data on a Postgresql database (if interested, more info on that here and here). I installed Postgresql and set up a "role" and used the script that was provided for loading the database. It worked but I underestimated the size of the dataset and the script quit saying there was no more space.
I have two drives on my computer a 250G SSD drive with Windows and Ubuntu installed (125G each). And a 1TB HDD NTFS formatted where I store my data. So I thought moving the database to a folder on the other drive would be helpful. I purged all the data and the database to start afresh and followed the instructions here to move the database directory. However, after moving the directory, when I try to connect using psql I get the following error:
~ psql -U username -d postgres 14:48:33
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
How can I fix this? I am running 64-bit Ubuntu 16.04 with Postgresql-9.5. As mentioned earlier, I moved the DB directory a NTFS formatted filesystem (not sure if that cause any problems).
Thanks.
As mentioned in the comments the NTFS was the problem. I ended up resizing my bigger hard drive with 100GB formatted as ext4 and was able to launch postgres with the new data directory without any problems.
I'm trying to test a small PostgreSQL setup, so I cobbled together a quick local install. However, when I'm trying to create my personal db with createdb, it chokes on errors like this (notably, it starts with base/16384 the first time, and increments each time I run it). Anyone know what's going on here, or if there's some trivial config I missed that would cause this? Thanks, and this is somewhat time-critical, so please respond if you do know anything. Thanks!
UPDATES:
I'm running this on a CentOS 5 server, apologies that I don't have too many further details (it's a shared account on that server). uname -a has the following output:
Linux {OMITTED} 2.6.18-194.11.4.el5 #1 SMP Tue Sep 21 05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
I installed PostgreSQL from source from:
http://wwwmaster.postgresql.org/download/mirrors-ftp/source/v9.0.1/postgresql-9.0.1.tar.bz2
built in my home directory and installed to prefix=$HOME/local/pgsql.
Here's a terminal readout for me attempting to create my user's db on a fresh data setup:
[htung#{OMITTED}:~]$ killall postgres
LOG: autovacuum launcher shutting down
LOG: received smart shutdown request
LOG: shutting down
LOG: database system is shut down
[htung#{OMITTED}:~]$ rm -r tmp
mk[1]+ Done ../local/pgsql/bin/postgres -D $HOME/tmp (wd: ~/tmp)
(wd now: ~)
[htung#{OMITTED}:~]$ mkdir tmp
[htung#{OMITTED}:~]$ local/pgsql/bin/initdb -D $HOME/tmp
The files belonging to this database system will be owned by user "htung".
This user must also own the server process.
The database cluster will be initialized with locale en_US.UTF-8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".
fixing permissions on existing directory /afs/{OMITTED}/htung/tmp ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /afs/{OMITTED}/htung/tmp/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.
Success. You can now start the database server using:
local/pgsql/bin/postgres -D /afs/{OMITTED}/htung/tmp
or
local/pgsql/bin/pg_ctl -D /afs/{OMITTED}/htung/tmp -l logfile start
[htung#{OMITTED}:~]$ local/pgsql/bin/postgres -D $HOME/tmp
LOG: database system was shut down at 2010-11-15 13:47:25 PST
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
[1]+ Stopped local/pgsql/bin/postgres -D $HOME/tmp
[htung#{OMITTED}:~]$ bg
[1]+ local/pgsql/bin/postgres -D $HOME/tmp &
[htung#{OMITTED}:~]$ local/pgsql/bin/createdb
ERROR: could not fsync file "base/16384": Invalid argument
STATEMENT: CREATE DATABASE htung;
createdb: database creation failed: ERROR: could not fsync file "base/16384": Invalid argument
[htung#{OMITTED}:~]$
I would guess that you're possibly running into the SE linux system here. I'd recommend to either turn off SELinux and see if that works, or to install from RPMs available from the postgresql website.