How to make Postgres containers shared data with amazon EFS? - database

Question

I have a cluster which has 2 instances.Both instance has Postgres container and its volume link to Elastic file system accesspoint as the Volume.
I map the volume of both instance to /var/lib/postgresql/data, but container didn't share its data.
Here is my configuration
docker-compose.yml
version: "3.0"
services:
front:
image: 540744822643.dkr.ecr.ap-southeast-1.amazonaws.com/front:latest
links:
- app:app.plasgate.com
networks:
- app
container_name: front
environment:
- NODE_OPTIONS=--max-old-space-size=2048
ports:
- "8080:80"
logging:
driver: awslogs
options:
awslogs-group: sms-gateway
awslogs-region: ap-southeast-1
awslogs-stream-prefix: "front"
app:
image: 540744822643.dkr.ecr.ap-southeast-1.amazonaws.com/plasgate:latest
links:
- jasmin:jasmin
- db:db
networks:
- app
container_name: app
environment:
- PYTHONUNBUFFERED=1
- PYTHONIOENCODING=UTF-8
restart: on-failure:10
ports:
- "5000:5000"
logging:
driver: awslogs
options:
awslogs-group: sms-gateway
awslogs-region: ap-southeast-1
awslogs-stream-prefix: "app"
nginx:
image: 540744822643.dkr.ecr.ap-southeast-1.amazonaws.com/nginx:latest
links:
- app:app
- front:front
container_name: nginx
networks:
- app
environment:
API_HOST: "service.wpdevelop.xyz"
API_PORT: 5000
FRONT_HOST: "customer.wpdevelop.xyz"
FRONT_PORT: 8080
ports:
- "80:80"
- "443:443"
logging:
driver: awslogs
options:
awslogs-group: sms-gateway
awslogs-region: ap-southeast-1
awslogs-stream-prefix: "nginx"
db:
image: 540744822643.dkr.ecr.ap-southeast-1.amazonaws.com/postgres:latest
volumes:
- postgres:/var/lib/postgresql/data:rw
restart: on-failure:10
networks:
- app
environment:
POSTGRES_PASSWORD: "xxx#2020"
POSTGRES_USER: webadmin
POSTGRES_DB: smsgwdev
ports:
- "5432:5432"
logging:
driver: awslogs
options:
awslogs-group: sms-gateway
awslogs-region: ap-southeast-1
awslogs-stream-prefix: "db"
redis:
image: 540744822643.dkr.ecr.ap-southeast-1.amazonaws.com/radis:latest
container_name: redis
restart: on-failure:10
networks:
- app
ports:
- "6379:6379"
logging:
driver: awslogs
options:
awslogs-group: sms-gateway
awslogs-region: ap-southeast-1
awslogs-stream-prefix: "redis"
volumes:
postgres:
networks:
app:
driver: bridge
ecs-params.yml
version: 1
task_definition:
family: sms-gateway
ecs_network_mode: bridge
services:
front:
essential: true
cpu_shares: 100
mem_limit: 2147483648
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost || exit 1"]
interval: 5s
timeout: 10s
retries: 3
start_period: 30s
app:
essential: false
cpu_shares: 100
mem_limit: 2147483648
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:5000 || exit 1"]
interval: 5s
timeout: 10s
retries: 3
start_period: 30s
depends_on:
- container_name: db
condition: HEALTHY
nginx:
essential: false
cpu_shares: 100
mem_limit: 2147483648
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost || exit 1"]
interval: 5s
timeout: 10s
retries: 3
start_period: 30s
db:
essential: false
cpu_shares: 100
mem_limit: 2147483648
healthcheck:
test: pg_isready -U webadmin -d smsgwdev
interval: 5s
timeout: 10s
retries: 2
start_period: 30s
redis:
essential: false
cpu_shares: 100
mem_limit: 2147483648
healthcheck:
test: ["CMD-SHELL", "redis-cli", "ping"]
interval: 5s
timeout: 10s
retries: 2
start_period: 30s
efs_volumes:
- name: postgres
filesystem_id: fs-a4aa73e4
transit_encryption: ENABLED
access_point: fsap-007405b3e9bc7bc2f
How can I make the two Postgres container use the same pgdata?

Answer 1

First and foremost, running Postgres on EFS is not a great idea. I think it’s fine if you need something quick and for very low loads in test environments but EFS is not the right backend for a database engine. Second, sharing an EFS share between 2 containers is an even worse idea. This setup is a no-no because each database will get simultaneus non-arbitrated access to the same data files and this is not how Postgres is supposed to work.
Second, you don't call it out explicitly but are you using the ecs-cli to get this deployed? If so, my suggestion would be to look at an alternative mechanism we (AWS) have introduced together with Docker which relies on the new Docker Compose capabilities to deploy to the Cloud (e.g. ECS). The new version of the ecs-cli is called Copilot and it moved away from Docker support. Note the new Docker Compose integration does not need a separate ecs-params file for now (albeit there are discussions to introduce one) and relies on x-aws- extensions in the docker compose file itself.
Third, regardless of whether this is a good idea or not (it’s not!) on the heels of and inspired by this example, the following simple compose allows you to deploy 2 x Postgres containers that share the same data directory:
version: '3.4'
services:
db1:
container_name: db1
image: postgres:latest
environment:
- POSTGRES_USER=me
- POSTGRES_PASSWORD=mypassword
volumes:
- my-vol:/var/lib/postgresql/data
db2:
depends_on:
- db1
container_name: db2
image: postgres:latest
environment:
- POSTGRES_USER=me
- POSTGRES_PASSWORD=mypassword
volumes:
- my-vol:/var/lib/postgresql/data
app:
container_name: app
image: nginx
volumes:
my-vol:
If you docker compose up in an ECS context (see the blog for more details) you will get 3 ECS services (1 x app/nginx and 2 x DB services) with the 2 DB services insisting on the same EFS Access Point. Again this is just an academic example to prove a working docker compose file. I DO NOT SUGGEST to use this in any meaningful deployment.
[UPDATE]: I have just noticed you only have 1 postgres in the compose above. So I assume you have two separate compose files with one postgres DB insisting against the same file system. All I said above still apply but note there is a limitation that will prevent you to even technically deploy this scenario.