React is running on Docker however isn't accessible to public IP - reactjs

Basically, I'm running a webapp with this stack:
Backend: FastAPI (python) which depends on database
Database: MySQL (being connected with via python-connector)
Frontend: React Functional (rn it's dependency is set to backend, but technically no dependencies)
The backend is connecting to the database via localhost, and is being served on the public ip I can access it at just fine at mydomain.com:8000/api
However when I'm trying to access mydomain.com, docker is setup to forward port 3000 to port 80, however it can't request react and, I get the ERR_CONNECTION_REFUSED error
In alot of similar issues people forget to forward port 3000 to 80 however i'm doing this...
Also, when I install npm & the react project on my Ubuntu Server, the frontend is accessible via mydomain.com:3000
In the docker-compose.yml when I change the ports from "REACT_PORT : 80 " To "3000 : 3000 " And that's accessible via our public ip.
Should I just port forward 3000 to 80 on the main linux server? how to do?
Docker Files
Frontend docker file
# pull official base image for node
FROM node:16-buster-slim
# set working directory
WORKDIR /app
# add `/app/node_modules/.bin` to PATH
ENV PATH /app/node_modules/.bin:$PATH
# install dependencies
COPY package.json ./
COPY package-lock.json ./
RUN npm i -g npm#latest
RUN npm install
# copy all src files to the container
COPY . .
# Expose port
EXPOSE 3000
EXPOSE 80
# start the web app
CMD ["npm", "run", "start"]
docker-compose.yml
version: "3.9"
services:
db:
image: mysql:${MYSQL_VERSION}
restart: always
environment:
- MYSQL_DATABASE=${MYSQL_DB}
- MYSQL_USER=${MYSQL_USERNAME}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
ports:
- "${MYSQL_PORT}:${MYSQL_PORT}"
expose:
- "${MYSQL_PORT}"
volumes:
- db:/var/lib/mysql
networks:
- mysql_network
backend:
container_name: fastapi-backend
build: ./backend/app
volumes:
- ./backend:/code
ports:
- "${FASTAPI_PORT}:${FASTAPI_PORT}"
env_file:
- .env
depends_on:
- db
networks:
- mysql_network
- backend
restart: always
frontend:
container_name: react-frontend
build: ./frontend/client
ports:
#This doesn't work for some reason?
- "${REACT_PORT}:80"
#When I do this, I can access react via public ip just fine..:
- "3000:3000"
depends_on:
- backend
networks:
- backend
restart: always
volumes:
db:
driver: local
networks:
backend:
driver: bridge
mysql_network:
driver: bridge
React/NPM Files
package.json
{
"name": "client",
"version": "0.1.0",
"private": true,
"dependencies": {
"#reduxjs/toolkit": "^1.8.2",
"#testing-library/jest-dom": "^5.16.4",
"#testing-library/react": "^13.1.1",
"#testing-library/user-event": "^13.5.0",
"bootstrap": "^5.1.3",
"react": "^18.0.0",
"react-dom": "^18.0.0",
"react-redux": "^8.0.2",
"react-router-dom": "^6.3.0",
"react-scripts": "5.0.1",
"universal-cookie": "^4.0.4",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
Ubuntu UFW status
root#localhost:~/director# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
3306/tcp ALLOW IN Anywhere
22/tcp ALLOW IN Anywhere
80/tcp ALLOW IN Anywhere
443 ALLOW IN Anywhere
3000 ALLOW IN Anywhere
3306/tcp (v6) ALLOW IN Anywhere (v6)
22/tcp (v6) ALLOW IN Anywhere (v6)
80/tcp (v6) ALLOW IN Anywhere (v6)
443 (v6) ALLOW IN Anywhere (v6)
3000 (v6) ALLOW IN Anywhere (v6)
doing curl -l mydomain.com:8000
curl -l domain.com:8000
{"detail":"Not authenticated"}
Our backend is working and public
docker inspect react_frontend
[
{
"Id": "533f96d538bcf28827d5ad5ead69dc97b97c79bfef1d1e31e3847f15ceb0621f",
"Created": "2022-06-27T04:00:18.459838372Z",
"Path": "docker-entrypoint.sh",
"Args": [
"npm",
"start"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 57578,
"ExitCode": 0,
"Error": "",
"StartedAt": "2022-06-27T04:00:20.253120387Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:698f3ce60c6ba12f39dcf371bd4be556c1cb4aac9c7f95fd4589c079ec4e337b",
"ResolvConfPath": "/var/lib/docker/containers/533f96d538bcf28827d5ad5ead69dc97b97c79bfef1d1e31e3847f15ceb0621f/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/533f96d538bcf28827d5ad5ead69dc97b97c79bfef1d1e31e3847f15ceb0621f/hostname",
"HostsPath": "/var/lib/docker/containers/533f96d538bcf28827d5ad5ead69dc97b97c79bfef1d1e31e3847f15ceb0621f/hosts",
"LogPath": "/var/lib/docker/containers/533f96d538bcf28827d5ad5ead69dc97b97c79bfef1d1e31e3847f15ceb0621f/533f96d538bcf28827d5ad5ead69dc97b97c79bfef1d1e31e3847f15ceb0621f-json.log",
"Name": "/react-frontend",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": [],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "scheduleplatform_backend",
"PortBindings": {
"80/tcp": [
{
"HostIp": "",
"HostPort": "3000"
}
]
},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": [],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/f9449860ca28f3641e74ef010fa6bb54383a33a975dce28519f5cfe609ff61d2-init/diff:/var/lib/docker/overlay2/7d8ad12b25cf6e4e0093e811370194cc6c1f72aa443d7d8f0ab69e97715f4c82/diff:/var/lib/docker/overlay2/b530ee218be07eb3a58df7a2f134b4e23cd22ad04b35dc7cbb75f3f07a206280/diff:/var/lib/docker/overlay2/94b5584c91af3dda4376092abdb9b7ff80937311741b96deca22e4b3aaf3a7c7/diff:/var/lib/docker/overlay2/4d8194094b484dfd94ca7473e3e13bfa5962879fd81942710f0aa096199a87c3/diff:/var/lib/docker/overlay2/aeb4c3cd54338efa1d44e1fc24c8a8f91bf672e47771fd0e308b3f65ab753f07/diff:/var/lib/docker/overlay2/9ddac11aa66b2d843b2391c3ad0798f8e372adfffeaec87882322de663b398e5/diff:/var/lib/docker/overlay2/e874192d15482d34ed8a80ef205b55005c84fcad0a7726822de258963191ec10/diff:/var/lib/docker/overlay2/8b162bdb3415e166a91518df47e1bed0bfff2d89568a2653f74e66cb3931773a/diff:/var/lib/docker/overlay2/a18946af0ee112612099819a9db0a3cbef5d1e2264904e6a8405868ab111634a/diff:/var/lib/docker/overlay2/b94b4e7d1af7280ad586d6f3366a161ec0d5cf9f858f0f20139b2ab0a72acfb7/diff:/var/lib/docker/overlay2/052c70a96ef76a7c32d609b98ec679ad7c6a74a28623b870bd729447bbc6086c/diff",
"MergedDir": "/var/lib/docker/overlay2/f9449860ca28f3641e74ef010fa6bb54383a33a975dce28519f5cfe609ff61d2/merged",
"UpperDir": "/var/lib/docker/overlay2/f9449860ca28f3641e74ef010fa6bb54383a33a975dce28519f5cfe609ff61d2/diff",
"WorkDir": "/var/lib/docker/overlay2/f9449860ca28f3641e74ef010fa6bb54383a33a975dce28519f5cfe609ff61d2/work"
},
"Name": "overlay2"
},
"Mounts": [],
"Config": {
"Hostname": "533f96d538bc",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"3000/tcp": {},
"80/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/app/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NODE_VERSION=16.15.1",
"YARN_VERSION=1.22.19"
],
"Cmd": [
"npm",
"start"
],
"Image": "scheduleplatform_frontend",
"Volumes": null,
"WorkingDir": "/app",
"Entrypoint": [
"docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {
"com.docker.compose.config-hash": "a02c886422905cf77d66a479bb9967d4a85bdd8c3ed0369665dae6afc9b34099",
"com.docker.compose.container-number": "1",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "scheduleplatform",
"com.docker.compose.project.config_files": "docker-compose.yml",
"com.docker.compose.project.working_dir": "/root/SchedulePlatform",
"com.docker.compose.service": "frontend",
"com.docker.compose.version": "1.29.2"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "014cba17fd5ce070c42b6257a05149d7c78b7556d941d5f790c8c0f27e70a8b1",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"3000/tcp": null,
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "3000"
},
{
"HostIp": "::",
"HostPort": "3000"
}
]
},
"SandboxKey": "/var/run/docker/netns/014cba17fd5c",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"scheduleplatform_backend": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"533f96d538bc",
"frontend"
],
"NetworkID": "9a6ff7cb0c0d725de46e21e309cd0e5708995faffb00f099c8980127f8a9c68c",
"EndpointID": "c6c22efd535ee4beaa732730143268a4d372d2cecb2537f00013a7b057416852",
"Gateway": "172.20.0.1",
"IPAddress": "172.20.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:14:00:03",
"DriverOpts": null
}
}
}
}
]

I gave up with docker and now use NGINX to point port 80 to 3000...
Basically, in my docker-compose.yml I point the react port to itself like:
frontend:
container_name: react-frontend
build: ./frontend/client
ports:
- "${REACT_PORT}:${REACT_PORT}"
depends_on:
- backend
networks:
- backend
restart: always
And with NGINX, I edit the config file doing:
nano etc/nginx/nginx.conf
then in the http {} section of this config, I removed include /etc/nginx/sites-enabled/*;
and added
server {
listen 80;
location / {proxy_pass http://localhost:3000/; }
}
and then reloaded nginx, now docker is forwarding at port 3000 and nginx is pointing 80 to that port.
This method works, I still don't understand why Docker wasn't able to port forward that, I even checked my netstat and nothing was running on port 80...
weird...

Related

Using Volttron aggregation agent

I'm trying to get the aggregate agent to work with timescale db.
https://volttron.readthedocs.io/en/main/developing-volttron/developing-agents/specifications/aggregate.html
I have a file aggregation.config
{
"connection": {
"type": "postgresql",
"params": {
"dbname": "volttrondb",
"host": "127.0.0.1",
"port": 5432,
"user": "user",
"password": "password",
"timescale_dialect": true
}
},
"aggregations":[
# list of aggregation groups each with unique aggregation_period and
# list of points that needs to be collected
{
"aggregation_period": "1h",
"use_calendar_time_periods": true,
"utc_collection_start_time":"2016-03-01T01:15:01.000000",
"points": [
{
"topic_names": ["campus/building/fake/EKG_Cos", "campus/building/fake/EKG_Sin"],
"aggregation_topic_name":"campus/building/fake/avg_of_EKG_Cos_EKG_Sin",
"aggregation_type": "avg",
"min_count": 2
}
]
}
]
}
And run the following command
vctl install services/core/SQLAggregateHistorian/ --tag aggregate-historian -c config/aggregation.config --start
It starts correctly - the vctl status shows it running and there are no errors in the log.
I do not see the point campus/building/fake/avg_of_EKG_Cos_EKG_Sin in the topics table.
Any suggestions?

Xdebug is not launched in WSL2 in a docker CakePHP 3 application

I am struggling with Xdebug + WSL2 + CakePHP 3 + VSCode. Checking the debug console it seems that I have running Xdebug correctly, but when I run a script in the browser, the Xdebug is not launched. This is the code:
Note: I forgot to mention that I am working on docker, that's why the "0.0.0.0" in the hostname parameter.
This is the xdebug.ini
zend_extension=xdebug
[xdebug]
zend_extension=xdebug
xdebug.mode=develop,debug
xdebug.client_host='host.docker.internal'
xdebug.start_with_request=yes
xdebug.client_port = 9003
xdebug.start_with_request=yes
xdebug.log=/var/log/xdebug/xdebug.log
xdebug.connect_timeout_ms=2000
launch.json
{
"version": "0.2.0",
"configurations": [
{
"name": "Listen for Xdebug",
"type": "php",
"request": "launch",
"port": 9003,
"hostname": "0.0.0.0",
"pathMappings": {
"/webroot": "${workspaceRoot}"
},
"log": true
},
{
"name": "Launch currently open script",
"type": "php",
"request": "launch",
"program": "${file}",
"cwd": "${fileDirname}",
"port": 0,
"runtimeArgs": [
"-dxdebug.start_with_request=yes"
],
"env": {
"XDEBUG_MODE": "debug,develop",
"XDEBUG_CONFIG": "client_port=${port}"
}
},
{
"name": "Launch Built-in web server",
"type": "php",
"request": "launch",
"runtimeArgs": [
"-dxdebug.mode=debug",
"-dxdebug.start_with_request=yes",
"-S",
"localhost:0"
],
"program": "",
"cwd": "${workspaceRoot}",
"port": 9003,
"serverReadyAction": {
"pattern": "Development Server \\(http://localhost:([0-9]+)\\) started",
"uriFormat": "http://localhost:%s",
"action": "openExternally"
}
}
]
}
This is the debug console:
Listening on { address: '0.0.0.0', family: 'IPv4', port: 9003 }
<- launchResponse
Response {
seq: 0,
type: 'response',
request_seq: 2,
command: 'launch',
success: true
}
<- initializedEvent
InitializedEvent { seq: 0, type: 'event', event: 'initialized' }
-> setBreakpointsRequest
{
command: 'setBreakpoints',
arguments: {
source: {
name: 'index.php',
path: '/root/server/webroot/index.php'
},
lines: [ 40 ],
breakpoints: [ { line: 40 } ],
sourceModified: false
},
type: 'request',
seq: 3
}
<- setBreakpointsResponse
Response {
seq: 0,
type: 'response',
request_seq: 3,
command: 'setBreakpoints',
success: true,
body: {
breakpoints: [
{
verified: true,
line: 40,
source: {
name: 'index.php',
path: '/root/server/webroot/index.php'
},
id: 1
}
]
}
}
The xdebug.log file
[20] Log opened at 2022-05-16 04:42:03.776649
[20] [Step Debug] INFO: Connecting to configured address/port: host.docker.internal:9003.
[20] [Step Debug] INFO: Connected to debugging client: host.docker.internal:9003 (through xdebug.client_host/xdebug.client_port). :-)
[20] [Step Debug] -> <init xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" fileuri="file:///var/www/html/webroot/info.php" language="PHP" xdebug:language_version="7.4.19" protocol_version="1.0" appid="20"><engine version="3.1.2"><![CDATA[Xdebug]]></engine><author><![CDATA[Derick Rethans]]></author><url><![CDATA[https://xdebug.org]]></url><copyright><![CDATA[Copyright (c) 2002-2021 by Derick Rethans]]></copyright></init>
[20] [Step Debug] <- feature_set -i 1 -n resolved_breakpoints -v 1
[20] [Step Debug] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="feature_set" transaction_id="1" feature="resolved_breakpoints" success="1"></response>
[20] [Step Debug] <- run -i 12
[20] [Step Debug] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="run" transaction_id="12" status="stopping" reason="ok"></response>
[20] [Step Debug] <- stop -i 13
[20] [Step Debug] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="stop" transaction_id="13" status="stopped" reason="ok"></response>
[20] Log closed at 2022-05-16 04:42:03.812679
UPDATE: Following this suggestion (I got from this link, by HolyGonzo) https://www.reddit.com/r/PHPhelp/comments/rqiw4h/need_help_troubleshooting_xdebug_configuration/ I added xdebug_break(); to my code, and then the debugger started working. It is pretty clear to understand that the issue it is in the VSCode configuration not in Xdebug.
SOLUTION:
After fight with this thing a few days, finally I found the issue:
In the launch.json in VSCode I updated this line and it works!!! (note that my path was wrong :( I had this "/var/www/webroot" instead of "/var/www/html/webroot").
"pathMappings": {
"/var/www/html/webroot": "${workspaceFolder}/webroot"
},
Update:
In order to allows Xdebug to look into the vendors folder, and the other folders outside /webroot, the code needs to be upated as follows (in my case, regarding to my server paths):
{
"name": "Listen for Xdebug",
"type": "php",
"request": "launch",
"port": 5902,
"hostname": "localhost",
"pathMappings": {
"/var/www/html": "${workspaceFolder}"
},
"log": true
}

Field: android - should NOT have additional property 'enableDangerousExperimentalLeanBuilds'

My expo version is 42.0.0. When I run "expo build:android -t app-bundle", I get this error:
Error: Problem validating fields in app.json. Learn more: https://docs.expo.io/workflow/configuration/
• Field: android - should NOT have additional property 'enableDangerousExperimentalLeanBuilds'.
Couldn't publish because errors were found. (See logs above.) Please fix the errors and try again.
{
"expo": {
"name": "XXXX",
"slug": "XXXX",
"version": "1.2.0",
"orientation": "portrait",
"icon": "./assets/XXXX.png",
"splash": {
"image": "./assets/XXXX.png",
"resizeMode": "contain",
"backgroundColor": "#FBDB5B"
},
"privacy": "public",
"platforms": [
"android"
],
"android": {
"config": {
"googleMobileAdsAppId": "XXXX"
},
"package": "com.XXXX",
"versionCode": 16,
"permissions": [],
"enableDangerousExperimentalLeanBuilds": true
},
"updates": {
"fallbackToCacheTimeout": 0
},
"assetBundlePatterns": ["**/*"]
}
}
The thing is, the property enableDangerousExperimentalLeanBuilds was removed.. https://docs.expo.dev/versions/latest/config/app/#android. Expo wants you to use EAS Build for smaller bundle sizes. You might have to drop the field, otherwise your app might work in development, but then crash once you put it in PlayStore.
Personally, I also had to eject as I wasn't satisfied with the sizes EAS gave me.

How to set database config variables for aws ecs in circle ci config.yml?

How are you.
I am deploying go backend api to ecs using docker.
And I am using circle ci for it.
I need to set database config environment variables to run backend api, but I don't know how to set that info in circle ci.
I am initializing aws resource using terraform, do I need to set db config environment variables in terraform? or can I set it on circle ci config.yml?
Thanks
You can define environment variables in the task definition so this will be available for your docker container in ECS.
resource "aws_ecs_task_definition" "backend-app" {
family = "backend"
container_definitions = <<EOF
[
{
"portMappings": [
{
"hostPort": 80,
"protocol": "tcp",
"containerPort": 3000
}
],
"environment":
[
{
"name": "NODE_ENV",
"value":"production"
},
{
"name": "DB_HOST",
"value": "HOST_ADDRESS"
},
{
"name": "DB_PASS",
"value": "DB_PASSWORD"
}
],
"cpu": 1000,
"memory": 1000,
"image": "***.dkr.ecr.us-west-2.amazonaws.com/backend:latest",
"name": "backend",
}
]
EOF
}

Ansible: Iterate and include multiple variable files

I have hundreds of files (generated through some application) which I am trying to iterate and include them as variable files.
See below files as example. There are many more variables in each of the files, I have toned down to make the example clear.
# cat /tmp/abc/dev1.yml
---
app_name: dev1
instance: dev
port: 1234
server: test1
#
# cat /tmp/abc/dev2.yml
---
app_name: dev2
instance: dev
port: 4567
server: test2
#
# cat /tmp/abc/dev3.yml
---
app_name: dev3
instance: dev
port: 2223
server: test3
#
Now, when I use these tasks in my playbook, I can see the variables (app_name, instance, port, etc) defined in the files (/tmp/abc/*.yml) in the output as ansible_facts.
- action: shell echo "{{ item }}"
with_fileglob: /tmp/abc/*
register: res
- include_vars: "{{ item.item }}"
with_items: res.results
when: item.changed == True
register: task1
This is my output, when I run the playbook.
root#vikas027:~# ansible-playbook -v configs.yml
PLAY [dev] **************************************************************
GATHERING FACTS ***************************************************************
ok: [vikas027.test.com]
TASK: [shell echo "{{ item }}"] ***********************************************
changed: [vikas027.test.com] => (item=/tmp/abc/dev3.yml) => {"changed": true, "cmd": "echo \"/tmp/abc/dev3.yml\"", "delta": "0:00:00.004915", "end": "2015-08-31 20:11:57.702623", "item": "/tmp/abc/dev3.yml", "rc": 0, "start": "2015-08-31 20:11:57.697708", "stderr": "", "stdout": "/tmp/abc/dev3.yml", "warnings": []}
changed: [vikas027.test.com] => (item=/tmp/abc/dev2.yml) => {"changed": true, "cmd": "echo \"/tmp/abc/dev2.yml\"", "delta": "0:00:00.004945", "end": "2015-08-31 20:11:58.130295", "item": "/tmp/abc/dev2.yml", "rc": 0, "start": "2015-08-31 20:11:58.125350", "stderr": "", "stdout": "/tmp/abc/dev2.yml", "warnings": []}
changed: [vikas027.test.com] => (item=/tmp/abc/dev1.yml) => {"changed": true, "cmd": "echo \"/tmp/abc/dev1.yml\"", "delta": "0:00:00.004864", "end": "2015-08-31 20:11:58.440205", "item": "/tmp/abc/dev1.yml", "rc": 0, "start": "2015-08-31 20:11:58.435341", "stderr": "", "stdout": "/tmp/abc/dev1.yml", "warnings": []}
TASK: [include_vars {{ item.item }}] ******************************************
ok: [vikas027.test.com] => (item={u'cmd': u'echo "/tmp/abc/dev3.yml"', u'end': u'2015-08-31 20:11:57.702623', u'stderr': u'', u'stdout': u'/tmp/abc/dev3.yml', u'changed': True, u'rc': 0, 'item': '/tmp/abc/dev3.yml', u'warnings': [], u'delta': u'0:00:00.004915', 'invocation': {'module_name': u'shell', 'module_args': u'echo "/tmp/abc/dev3.yml"'}, 'stdout_lines': [u'/tmp/abc/dev3.yml'], u'start': u'2015-08-31 20:11:57.697708'}) => {"ansible_facts": {"app_name": "dev3", "instance": "dev", "port": 2223, "server": "test3"}, "item": {"changed": true, "cmd": "echo \"/tmp/abc/dev3.yml\"", "delta": "0:00:00.004915", "end": "2015-08-31 20:11:57.702623", "invocation": {"module_args": "echo \"/tmp/abc/dev3.yml\"", "module_name": "shell"}, "item": "/tmp/abc/dev3.yml", "rc": 0, "start": "2015-08-31 20:11:57.697708", "stderr": "", "stdout": "/tmp/abc/dev3.yml", "stdout_lines": ["/tmp/abc/dev3.yml"], "warnings": []}}
ok: [vikas027.test.com] => (item={u'cmd': u'echo "/tmp/abc/dev2.yml"', u'end': u'2015-08-31 20:11:58.130295', u'stderr': u'', u'stdout': u'/tmp/abc/dev2.yml', u'changed': True, u'rc': 0, 'item': '/tmp/abc/dev2.yml', u'warnings': [], u'delta': u'0:00:00.004945', 'invocation': {'module_name': u'shell', 'module_args': u'echo "/tmp/abc/dev2.yml"'}, 'stdout_lines': [u'/tmp/abc/dev2.yml'], u'start': u'2015-08-31 20:11:58.125350'}) => {"ansible_facts": {"app_name": "dev2", "instance": "dev", "port": 4567, "server": "test2"}, "item": {"changed": true, "cmd": "echo \"/tmp/abc/dev2.yml\"", "delta": "0:00:00.004945", "end": "2015-08-31 20:11:58.130295", "invocation": {"module_args": "echo \"/tmp/abc/dev2.yml\"", "module_name": "shell"}, "item": "/tmp/abc/dev2.yml", "rc": 0, "start": "2015-08-31 20:11:58.125350", "stderr": "", "stdout": "/tmp/abc/dev2.yml", "stdout_lines": ["/tmp/abc/dev2.yml"], "warnings": []}}
ok: [vikas027.test.com] => (item={u'cmd': u'echo "/tmp/abc/dev1.yml"', u'end': u'2015-08-31 20:11:58.440205', u'stderr': u'', u'stdout': u'/tmp/abc/dev1.yml', u'changed': True, u'rc': 0, 'item': '/tmp/abc/dev1.yml', u'warnings': [], u'delta': u'0:00:00.004864', 'invocation': {'module_name': u'shell', 'module_args': u'echo "/tmp/abc/dev1.yml"'}, 'stdout_lines': [u'/tmp/abc/dev1.yml'], u'start': u'2015-08-31 20:11:58.435341'}) => {"ansible_facts": {"app_name": "dev1", "instance": "dev", "port": 1234, "server": "test1"}, "item": {"changed": true, "cmd": "echo \"/tmp/abc/dev1.yml\"", "delta": "0:00:00.004864", "end": "2015-08-31 20:11:58.440205", "invocation": {"module_args": "echo \"/tmp/abc/dev1.yml\"", "module_name": "shell"}, "item": "/tmp/abc/dev1.yml", "rc": 0, "start": "2015-08-31 20:11:58.435341", "stderr": "", "stdout": "/tmp/abc/dev1.yml", "stdout_lines": ["/tmp/abc/dev1.yml"], "warnings": []}}
PLAY RECAP ********************************************************************
vikas027.test.com : ok=3 changed=1 unreachable=0 failed=0
root#vikas027:~#
How can I reference variables like app_name, instance, port, etc in other tasks ? I tried using below code and few other combinations in vain.
- debug: msg="{{ task1.app_name }}"
with_items: task1.results
Your variable files, dev1.yml, dev2.yml, etc. all reference the same variable names. Is this on purpose, or just part of your example? I ask because your example as it's currently shown, would result in just the last set of variables being defined, so as far as ansible is concerned it appears that the variables would ultimately just be defined as if you did this:
vars:
app_name: dev3
instance: dev
port: 2223
server: test3
You would just reference the variables by their given names:
- debug: var=app_name
- debug: var=instance
etc.
What I'm guessing you actually want to be doing is having those variable files look something like this:
---
app:
dev1:
instance: "dev"
port: "1234"
server: "host1"
and
---
app:
dev2:
instance: "dev"
port: "4321"
server: "host2"
You would then reference your objects something like this:
# should list "dev1", "dev2", "dev3"...
- debug: msg={{ item }}
with_dict: app
# should list the server names for each device
- debug: var = app[item][server]
with_dict: app
I was working on this whole day today, tried umpteen configuration changes in vain. Finally, it is working the way I wanted it to work.
This is what one needs to do if in a similar situation. Hope this helps someone.
First, register your facts locally. I chose the default /etc/ansible/facts.d/ directory for the same. Here are more details.
Key things to remember:-
Extension should be .fact
File should be executable (I gave 0755)
Format is JSON (I've used yaml-to-json to convert my yaml files to json. You can use ruby or perl one-liners too.)
Then, to iterate the facts registered in the previous step, we need to load/reload the facts in the playbook in order to use them in tasks/playbooks.
- local_action: setup filter=ansible_local
- template: src=nginx_lb.conf.j2 dest=/etc/nginx/conf.d/{{ item.key }}.conf
with_dict: "{{ ansible_local }}"
All variables can now be used in the jinja2 template. For example, port can be referenced as item.value.port.

Resources