How to read CSV file data in ansible-playbook using with_lines? - loops

I have situation where in one csv file i have 2 columns like below
cat report.csv
Field1,Field2,Field3
name3,3,5
name4,5,6
now i want to use the lines which are in bold.
Each column will be an input to one of the ansible role.
it should go like
roles:
- { role: arti_master, mod_name: "{{ item.name}}" , version: "{{ item.version}}"
with_lines:
- "cat report.csv|cut -d, -f2"

Here is an example:
- name: "Sending email"
hosts: localhost
gather_facts: no
tasks:
- name: "Reading user information"
read_csv:
path: users.csv
register: users
- name: "Sending an e-mail using Gmail SMTP servers"
mail:
host: smtp.gmail.com
port: 587
username: <email>
password: <pass>
to: "{{ user.email }}"
subject: Email Subjet
body: |
Hi {{ user.first_name }},
How are you?
loop: "{{ users.list }}"
loop_control:
loop_var: user
And the CSV:
first_name,last_name,email
Joel,Zamboni,joel.zamboni#example.com
The read_csv returns the data in two formats, as a dict or as a list and in this case I am 'looping' over the list to send emails.
Best,
Joel

I believe you have two (and a half) ways that I can think of:
Do as you said and run the file through cut or python -c "import csv;..." or other "manual" processing, then capture the output in a variable
Anything that looks like JSON when fed into a vars: or set_fact: will become a list or dict, so you'd just want the text to go into a tool looking like CSV and come out of the tool looking like JSON
Use the lookup("csvfile") to actually read the file using an "approved" mechanism
(this is the "half" part:) if the csv is on the remote machine, then use fetch: to pull it to your controlling machine, then run lookup("csvfile") on it

read_csv module was recently added to ansible, and is now available from ansible 2.8. After upgrading ansible, you can read line by line as follows:
- name: read the csv file
read_csv:
path: "{{ report.csv }}"
delimiter: ','
register: report_csv
You can then access it as list by using report_csv.list and it'll hold values as a list of dictionaries:
[{'Field1': 'name3', 'Field2': 3, 'Field3': 5}, {'Field1': 'name4', 'Field2': 5, 'Field3': 6}]

Sharing my ansible code as well this is what worked for me https://stackoverflow.com/a/56910479/1679541
playbook.yaml
---
- name: Read Users
gather_facts: True
hosts: localhost
tasks:
- read_csv:
path: users.csv
register: userlist
- debug:
msg: "{{ user.username }} and password is {{ user.password }}"
loop: "{{ userlist.list }}"
loop_control:
loop_var: user
users.csv
username,password
user0,test123
user1,test123
Ansible Output
PLAY [Read Users] *************************************************
TASK [Gathering Facts] *********************************************************
ok: [127.0.0.1]
TASK [read_csv] ****************************************************************
ok: [127.0.0.1]
TASK [debug] *******************************************************************
ok: [127.0.0.1] => (item={u'username': u'user0', u'password': u'test123'}) => {
"msg": "user0 and password is test123"
}
ok: [127.0.0.1] => (item={u'username': u'user1', u'password': u'test123'}) => {
"msg": "user1 and password is test123"
}
PLAY RECAP *********************************************************************
127.0.0.1 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Related

Using pattern search in when condition in ansible

Here is the playbook. Expected outcome is when there there is a string(unx) match found in systemename variable, should touch a file /var/tmp/testfile
hosts: all
gather_facts: yes
vars:
systemname: "{{ ansible_hostname }}"
tasks:
name: Capture hostname
shell: hostname -s
register: host_name
name: Display Variable
debug:
msg: "{{ systemname }}"
name: Create test file when it matches a server string
file:
path: /var/tmp/testfile
state: touch
when: "{{ systemname | lower | search('unx') }}"
Failing with below error message
PLAY [all] ***********************************************************************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************************************************
ok: [sr1punxpe01]
TASK [Capture hostname] **********************************************************************************************************************************
changed: [sr1punxpe01]
TASK [Display Variable] **********************************************************************************************************************************
ok: [sr1punxpe01] => {
"msg": "sr1punxpe01"
}
TASK [Create test file when it matches a server string] **************************************************************************************************
[WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ systemname | lower | search('unx') }}
fatal: [sr1punxpe01]: FAILED! => {"msg": "The conditional check '{{ systemname | lower | search('unx') }}' failed. The error was: template error while templating string: no filter named 'search'. String: {{ systemname | lower | search('unx') }}\n\nThe error appears to be in '/home_ldap/smandal/ansible/others/test.yml': line 16, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Create test file when it matches a server string\n ^ here\n"}
PLAY RECAP ***********************************************************************************************************************************************
sr1punxpe01 : ok=3 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
What am i doing wrong here? Your suggestion would be highly appreciated.
Here is the playbook. Expected outcome is when there there is a string(unx) match found in systemename variable, should touch a file /var/tmp/testfile
---
- hosts: all
gather_facts: yes
vars:
systemname: "{{ ansible_hostname }}"
tasks:
- name: Capture hostname
shell: hostname -s
register: host_name
- name: Display Variable
debug:
msg: "{{ systemname }}"
- name: Create test file when it matches a server string
file:
path: /var/tmp/testfile
state: touch
when: "{{ systemname | lower | search('unx') }}"
Failing with below error message
PLAY [all] ***********************************************************************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************************************************
ok: [sr1punxpe01]
TASK [Capture hostname] **********************************************************************************************************************************
changed: [sr1punxpe01]
TASK [Display Variable] **********************************************************************************************************************************
ok: [sr1punxpe01] => {
"msg": "sr1punxpe01"
}
TASK [Create test file when it matches a server string] **************************************************************************************************
[WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ systemname | lower | search('unx') }}
fatal: [sr1punxpe01]: FAILED! => {"msg": "The conditional check '{{ systemname | lower | search('unx') }}' failed. The error was: template error while templating string: no filter named 'search'. String: {{ systemname | lower | search('unx') }}\n\nThe error appears to be in '/home_ldap/smandal/ansible/others/test.yml': line 16, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Create test file when it matches a server string\n ^ here\n"}
PLAY RECAP ***********************************************************************************************************************************************
sr1punxpe01 : ok=3 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
What am i doing wrong here? Your suggestion would be highly appreciated.

creating dictionary with ansible

I want to create a dictionary in ansible from a list; using some variables for the value in the key-value pair of the dictionary, but it seems to be not working.
I've simplified just to the problem and created a sample playbook to reproduce the issue, can someone help me out. Thanks!
here is my playbook
---
- name: create dictionary test
hosts: all
connection: local
gather_facts: False
vars:
ports: [80, 443]
server_base: "org.com"
tasks:
- name: print the ports
debug:
msg: "ports: {{ports}}"
- name: create a dictionary
set_fact:
#server_rules: "{{server_rules|default([]) + [{'server': '{{server_base}}-{{item}}', 'port': item}]}}"
server_rules: "{{server_rules|default([]) + [{'server': '{{server_base}}', 'port': item}]}}"
loop: "{{ports|flatten(1)}}"
- name: output
debug:
msg: "server_rules: {{server_rules}}"
With the above it works, the output as below:
$ansible-playbook -i "localhost," dicttest.yaml
PLAY [create dictionary test] ***************************************************************************************************************************************
TASK [print the ports] **********************************************************************************************************************************************
ok: [localhost] => {
"msg": "ports: [80, 443]"
}
TASK [create a dictionary] ******************************************************************************************************************************************
ok: [localhost] => (item=80)
ok: [localhost] => (item=443)
TASK [output] *******************************************************************************************************************************************************
ok: [localhost] => {
"msg": "server_rules: [{'server': 'org.com', 'port': 80}, {'server': 'org.com', 'port': 443}]"
}
PLAY RECAP **********************************************************************************************************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
But when I change set fact to (uncomment the one commented line and comment the other one)
server_rules: "{{server_rules|default([]) + [{'server': '{{server_base}}-{{item}}', 'port': item}]}}"
it fails with the following error
$ansible-playbook -i "localhost," dicttest.yaml
PLAY [create dictionary test] ***************************************************************************************************************************************
TASK [print the ports] **********************************************************************************************************************************************
ok: [localhost] => {
"msg": "ports: [80, 443]"
}
TASK [create a dictionary] ******************************************************************************************************************************************
ok: [localhost] => (item=80)
ok: [localhost] => (item=443)
TASK [output] *******************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'item' is undefined\n\nThe error appears to be in '/Users/dev/dicttest.yaml': line 22, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: output\n ^ here\n"}
PLAY RECAP **********************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Can someone explain how to get this working.
Q: Given the data
ports: [80, 443]
server_base: org.com
create a list of dictionaries
server_rules:
- port: 80
server: org.com-80
- port: 443
server: org.com-443
A: The task below gives the expected result
- name: create a list of dictionaries
set_fact:
server_rules: "{{ server_rules|default([]) +
[{'server': server_base + '-' + item|string,
'port': item}] }}"
loop: "{{ ports }}"
It's not necessary to iterate the list in a task. The declaration of the variables below does the same job
servers: "{{ [server_base]|
product(ports)|
map('join', '-')|
map('community.general.dict_kv', 'server')|
list }}"
server_rules: "{{ servers|
zip(ports|map('community.general.dict_kv', 'port'))|
map('combine')|
list }}"
Notes:
Double braces "{{ }}" can't be nested. The expression below is wrong
"{{ var1 + ['{{server_base}}-'] }}"
Correct
"{{ var1 + [server_base + '-'] }}"
In YAML, the operator plus "+" is used both to concatenate strings and lists. This is because a string in YAML is technically a list of characters. It's recommended to use "~" to concatenate strings
Also correct
"{{ var1 + [server_base ~ '-'] }}"
Use var attribute in debug. The output is more readable with stdout_callback = yaml
- debug:
var: server_rules
The created variable server_rules is a list. The items are dictionaries. Hence, it's a list of dictionaries.
The variable ports is a simple list. There is no need to use the filter flatten.
The combination of "hosts: all" and "connection: local" would make to run all hosts at the localhost
hosts: all
connection: local
Use "hosts: localhost" if you want to run the playbook at the localhost. In this case "connection: local" is the default
hosts: localhost
If you want to run a task at the localhost, but still want the play to read the variables for all hosts, use "delegate_to: localhost" and limit the task to "run_once: true". For example
- hosts: all
tasks:
- copy:
content: "{{ ansible_play_hosts|to_nice_yaml }}"
dest: /tmp/ansible_play_hosts.yml
delegate_to: localhost
run_once: true
got an answer from another person outside of stackoverflow, posting here just to make sure anyone looking at this, gets the solution as well
changing as follows fixes the problem
server_rules: "{{server_rules|default([]) + [{'server': '{{server_base}}-' + item|string, 'port': item}]}}"
The complete working playbook
---
- name: create dictionary test
hosts: all
connection: local
gather_facts: False
vars:
ports: [80, 443]
server_base: "org.com"
tasks:
- name: print the ports
debug:
msg: "ports: {{ports}}"
- name: create a dictionary
set_fact:
server_rules: "{{server_rules|default([]) + [{'server': '{{server_base}}-' + item|string, 'port': item}]}}"
loop: "{{ports|flatten(1)}}"
- name: output
debug:
msg: "server_rules: {{server_rules}}"

Ansible: filter a merged yaml file

My current setup is as follows:
ansible.cfg
[defaults]
retry_files_enabled = false
hash_behaviour = merge
hosts
[sample]
sampleserver
group_vars
content of group_vars/1.yml:
tenants:
1:
name: name1
ip: ip1
content of group_vars/2.yml:
tenants:
2:
name: name2
ip: ip2
content of group_vars/3.yml:
tenants:
3:
name: name3
ip: ip3
playbook.yml
- hosts: sample
tasks:
- set_fact:
mytenants: "{{ {customtenants: tenants[customtenants]} }}"
when: customtenants is defined
- name: Output tenantname
debug:
msg: "mytenant: {{ mytenant }} name: {{ mytenant.name }}"
with_dict: "{{ mytenants|default(tenants) }}"
loop_control:
loop_var: mytenant
command
run for all tenants (works fine)
ansible-playbook playbook.yml -i inventory/hosts
run only for one defined tenant (works fine)
ansible-playbook tenants.yml -i inventory/hosts -e "customtenants=1"
run for multiple defined tenants
ansible-playbook playbook.yml -i inventory/hosts -e '{"customtenants": ["1", "3"]}'
Can anyone help me, so that it works for multiple tenants (last command example)?
Or is there a better way to achieve such deynamic solutions?
What I have done in the past is put that variable in the hosts line, e.g.:
- hosts: "{{ customtenants | default( 'all' ) }}"
Then you can delete the when and with_dict lines.

Filtering Array of Vars in Ansible

Currently we have a huge file that contains all of our nginx configs for each site we work on. The file has about 150 lines or so of sites like this:
- { nginx_tempalte: 'site.conf.tpl', domain: 'example.com', server: 'ServerA', enabled: true, conf_name: 'example_site' }
Our playbook loops through each var 2 times. Once for getting it into sites-enabled and another for the symlink. This takes about 5 minutes each loop which isn't ideal.
I tried setting up a nested loop that takes in a registered variabled that has all the config names from the sites-available and checks them against the given var from earlier. However this seems like more of the same approach.
I would love some help filtering down these files.
It depends on how are you getting the dict. If it is a variable, you can have:
---
- name: Test
hosts: localhost
gather_facts: False
# with predefined vars
vars:
nginx: { nginx_tempalte: 'site.conf.tpl', domain: 'example.com', server: 'ServerA', enabled: true, conf_name: 'example_site' }
tasks:
- name: Fact
set_fact:
domain: "{{ nginx['domain'] }}"
server: "{{ nginx['server'] }}"
- name: Print Domain
debug:
var: domain
- name: Print Server
debug:
var: server
And you will have both values at the same time:
PLAY [Test] ********************************************************************************************************************
TASK [Fact] ********************************************************************************************************************
ok: [localhost]
TASK [Print Domain] ************************************************************************************************************
ok: [localhost] => {
"domain": "example.com"
}
TASK [Print Server] ************************************************************************************************************
ok: [localhost] => {
"server": "ServerA"
}
PLAY RECAP *********************************************************************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0

Ansible and nested variables in loops

I there a way in Ansible to access variables in variables in a loop? For instance, configuring debconf to install a MySQL/MariaDB server needs two instructions like so:
- name: Define maria root password
shell: echo mysql-server mysql-server/root_password password {{ mysqlRootPass }} | debconf-set-selections
- name: Define maria root password again
shell: echo mysql-server mysql-server/root_password_again password {{ mysqlRootPass }} | debconf-set-selections
But that would be way more compact if I could do that:
- name: Define maria root password
shell: {{ item }}
with_items:
- "{ echo mysql-server mysql-server/root_password password {{ mysqlRootPass }} | debconf-set-selections }"
- "{ echo mysql-server mysql-server/root_password_again password {{ mysqlRootPass }} | debconf-set-selections }"
Obviously, that doesn't work.
so is there a way to make it work? Is there a better way of doing it? Am I missing something?
The {{ }} notation in Ansible comes from Jinja2 templates, it essentially directs Ansible to replace this with the contents of that variable.
with_items introduces the special variable item, which is the particular item of the current loop.
Therefore, you instruct the command module to execute { echo..., namely whatever it is you have in with_items. You only need the command itself: echo....
Reference: Ansible docs: using variables and Jinja2
Here is my complete working MySQL role, that might help you.
vars/main.yml
mysql_root_pass: mypassword #MySQL Root Password
tasks/main.yml
---
- name: Install the MySQL packages
apt:
name: "{{ item }}"
state: installed
update_cache: yes
with_items:
- mysql-server-5.6
- mysql-client-5.6
- python-mysqldb
- libmysqlclient-dev
- name: Update MySQL root password for all root accounts
mysql_user:
name: root
host: "{{ item }}"
password: "{{ mysql_root_pass }}"
state: present
with_items:
- "{{ ansible_hostname }}"
- 127.0.0.1
- ::1
- localhost
- name: Copy the root credentials as .my.cnf file
template:
src: root.cnf.j2
dest: "~/.my.cnf"
mode: 0600
- name: Ensure Anonymous user(s) are not in the database
mysql_user:
name=''
host: "{{ item }}"
state: absent
with_items:
- localhost
- "{{ ansible_hostname }}"
- name: Remove the test database
mysql_db:
name: "test"
state: absent
notify:
- Restart MySQL
templates/root.cnf.j2
[client]
user=root
password={{ mysql_root_pass }}
handlers/main.yml
---
- name: Restart MySQL
service:
name: mysql
state: restarted
My site.yml look like this:
---
- hosts: all
become: yes
gather_facts: yes
roles:
- mysql
If you are just looking to update the password of already configured MySQL:
In vars/main.yml
mysql_old_root_pass: olmysqldpassword
mysql_root_pass: newmysqlpassword
In tasks/main.yml
- name: Set root user password
mysql_user:
name: root
host: "{{ item }}"
password: "{{ mysql_root_pass }}"
check_implicit_admin=yes
login_user: root
login_password: "{{ mysql_old_root_password }}"
state: present
with_items:
- "{{ ansible_hostname }}"
- 127.0.0.1
- ::1
- localhost
Hope this will help you. If you need any help, please check this github link.

Resources