I'm currently making the transition from Puppet to Ansible and so far so good. Yet I want to automatize as much as possible.
I'm trying to use a with_items loop inside vars_files to load variable files based on a given list of items. Ansible complains about the syntax and I can't seem to find an example of a similar solution, only examples that use with_items inside tasks and roles.
For example:
vars_files:
- ["vars/{{ item }}-{{ ansible_fqdn }}.yml", "vars/{{ item }}-{{ system_environment }}.yml", "vars/{{ item }}.yml"]
with_items:
- php
- nginx
The goal here is to loop the second line for as long as there are items in with_items using an array to fallback on the next item if it can't find the given file (which works).
Not sure if this is at all possible, but I wanted to ask before taking another direction.
with_items, or in general all loops, are a feature of tasks. vars_files though is no task. So it won't work the way you tried it and the short answer would be: It is not possible.
I don't know of a clean way to solve your exact problem. A custom vars plugin might be an option. But vars plugin work on a global level while your vars seem to be used in a role.
A custom lookup plugin might be a solution if solving this on task level is an option for you. The lookup plugin takes your input, checks for presence of the files and returns an array of the files which need to be include. This then can be used with the include_vars module.
- include_vars: "{{ item }}"
with_my_custom_plugin:
- php
- nginx
An ugly solution would be to combine the with_items loop with a with_first_found loop. Though, since you cannot directly nest loops, you need to work with an include.
- include: include_vars.yml
with_items:
- php
- nginx
And inside include_vars.yml you then can use with_first_found with the include_vars module.
- include_vars: "{{ item }}"
with_first_found:
- vars/{{ item }}-{{ ansible_fqdn }}.yml
- vars/{{ item }}-{{ system_environment }}.yml
- vars/{{ item }}.yml
Putting this in a separate answer to expand on the group and host variables solution I eventually came up with (cc #udondan).
Basically I group all my hosts in my inventory file under several sub and parent groups no matter what. Then I create files for group vars whenever applicable so it follows a certain order of precedence (first is highest and overrides all others, last applies to all hosts and can be overridden down the chain):
task vars > playbook vars > host_vars > web/database-local > local > web/database > all
That way I can define variables for all hosts to use (all), just web/database (mostly production values), all local servers (local group), all local web/database servers, et cetera, or per-host (the standard host_vars). Of course playbook and task vars override these further. All of this following the Ansible guidelines.
An example of a local inventory (replace default with your hostname or IP, add as many as you like per group, x-local can be omitted if this would be a production inventory):
[web-local]
default
[database-local]
default
[local:children]
web-local
database-local
[web:children]
web-local
[database:children]
database-local
Then my group_vars folder with directories for each inventory group and variables split into files to keep it structured (could just have one database-local.yaml file for the database-local group for instance instead of folders and split YAML-files):
group_vars/
all/
always_applied_variables.yaml
swap.yaml
web/
database/
database_only_variables.yaml
database-production/
production_database_variables.yaml
production/
random_production_only_variables.yaml
local/
users.yaml
web-local/
database-local/
local_database_variables.yaml
host_vars/
default/
php.yaml
mysql.yaml
other_specific_host_variables.yaml
Hope this is somewhat clear. I'd be happy to answer any questions.
Related
I am using getdbt on redshift for data analytics operation. Can anyone please suggest, how to use --selector & --defer with "dbt run" commands.
What is the syntax ? What is the use of selectors.yml file?
Please share some examples.
Thanks
My interpretation of defer is a way to utilize the dbt cli to work with unbuilt or differential versions of the current & futures state defined versions of a model.
Example of why you may want to interact with that here: #2740 - Automating Non Regression Test
selectors being a relatively new feature, I also haven't seen much documentation to back this up but it is effectively a naming convention for a set of logical criteria (more than 1 tag, multiple directories, etc.)
I'd recommend this article in general for understanding the build path generation of a typical dbt run: How we made dbt runs 30% faster
From there, you can imagine that within a large project, there are huge interconnecting chains for each raw -> analytics ready transformation pipeline that you have.
We'll use Gitlab's open dbt project as an example.
Gitlab doesn't currently use selectors but they do make use of tags.
So they could build up a selectors.yml file using logical definitions like:
selectors.yml
selectors:
- name: sales_funnel
definition:
tag: salesforce
tag: sales_funnel
- name: arr
description: builds all arr models to current state + all upstream dependencies (zoho, zuora subscriptions, etc.)
default: true
definition:
tag: zuora_revenue
tag: arr
- name: month_end_process
description: builds reporting models about customer segments based on subscription activity for latest closed month
definition:
- union:
- method: fqn
value: rpt_available_to_renew_month_end
greedy: eager # default: will include all tests that touch selected model
- method: fqn
value: rpt_possible_to_churn_month_end
greedy: eager
Full list of valid selector definitions here: https://docs.getdbt.com/reference/node-selection/yaml-selectors#default
What that gives them the ability to do is on a cron job, via airflow, or some other orchestrator simply execute:
dbt run --selector month_end_process --full-refresh
And have confidence that the logical selection of models to run for that process is 100% accurately reproduced instead of another more fallible approach like assuming that all the models needed are in a single directory:
dbt run --models marts.finance.restricted_safe.reports --full-refresh
Architecturally, you likely won't need selectors until you get to the level of having multiple layers of tags and / or multiple layers of use-case directories to be mindful of within a single run.
Example: tags for the models' function, tags for the sources, tags for the bi/analyst consumers, tags for the materialization schedule, etc.
Hugo's one-file->one-page model is nice and simple, especially for a blog. But sometimes you want to write an "article" for a blog and split it into 2 or more pieces for separate pages (perhaps to be posted on separate days, or whatever).
Is there a way to do this within Hugo? Perhaps a combination of something you put in a layout/theme/shortcode and internal markup within the page (to signal where to split the article)?
Possible models might include:
1 input post "splits" into 2/3/4 posts when the site is built to public
1 input post is duplicated into 2/3/4 posts when the site is built to public but somehow each duplicate isn't an exact duplicate but instead has the whole post but certain parts of the post are hidden/invisible, via CSS, such that they represent the 2/3/4 "pages" of the post.
Or, is this something you do external to Hugo?
UPDATE: I can see I need to clarify this. Consider this random illustrative blog post - it is the third of three closely related posts, and even has a set of links at the top so you can find the earlier posts in the series. Lots of technical blogs do this sort of thing (at least the ones I read).
Now, I'm not looking for a CMS or anything complex. What I do now with Hugo is hugo new posts/an-article-about-constexpr.md and I write one markdown file and it becomes one "post" in standard Hugo fashion. Exactly what you want a SSG to do.
What I want to do is write one markdown file but have some kind of markup in it separating it into sections (like <!-- More --> on steroids) so that instead of generating one page of my site it generates three (in this example) - three separate articles with links from the main page in the "posts" section, etc. etc. And for bonus points, I'd like to generate these "table of contents" sections with links to each of the pages.
So I've been doing that with a cobbled-up awk script that generates pages right next to the post, in the posts directory. I set the post to draft so it doesn't get published, but the pages generated by the awk script have draft=false so they do get published. And the dates get set so they're "in order".
And that's working, but before I invest more time in my little script, I wanted to see if there was a proper way to do this within hugo.
Not sure what you mean by one-file->one-page model.
I have very few parts of any hugo site which one markdown file=one rendered html page.
Could just be the way I build, but everything I've done so far has been vanilla hugo.
To answer your question: Yes, you are correct that would work. There a few ways to do this (I list one below), but maybe a deeper look would be separating the concept of a "tool-chain" and what Hugo is in that tool chain, from a CMS, which Hugo is not.
So, to possibly answer your specific question though:
You can store content in markdown, markdown front matter, or a Data form (XML/JSON) in hugo. Using the page resources {{ .GetPage }} you can access any content and load it in any template or using shortcodes, load it in other markdown.
If I needed to do this as part of a tool chain, i.e. use specific markdown and re-use it in multiple places, I would create a front matter variable, or taxonomy or tag depending on what groupings I needed where, so this was scalable. params such as
"articleAuthor: Jessie P."
"date: DATE HERE"
"tags: etc. etc."
Then lets say I know that's going to be a blog, well fine, then it will be in the corresponding content folder, but if I needed all of Jessie's articles, or articles on that date, or that specific article, I would use the shortcode I make or directly in a template, using .GetPage Match - import the markdown pages I need based on the parameters I need.
But on the other hand, I would need to understand the problem being solved, but, here are a few hugo docs to help you out:
https://gohugo.io/functions/getpage/#readout
https://gohugo.io/content-management/page-bundles/
Remember, Hugo is not a CMS, it is a site generator. If you want a CMS, you can always use Wordpress headless, or any other solution out there.
(off the top of my head using page bundles)
{{ $headlessBundle := .Site.GetPage "/blogs/specific-blog/index" }}
{{ with $getContent := $headlessBundle.Resources.Match "intro.md" }}
{{ (index $getContent 0).Content }}
(You would use various "Where" statements to "filter" content based on the params or however you delineate what you want).
Or for instance if I wanted only the text that had an H1 tag:
{{ $.Scratch.Set "summary" ((delimit (findRE "(<h1.*?>.*?</h1>\\s*)+" .Content) "[…]") | plainify | replaceRE "&" "&" | safeHTML) }}
{{ $.Scratch.Get "summary" }}
Based on the update to the question:
https://discourse.gohugo.io/t/split-markdown-content-in-two-files-but-dont-render-shortcodes-as-raw-text/32080/2
https://discourse.gohugo.io/t/getting-a-list-from-within-a-shortcode/28126
https://discourse.gohugo.io/t/splitting-content-into-sections-based-on-header-level/33749
https://discourse.gohugo.io/t/multiple-content-blocks-on-a-single-page/9092/3
jrmooring answered it best in the above with clear examples and code.
Though, note: If I was doing this in a technical blog this would be integrated into the CMS and coordinated with the builder.
I am writing a playbook that creates a network with a name I assign. Later on in the playbook I need to access the IP that is assigned by this network task, so I get it from the hostvars. So for example it if I was calling the network 'my_website' the value I would be targeting in hostvars would be
server: "{{hostvars.localhost.rax_nfs.results[0].success[0].rax_addresses.my_website[0].addr}}"
This is fine, but I want to name the network based on the contents of a variable passed in by a var file to make it reusable across multiple setups, and then still be able to get that IP back, so
network_label: "{{ my_website }}"
server: "{{hostvars.localhost.rax_nfs.results[0].success[0].rax_addresses.network_label[0].addr}}"
Obviously this doesn't work as it just assigns a string. How do I use network_label as the key inside another variable? Like in php something like
$array[$variable], or $object->$variable
Is this possible?
Not sure what the playbook looks like but does accessing the IP based on the gathered facts not work in your context? Example: {{ ansible_eth0.ipv4.address }}
Alternatively on the task that creates the network you could register that as a variable for later access using that variable name.
You can also access elements of an array in a variable: {{ var_name[0] }}
I've created an Ansible playbook that creates a cloud instance and then installs some programs on the instance. I want to run this playbook multiple times (without using a bash script). Is it possible to use a loop to loop over those two tasks together (I.E. One loop for two tasks?). All I've been able to find so far is one loop for each individual task
An update:
In 2.0 you are able to use with_ loops and task includes (but not playbook includes), this adds the ability to loop over the set of tasks in one shot. There are a couple of things that you need to keep in mind, a included task that has it’s own with_ loop will overwrite the value of the special item variable. So if you want access to both the include’s item and the current task’s item you should use set_fact to create a alias to the outer one.:
- include_tasks: test.yml
with_items:
- 1
- 2
- 3
in test.yml:
- set_fact: outer_loop="{{item}}"
- debug: msg="outer item={{outer_loop}} inner item={{item}}"
with_items:
- a
- b
- c
Source: Ansible Docs
No that's currently not possible. with_items used to work with the include statement in previous versions of Ansible but was unfortunately dropped.
Though it will be brought back in Ansible 2.0, see slide 14/15 of What's New in v2 - AnsibleFest London 2015
You could try to work with the v2 branch from github, the feature should be available in there.
What you can do with 1.9.1 is to move your tasks into a role and reference this role multiple times in your playbook.
I managed to do this by recursively including the same yaml file based on a condition. Here is the gist: https://gist.github.com/ParagDoke/5ddfc3d5647ce9b0110d1b9790090092. Effectively, in your playbook, include a file with some vars:
- name: Invoke poller
vars:
some_condition: '"failed" not in response.content and response.json.status=="running"'
include_tasks: status-poller.yml
Then in status-poller.yml, include itself:
- include_tasks: includes/status-poller.yml
when: some_condition
I created a block that I want to appear on these paths:
example.com/sample/1
example.com/sample/2 example.com/sample/3
example.com/sample/4 example.com/sample/6
However, I don't want it to appear on:
example.com/sample/5
Under the visibility setting for the block, I can select show block on "Only the listed pages"
and enter something like /sample/*
Howevever, how do I tell it not to show up in /sample/5 without typing out all other paths individually? Is there an "except" or "not" indicator somehow like how the * indicates all?
Use the context module to handle the placement of your block. It allows you to specify which paths the block should display on, as well as which it should not (by starting the path with a ~)
For example, in your context you can specify your paths like so:
sample/*
~sample/5
this tells drupal to display your block on all paths that match "sample/*" except for "sample/5"
There is only two ways of getting the fine tuning you need:
You type one by one all the URLs you want to include/exclude
You go for the perfectly customizable php code mode.
Maybe you should try Context module http://drupal.org/project/context and see if the more complex, configurable options it provide serve your purpose/solve your problem.
PD. My first answer completely missed the point, i was thinking on views... sorry!