I have a set of 20 jump servers jump_nodes which has connectivity to a set of 100 destination target servers dest_nodes.
Note: We are not sure which of the 20 jump servers has connectivity to which of the 100 destination host.
We have a set of tasks performtask.yml that need to be performed on all dest_nodes hosts.
Below is the playbook for the same.
---
- name: "Play 1"
hosts: localhost
tasks:
- add_host:
hostname: "{{ item }}"
groups: jump_nodes
with_items: "{{ jump_server_list.split('\n') }}"
- add_host:
hostname: "{{ item }}"
groups: dest_nodes
with_items: "{{ target_server_list.split('\n') }}"
- name: "Play 3"
hosts: jump_nodes
tasks:
- name: Perform tasks on dest_nodes
include_tasks: performtask.yml
with_items: "{{ groups['dest_nodes'] }}"
The issue with the above approach is that it takes a lot of time i.e 20 X 100 X <number of tasks> = 2000 X <number of tasks>
My requirement is that if anyone of the jump_nodes host is able to successfully connect and perform the task on a particular dest_nodes IP say 10.0.0.99; then this host 10.0.0.99 should be removed from the dest_nodes group and other jump servers should simply skip that dest_nodes IP.
That way we will be able to considerably reduce the total execution time and the number of tasks.
A feature like remove_hosts equivalent to add_hosts would have done the trick but I understand that there is no such feature.
Update: With the solution proposed by #Kevin C I'm unable to get the difference between hosts, instead i get a syntax error. Nothing of the below works.
groups['dest_nodes' | difference(success_nodes | default([]) )]
groups['dest_nodes' | difference('success_nodes' | default([]))]
groups[dest_nodes | difference(success_nodes | default([]) )]
Can you please suggest?
It's still a bit vague what is expected, and how your current execution looks like.
However, you could create a list of servers which are able to connect from the jumphost to the dest_nodes. This task would take time, but you could use run_once: true to limit execution to one host for this task.
Also, you could use the difference filter to exclude items (dest_nodes) from a list. And then execute task performtask.yml with a when clause.
Ensure to first use:
- set_fact:
list_1: "{{ groups['dest_nodes'] }}"
- set_fact:
list_2: "{{ groups['success_nodes'] }}"
first..
As we spoke in chat, it finally worked!
Related
I was hoping someone could help me. I am new to powershell and struggling with trying to find the right way to approach something in my script. The script is to do a restore check on our backups using the veeam powershell commands. We have 7 backup jobs with various servers in each job and various drives being backed up in each job. Sometimes the servers are in multiple jobs as we have to split the drives across jobs as we copy the backup jobs to USB every day (so we have to balance out the amount of data we backup in each job so that we can copy the drives to the USB drives). I can write the entire script and get it to work but its around 800 lines long and is very inefficient although easy to understand for my team (and me later on!). My current approach pretty much revolves around performing certain actions for each backup job then each server within that job on each disk. I would like to cut it down using an array/loop. I have figured out how to use basic loops and arrays but I am struggling with being able to link the drives and servers in an array for example:
$Backupjob1 = "Backup Job 1E"
$Backupjob1Servers =#('Server1','Server2')
Somekind of array that allows different numbers of multiple drives for Server1 and Server 2.
$Backupjob1ServeDrives =#(Server1.Drive1 ='C', Server1.Drive2 ='F', Server2.Drive1 = 'C')
and then I need to loop through it so that on each loop, it performs an action on server1 and its first drive then does the next drive. Once server1 has finished, it performs the action on server2 on its first drive and then repeats on the second drive.
I understand the basics of looping through an array but I am struggling to understand how i would create an array of arrays that can deal with the above and then loop through it correctly.
I am stuck as not sure what array works and what options there are.
One way would be to externalise your config into a json file or similar and feed that to your script. Here is an example using a json string:
$serverConfig = #"
[{
"serverName": "server1",
"backupDrives": [
"D","F"
]
},
{
"serverName": "server2",
"backupDrives": [
"D","G","H"
]
}]
"#
$serverList = $serverConfig | ConvertFrom-Json
foreach($server in $serverList) {
Write-Host "Server: $($server.serverName)"
foreach($drive in $server.backupDrives) {
Write-Host "Backing up drive $drive..."
# Logic for the backup
}
}
This demonstrates handling the array of arrays you reference, you have an array of server objects and each server object has an array of drives, so on each iteration of the servers you iterate the server's drives too.
If you save the string into a config file your script could read that in, and then if you make any changes to the backup configuration you only need to change the config file and not the script.
It sounds like you want a dictionary type - thankfully PowerShell has a builtin unordered dictionary type called a hashtable that can be used.
In order to construct a hashtable literal, use #{...} instead of #(...):
$driveMapping = #{
Server1 = #{
Drive1 = 'C'
Drive2 = 'F'
}
Server2 = #{
Drive1 = 'C'
}
}
To traverse this data structure, use a simple nested loop:
foreach($server in $drivemapping.psbase.Keys){
Write-Host "About to backup the drives on server '$server'"
foreach($drive in $drivemapping[$server].psbase.Keys){
$driveLetter = $drivemapping[$server][$drive]
Write-Host "About to backup drive '$drive' with letter '$driveLetter' on server '$server'"
}
}
I am using this command https://docs.dbatools.io/#Test-DbaNetworkLatency to test network latency with SQL Server 2016. And it gives me 100ms network latency result (from NetworkOnlyTotal output). However, if I ping the sql server instance I get only 11ms. I wonder what causes the extra 90ms latency in SQL Server. Is it expected? Or what configuration should I look at?
I tried with the -Count parameter and found that the NetworkOnlyTotal doesn't change too much or sometimes even dropped. Does this value mean average?
See below two examples, one is to run query 1 time while the other is to run the query 10 times. The result about NetworkOnlyTotal is even better for 10 times query. From its name, it looks like it is the total time of the 10 requests. But why is the value dropping?
Test-DbaNetworkLatency -SqlCredential $credential -SqlInstance $instance -Count 1
output:
ExecutionCount : 1
Total : 141.55 ms
Average : 141.55 ms
ExecuteOnlyTotal : 69.13 ms
ExecuteOnlyAverage : 69.13 ms
NetworkOnlyTotal : 72.42 ms
Test-DbaNetworkLatency -SqlCredential $credential -SqlInstance $instance -Count 10
output:
ExecutionCount : 10
Total : 180.33 ms
Average : 18.03 ms
ExecuteOnlyTotal : 127.38 ms
ExecuteOnlyAverage : 12.74 ms
NetworkOnlyTotal : 52.95 ms
I wonder what causes the extra 90ms latency in SQL Server. Is it expected?
Probably the one-time connection stuff.
1) Establishing a TCP/IP session
2) Negotiating connection protocol encryption
3) Logging in and creating a session
Try a higher -Count Establishing a connection and a session take some time, and shouldn't really be counted as "network latency", as clients will hold open and reuse connections.
The description of the product indicates that "It will then output how long the entire connection and command took, as well as how long only the execution of the command took." Additionally, it says that it will execute the command three times. And the tool will need to take a little time to authenticate the connection with SQL Server. So, it seems reasonable to me.
I am trying to apply the thoward-windows_firewall module to Windows 10 and hit a snag whereby most of the built in rules can be purged with the module's purge function, but some others (such as cortana) cannot. This problem didn't occur in earlier Windows builds. The workaround to this seems to be to run an exec with 'netsh advfirewall reset' which purges them and stops them reappearing.
However, I only want to apply the 'netsh advfirewall reset' exec if firewall rules other than what I have specified in Hiera exist.
My approach is to build an array (or list) of only the titles in hiera and then iterate over them in a PowerShell 'only if' or 'unless'.
Code so far (which is not working) is:
Hiera snippet:
harden::firewall:
'Remote Desktop - User Mode (UDP-In)':
description: 'Inbound rule for the Remote Desktop service to allow RDP traffic. [UDP 3389]'
application_name: 'C:\Windows\system32\svchost.exe'
service_name: 'termservice'
protocol: 17
local_ports: '3389'
remote_ports: '*'
local_addresses: '%{facts.networking.ip}'
remote_addresses: '*'
direction: 1
interface_types: 'All'
enabled: true
grouping: '#FirewallAPI.dll,-28752'
profiles: 3
action: 1
'File and Printer Sharing (Echo Request - ICMPv4-In) 1':
description: 'Echo Request messages are sent as ping requests to other nodes.'
protocol: 1
local_addresses: '%{facts.networking.ip}'
remote_addresses: 'LocalSubnet'
icmp_types_and_codes: '8:*'
direction: 1
interface_types: 'All'
enabled: true
grouping: '#FirewallAPI.dll,-28502'
profiles: 6
action: 1
Manifest (extract):
class harden (
Hash $firewall_rule_names = lookup('harden::firewall'),
){
# reset firewall rules
exec { 'reset_firewall':
command => 'netsh advfirewall reset',
onlyif => 'if (Get-NetFirewallRule | where {\$_.DisplayName -notmatch $firewall_rule_names}) { exit 0 } else { exit 1 }',
provider => powershell,
}
Class { 'windows_firewall':
profile_state => 'on',
in_policy => 'BlockInbound',
out_policy => 'BlockOutbound',
rule_key => 'harden::firewall',
purge_rules => true,
}
I know I need to have a .each look in there somewhere, and also tidy up the powershell 'only if' so that it looks at just the titles in the hash (perhaps re-written to an array of just hash titles) and runs the exec if there are rules on the host that aren't in Hiera, but am getting a bit lost.
Any help sincerely appreciated.
You wrote:
My approach is to build an array (or list) of only the titles in hiera
and then iterate over them in a PowerShell 'only if' or 'unless'.
Apparently, you attempt to build that array in class parameter $firewall_rule_names, with this declaration:
Hash $firewall_rule_names = lookup('harden::firewall'),
and then you attempt to use that list in the onlyif parameter of your Exec:
onlyif => 'if (Get-NetFirewallRule | where {\$_.DisplayName -notmatch $firewall_rule_names}) { exit 0 } else { exit 1 }',
There are multiple problems with that.
First, if you want to interpolate a puppet variable into a string, then that string needs to be quoted with double quotes ("); quoting with single quotes (') suppresses interpolation (and also treats \$ as two literal characters, not an escape sequence).
Second, and where you seem to be at loss, as how to extract the keys of your $firewall_rule_names hash and format them appropriately. I'm uncertain exactly what Powershell requires here, but some of the best tools to use to get it would be the keys() and join() functions provided by the puppetlabs/stdlib module or by Puppet itself if you're using version 5.5 or later. For example, if all you need is a comma-delimited list of the names then something like this would do it:
$really_the_rule_names = join(keys($firewall_rule_names), ', ')
I'm suspicious, though, that you may need to quote the keys. You could get most of that by being clever with the delimiter you specify to join, but you might also want to consider processing the key array with the built-in regsubst() function before joining results together into a string.
I can see a problem in the Only IF statement.
$_DisplayName should be $_.DisplayName , you need to use Dot operator to cherry pick the properties of an Object and \ is also not required.
#John-Bollinger - Puppet and PowerShell = PuppetHell! :)
I finally managed to solve this problem but feel that my code blocks are way to long with too many substitutions. I've tried shorter methods to no avail, so if anyone can suggest working shorter code that would be appreciated.
In short, I ended up having to
get the hiera hash keys (titles) into suitable array syntax for Powershell comparison by:
a. adding single quotes and commas as separators
b. adding a single quote and double quote at the beginning and end of array
c. removing parenthesis from anywhere in the array as PowerShell spat it on them
$firewall_hiera = regsubst(regsubst(regsubst(join(keys($firewall_rules), "', '"), '^', '"\''), '$', '\'"'), '[\(\)]', '', 'G')
produces
"'Core Networking - DNS UDP-Out', 'Core Networking - Dynamic Host Configuration Protocol DHCP-Out', 'File and Printer Sharing Echo Request - ICMPv4-Out', 'Internet Browsing HTTP-Out', 'Internet Browsing HTTPS-Out'"
…
Use a Powershell exec to clean out the Firewall if undefined in Puppet hiera by:
a. using an exec command to purge the host firewall rules with:
command => 'netsh advfirewall firewall delete rule name=all; reg delete "HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules" /f',
b. ensuring idempotency with a very long unless statement:
unless => "if (#(Compare-Object((Get-NetFirewallRule | foreach {\"'\"+\$_.DisplayName+\"'\"}) -join ', ' -replace '[()]') $firewall_hiera).length -eq 0) {exit 0} else {exit 1}",
which produces for the following to compare-object with the above created $firewall_hiera
'Core Networking - DNS UDP-Out', 'Core Networking - Dynamic Host Configuration Protocol DHCP-Out', 'File and Printer Sharing Echo Request - ICMPv4-Out', 'Internet Browsing HTTP-Out', 'Internet Browsing HTTPS-Out'
Desired rules are then recreated (if a purge occurs) using the thoward-windows_firewall module which iterates over the same source hiera. This module is HEAPS better than the rather cr4ppy puppetlabs-windows_firewall module, but can't handle Windows 10's horrible firewall additives after user logon.
Interestingly, only the puppet created array needs outer double quotes. As the puppetlabs-powershell module doesn't echo the .ps1's to disk, I had to echo them out manually to get an idea of what was actually being created and test them in Powershell ISE (the older Josh Cooper powershell module did create temporary .ps1's which was handy though less secure)
I was able to put some regsubst or replace characters in arrays, but ^ and $ do not go well in an array.
All this was to solve Windows 10's incessant creation of additional pointless firewall rules only after a user logs on. This continues intermitently for about 6 puppet runs and finally stops. Never had this problem with Server 2012 R2 for example.
Deleting the following registry key (with Puppetlabs registry module ensure => absent) provides some additional quietening but its not an end in itself:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\RestrictedServices\AppIso\FirewallRules
Again, ANY assistance with shortening the above code, particularly with regards to the regsubst and -replace commands is appreciated. Keep in mind that the produced arrays need to be comparable using Powershell's 'compare-object' feature or similar.
Thanks to #John-Bollinger for getting me started with creating the array from a hiera hash using the keys and join functions.
I have a host file that looks like
[foo]
foox 192.168.0.1 id=1
fooy 192.168.0.1 id=2
fooz 192.168.0.1 id=3
However, I'd like to more concisely write this using patterns like:
[foo]
foo[x:z] 192.168.0.1 id=[1:3]
But this is getting interpreted as id equaling the raw text of "[1:3]", rather than 1, 2, or 3. Is there a way to achieve this in the inventory file, or will I need to do something through host vars and/or group vars?
This can't be done within an inventory file. I think set_fact is your best bet to programmatically build an inventory this simple.
---
- hosts: all
tasks:
- add_host:
name: "host{{ item }}"
ansible_ssh_host: "127.0.0.1"
ansible_connection: "local"
group: "new"
id: "{{ item }}"
with_sequence: count=3
delegate_to: localhost
run_once: yes
- hosts: new
tasks:
- debug:
msg: "{{ id }}"
If I recall correctly, Jinja capabilities have been removed from every place they shouldn't have been, i.e. outside quotes, braces, special cases like when: in YML files.
When I say programmatically, though, we're talking about Ansible.. one of the last candidates on earth for general purpose scripting. Dynamic inventory scripts are a better approach to problems like these, unless we're talking three servers exactly.
The simplest inventory script to accomplish this would be (in your hosts dir or pointed to by the -i switch:
#!/usr/bin/env python
import json
inv = {}
for i in range(3):
inv[i] = {"hosts":["host%s" % i],"vars":{"id":i,"ansible_ssh_host":"127.0.0.1", "ansible_connection":"local"}}
print json.dumps(inv)
Again, I'm afraid there is nothing as "pretty" as what you're looking for. If your use case grows more complex, then set_fact, set_host and group_by may come in handy, or an inventory script, or group_vars (I do currently use group_vars files for server number).
This is best done using Ansible's Dynamic Inventory features. See Developing Dynamic Inventory Sources.
This means writing a script that returns your hostname in a JSON format.
We have introduced a new model to our Datastore a few days ago. Surprisingly I still get Index warnings
W 2014-02-09 03:38:28.480
suspended generator run_to_queue(query.py:938) raised NeedIndexError(no matching index found.
The suggested index for this query is:
- kind: FeelTrackerRecord
ancestor: yes
properties:
- name: timestamp)
W 2014-02-09 03:38:28.480
suspended generator helper(context.py:814) raised NeedIndexError(no matching index found.
The suggested index for this query is:
- kind: FeelTrackerRecord
ancestor: yes
properties:
- name: timestamp)
even though the index is served under DataStore Indexes
indexes:
# AUTOGENERATED
# This index.yaml is automatically updated whenever the dev_appserver
# detects that a new type of query is run. If you want to manage the
# index.yaml file manually, remove the above marker line (the line
# saying "# AUTOGENERATED"). If you want to manage some indexes
# manually, move them above the marker line. The index.yaml file is
# automatically uploaded to the admin console when you next deploy
# your application using appcfg.py.
- kind: FeelTrackerRecord
ancestor: yes
properties:
- name: record_date
- name: timestamp
What am I missing please?
I finally found the problem.
Best way to solve this is to make sure the local index.yaml is empty (delete all the indices). Then simply run your GAE app on localhost and access your app as you would expect.
Http access is pretty straightforward over browser and if GET/POST over REST is required you can use curl from a terminal:
GET:
curl --user test#gmail.com:test123 http://localhost:8080/api/v1.0/records/1391944029
POST:
curl --user test#gmail.com:test123 http://localhost:8080/api/v1.0/records/1391944029 -d '{"records": [
{
"notes": "update",
"record_date": "2014-02-02",
"timestamp": 1391944929
}
], "server_sync_timestamp": null}' -X POST -v -H "Accept: application/json" -H "Content-type: application/json"
GAE is now updating the index.yaml automatically and add the correct indices in there.
After your deploying your app, you need to cleanup the old indices.
This is done through a terminal:
appcfg.py vacuum_indexes src
After login with credentials it will ask you about the old indices that are no longer in your index.yaml and if they should be deleted. Press y and continue.
I mentioned in a comment your indexes don't match the required. The error says
raised NeedIndexError(no matching index found.
The suggested index for this query is:
- kind: FeelTrackerRecord
ancestor: yes
properties:
- name: timestamp)
However you the index you list is different
- kind: FeelTrackerRecord
ancestor: yes
properties:
- name: record_date
- name: timestamp
Do you see the difference ?
Just add the index as listed and update your indexes.