I have a host file that looks like
[foo]
foox 192.168.0.1 id=1
fooy 192.168.0.1 id=2
fooz 192.168.0.1 id=3
However, I'd like to more concisely write this using patterns like:
[foo]
foo[x:z] 192.168.0.1 id=[1:3]
But this is getting interpreted as id equaling the raw text of "[1:3]", rather than 1, 2, or 3. Is there a way to achieve this in the inventory file, or will I need to do something through host vars and/or group vars?
This can't be done within an inventory file. I think set_fact is your best bet to programmatically build an inventory this simple.
---
- hosts: all
tasks:
- add_host:
name: "host{{ item }}"
ansible_ssh_host: "127.0.0.1"
ansible_connection: "local"
group: "new"
id: "{{ item }}"
with_sequence: count=3
delegate_to: localhost
run_once: yes
- hosts: new
tasks:
- debug:
msg: "{{ id }}"
If I recall correctly, Jinja capabilities have been removed from every place they shouldn't have been, i.e. outside quotes, braces, special cases like when: in YML files.
When I say programmatically, though, we're talking about Ansible.. one of the last candidates on earth for general purpose scripting. Dynamic inventory scripts are a better approach to problems like these, unless we're talking three servers exactly.
The simplest inventory script to accomplish this would be (in your hosts dir or pointed to by the -i switch:
#!/usr/bin/env python
import json
inv = {}
for i in range(3):
inv[i] = {"hosts":["host%s" % i],"vars":{"id":i,"ansible_ssh_host":"127.0.0.1", "ansible_connection":"local"}}
print json.dumps(inv)
Again, I'm afraid there is nothing as "pretty" as what you're looking for. If your use case grows more complex, then set_fact, set_host and group_by may come in handy, or an inventory script, or group_vars (I do currently use group_vars files for server number).
This is best done using Ansible's Dynamic Inventory features. See Developing Dynamic Inventory Sources.
This means writing a script that returns your hostname in a JSON format.
Related
I was hoping someone could help me. I am new to powershell and struggling with trying to find the right way to approach something in my script. The script is to do a restore check on our backups using the veeam powershell commands. We have 7 backup jobs with various servers in each job and various drives being backed up in each job. Sometimes the servers are in multiple jobs as we have to split the drives across jobs as we copy the backup jobs to USB every day (so we have to balance out the amount of data we backup in each job so that we can copy the drives to the USB drives). I can write the entire script and get it to work but its around 800 lines long and is very inefficient although easy to understand for my team (and me later on!). My current approach pretty much revolves around performing certain actions for each backup job then each server within that job on each disk. I would like to cut it down using an array/loop. I have figured out how to use basic loops and arrays but I am struggling with being able to link the drives and servers in an array for example:
$Backupjob1 = "Backup Job 1E"
$Backupjob1Servers =#('Server1','Server2')
Somekind of array that allows different numbers of multiple drives for Server1 and Server 2.
$Backupjob1ServeDrives =#(Server1.Drive1 ='C', Server1.Drive2 ='F', Server2.Drive1 = 'C')
and then I need to loop through it so that on each loop, it performs an action on server1 and its first drive then does the next drive. Once server1 has finished, it performs the action on server2 on its first drive and then repeats on the second drive.
I understand the basics of looping through an array but I am struggling to understand how i would create an array of arrays that can deal with the above and then loop through it correctly.
I am stuck as not sure what array works and what options there are.
One way would be to externalise your config into a json file or similar and feed that to your script. Here is an example using a json string:
$serverConfig = #"
[{
"serverName": "server1",
"backupDrives": [
"D","F"
]
},
{
"serverName": "server2",
"backupDrives": [
"D","G","H"
]
}]
"#
$serverList = $serverConfig | ConvertFrom-Json
foreach($server in $serverList) {
Write-Host "Server: $($server.serverName)"
foreach($drive in $server.backupDrives) {
Write-Host "Backing up drive $drive..."
# Logic for the backup
}
}
This demonstrates handling the array of arrays you reference, you have an array of server objects and each server object has an array of drives, so on each iteration of the servers you iterate the server's drives too.
If you save the string into a config file your script could read that in, and then if you make any changes to the backup configuration you only need to change the config file and not the script.
It sounds like you want a dictionary type - thankfully PowerShell has a builtin unordered dictionary type called a hashtable that can be used.
In order to construct a hashtable literal, use #{...} instead of #(...):
$driveMapping = #{
Server1 = #{
Drive1 = 'C'
Drive2 = 'F'
}
Server2 = #{
Drive1 = 'C'
}
}
To traverse this data structure, use a simple nested loop:
foreach($server in $drivemapping.psbase.Keys){
Write-Host "About to backup the drives on server '$server'"
foreach($drive in $drivemapping[$server].psbase.Keys){
$driveLetter = $drivemapping[$server][$drive]
Write-Host "About to backup drive '$drive' with letter '$driveLetter' on server '$server'"
}
}
I have a requirement where I need to run query like below and fetch 2-3 attributes for all entities satisfying this query. The number of distinguishedName would be around 100 in a single query. As I see in the microsoft documentation, that distinguishedName is not indexed, I suspect that this might cause performance issues.
Does anybody know if this would indeed cause performance issues? Apart from the below ldap filter, I would obviously have to use SUBTREE scope.
(|(distinguishedName=<DN 1 goes here>)(distinguishedName=<DN 2 goes here>))
Edit 1:
I tried this in my test Active Directory which has ~6k entries.
Internal event: A client issued a search operation with the following options.
Starting node:
DC=example,DC=com
Filter:
( |
(distinguishedName=CN=user-1,CN=large-test,CN=Users,DC=example,DC=com)
(distinguishedName=CN=user-2,CN=large-test,CN=Users,DC=example,DC=com)
(distinguishedName=CN=user-3,CN=large-test,CN=Users,DC=example,DC=com)
(distinguishedName=CN=group1,CN=large-test,CN=Groups,DC=example,DC=com)
)
Search scope:
subtree
Attribute selection:
mail,objectClass
Server controls:
Visited entries:
4
Returned entries:
4
Used indexes:
idx_distinguishedName:4:N;idx_distinguishedName:1:N;idx_distinguishedName:1:N;idx_distinguishedName:1:N;
Pages referenced:
123
Pages read from disk:
0
From the results it looks like it only visited 4 entries that I searched for using some indexes. I checked with schema snap-in tool (just to be sure) and it doesn't show indexes on distinguishedName. Not sure how it's using these indexes though.
Microsoft Active Directory stores the group memberships at the entry level, so you could use this to fetch the email attribute.
E.g.
ldapsearch .... -b SEARCH_BASE "(|(memberOf=GROUP_DN_1)(memberOf=GROUP_DN_2)...)" mail
I have a set of 20 jump servers jump_nodes which has connectivity to a set of 100 destination target servers dest_nodes.
Note: We are not sure which of the 20 jump servers has connectivity to which of the 100 destination host.
We have a set of tasks performtask.yml that need to be performed on all dest_nodes hosts.
Below is the playbook for the same.
---
- name: "Play 1"
hosts: localhost
tasks:
- add_host:
hostname: "{{ item }}"
groups: jump_nodes
with_items: "{{ jump_server_list.split('\n') }}"
- add_host:
hostname: "{{ item }}"
groups: dest_nodes
with_items: "{{ target_server_list.split('\n') }}"
- name: "Play 3"
hosts: jump_nodes
tasks:
- name: Perform tasks on dest_nodes
include_tasks: performtask.yml
with_items: "{{ groups['dest_nodes'] }}"
The issue with the above approach is that it takes a lot of time i.e 20 X 100 X <number of tasks> = 2000 X <number of tasks>
My requirement is that if anyone of the jump_nodes host is able to successfully connect and perform the task on a particular dest_nodes IP say 10.0.0.99; then this host 10.0.0.99 should be removed from the dest_nodes group and other jump servers should simply skip that dest_nodes IP.
That way we will be able to considerably reduce the total execution time and the number of tasks.
A feature like remove_hosts equivalent to add_hosts would have done the trick but I understand that there is no such feature.
Update: With the solution proposed by #Kevin C I'm unable to get the difference between hosts, instead i get a syntax error. Nothing of the below works.
groups['dest_nodes' | difference(success_nodes | default([]) )]
groups['dest_nodes' | difference('success_nodes' | default([]))]
groups[dest_nodes | difference(success_nodes | default([]) )]
Can you please suggest?
It's still a bit vague what is expected, and how your current execution looks like.
However, you could create a list of servers which are able to connect from the jumphost to the dest_nodes. This task would take time, but you could use run_once: true to limit execution to one host for this task.
Also, you could use the difference filter to exclude items (dest_nodes) from a list. And then execute task performtask.yml with a when clause.
Ensure to first use:
- set_fact:
list_1: "{{ groups['dest_nodes'] }}"
- set_fact:
list_2: "{{ groups['success_nodes'] }}"
first..
As we spoke in chat, it finally worked!
I am trying to apply the thoward-windows_firewall module to Windows 10 and hit a snag whereby most of the built in rules can be purged with the module's purge function, but some others (such as cortana) cannot. This problem didn't occur in earlier Windows builds. The workaround to this seems to be to run an exec with 'netsh advfirewall reset' which purges them and stops them reappearing.
However, I only want to apply the 'netsh advfirewall reset' exec if firewall rules other than what I have specified in Hiera exist.
My approach is to build an array (or list) of only the titles in hiera and then iterate over them in a PowerShell 'only if' or 'unless'.
Code so far (which is not working) is:
Hiera snippet:
harden::firewall:
'Remote Desktop - User Mode (UDP-In)':
description: 'Inbound rule for the Remote Desktop service to allow RDP traffic. [UDP 3389]'
application_name: 'C:\Windows\system32\svchost.exe'
service_name: 'termservice'
protocol: 17
local_ports: '3389'
remote_ports: '*'
local_addresses: '%{facts.networking.ip}'
remote_addresses: '*'
direction: 1
interface_types: 'All'
enabled: true
grouping: '#FirewallAPI.dll,-28752'
profiles: 3
action: 1
'File and Printer Sharing (Echo Request - ICMPv4-In) 1':
description: 'Echo Request messages are sent as ping requests to other nodes.'
protocol: 1
local_addresses: '%{facts.networking.ip}'
remote_addresses: 'LocalSubnet'
icmp_types_and_codes: '8:*'
direction: 1
interface_types: 'All'
enabled: true
grouping: '#FirewallAPI.dll,-28502'
profiles: 6
action: 1
Manifest (extract):
class harden (
Hash $firewall_rule_names = lookup('harden::firewall'),
){
# reset firewall rules
exec { 'reset_firewall':
command => 'netsh advfirewall reset',
onlyif => 'if (Get-NetFirewallRule | where {\$_.DisplayName -notmatch $firewall_rule_names}) { exit 0 } else { exit 1 }',
provider => powershell,
}
Class { 'windows_firewall':
profile_state => 'on',
in_policy => 'BlockInbound',
out_policy => 'BlockOutbound',
rule_key => 'harden::firewall',
purge_rules => true,
}
I know I need to have a .each look in there somewhere, and also tidy up the powershell 'only if' so that it looks at just the titles in the hash (perhaps re-written to an array of just hash titles) and runs the exec if there are rules on the host that aren't in Hiera, but am getting a bit lost.
Any help sincerely appreciated.
You wrote:
My approach is to build an array (or list) of only the titles in hiera
and then iterate over them in a PowerShell 'only if' or 'unless'.
Apparently, you attempt to build that array in class parameter $firewall_rule_names, with this declaration:
Hash $firewall_rule_names = lookup('harden::firewall'),
and then you attempt to use that list in the onlyif parameter of your Exec:
onlyif => 'if (Get-NetFirewallRule | where {\$_.DisplayName -notmatch $firewall_rule_names}) { exit 0 } else { exit 1 }',
There are multiple problems with that.
First, if you want to interpolate a puppet variable into a string, then that string needs to be quoted with double quotes ("); quoting with single quotes (') suppresses interpolation (and also treats \$ as two literal characters, not an escape sequence).
Second, and where you seem to be at loss, as how to extract the keys of your $firewall_rule_names hash and format them appropriately. I'm uncertain exactly what Powershell requires here, but some of the best tools to use to get it would be the keys() and join() functions provided by the puppetlabs/stdlib module or by Puppet itself if you're using version 5.5 or later. For example, if all you need is a comma-delimited list of the names then something like this would do it:
$really_the_rule_names = join(keys($firewall_rule_names), ', ')
I'm suspicious, though, that you may need to quote the keys. You could get most of that by being clever with the delimiter you specify to join, but you might also want to consider processing the key array with the built-in regsubst() function before joining results together into a string.
I can see a problem in the Only IF statement.
$_DisplayName should be $_.DisplayName , you need to use Dot operator to cherry pick the properties of an Object and \ is also not required.
#John-Bollinger - Puppet and PowerShell = PuppetHell! :)
I finally managed to solve this problem but feel that my code blocks are way to long with too many substitutions. I've tried shorter methods to no avail, so if anyone can suggest working shorter code that would be appreciated.
In short, I ended up having to
get the hiera hash keys (titles) into suitable array syntax for Powershell comparison by:
a. adding single quotes and commas as separators
b. adding a single quote and double quote at the beginning and end of array
c. removing parenthesis from anywhere in the array as PowerShell spat it on them
$firewall_hiera = regsubst(regsubst(regsubst(join(keys($firewall_rules), "', '"), '^', '"\''), '$', '\'"'), '[\(\)]', '', 'G')
produces
"'Core Networking - DNS UDP-Out', 'Core Networking - Dynamic Host Configuration Protocol DHCP-Out', 'File and Printer Sharing Echo Request - ICMPv4-Out', 'Internet Browsing HTTP-Out', 'Internet Browsing HTTPS-Out'"
…
Use a Powershell exec to clean out the Firewall if undefined in Puppet hiera by:
a. using an exec command to purge the host firewall rules with:
command => 'netsh advfirewall firewall delete rule name=all; reg delete "HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules" /f',
b. ensuring idempotency with a very long unless statement:
unless => "if (#(Compare-Object((Get-NetFirewallRule | foreach {\"'\"+\$_.DisplayName+\"'\"}) -join ', ' -replace '[()]') $firewall_hiera).length -eq 0) {exit 0} else {exit 1}",
which produces for the following to compare-object with the above created $firewall_hiera
'Core Networking - DNS UDP-Out', 'Core Networking - Dynamic Host Configuration Protocol DHCP-Out', 'File and Printer Sharing Echo Request - ICMPv4-Out', 'Internet Browsing HTTP-Out', 'Internet Browsing HTTPS-Out'
Desired rules are then recreated (if a purge occurs) using the thoward-windows_firewall module which iterates over the same source hiera. This module is HEAPS better than the rather cr4ppy puppetlabs-windows_firewall module, but can't handle Windows 10's horrible firewall additives after user logon.
Interestingly, only the puppet created array needs outer double quotes. As the puppetlabs-powershell module doesn't echo the .ps1's to disk, I had to echo them out manually to get an idea of what was actually being created and test them in Powershell ISE (the older Josh Cooper powershell module did create temporary .ps1's which was handy though less secure)
I was able to put some regsubst or replace characters in arrays, but ^ and $ do not go well in an array.
All this was to solve Windows 10's incessant creation of additional pointless firewall rules only after a user logs on. This continues intermitently for about 6 puppet runs and finally stops. Never had this problem with Server 2012 R2 for example.
Deleting the following registry key (with Puppetlabs registry module ensure => absent) provides some additional quietening but its not an end in itself:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\RestrictedServices\AppIso\FirewallRules
Again, ANY assistance with shortening the above code, particularly with regards to the regsubst and -replace commands is appreciated. Keep in mind that the produced arrays need to be comparable using Powershell's 'compare-object' feature or similar.
Thanks to #John-Bollinger for getting me started with creating the array from a hiera hash using the keys and join functions.
Currently I'm using OpenDS and have to migrate to Active Directory (AD LDS).
I have a few custom attributes/objects that are defined in .ldif files in the OpenDS/config/schema directory like this:
attributeTypes: ( 1.3.6.1.4.1.99.1
NAME 'myNewAttribute'
DESC 'some text'
EQUALITY caseIgnoreMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
SINGLE-VALUE
)
objectClasses: ( 1.3.6.1.4.1.99.2
NAME 'myNewClass'
SUP top STRUCTURAL
MUST ( person $ myNewAttribute )
MAY someOtherAttribute
)
Unfortunately ldifde does not understand this format, so I used ADSI Edit to import my definitions manually one by one (cumbersome!) - but encountered some problems:
It seems AD supports only a handful of different SYNTAX definitions, like 2.5.5.12. How can I map something like 1.3.6.1.4.1.1466.115.121.1.15 to AD?
oMSyntax: Seems to be AD specific and not known to the rest of the LDAP world. oMSyntax in combination with attributeSyntax seems to define the data type in the AD world.
things like EQUALITY seem to be missing completely in AD! How to deal with this?
Question: Is there a tool to convert an LDIF file with attribute/object definitions to a format that is understood by MS / AD / ldifde?
Or a more general question: What is the best practice to migrate attribute/object definitions from OpenDS, OpenLDAP, etc. to the Microsoft world?
Welcome to the Diretories compatibility world. First of all the following syntax :
attributeTypes: ( 1.3.6.1.4.1.99.1
NAME 'myNewAttribute'
DESC 'some text'
EQUALITY caseIgnoreMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
SINGLE-VALUE
)
is not an LDIF syntax. it's a syntax used in Netscape like Directories to specify the Schema (OpenLDAP, Ex Sun directory service etc.). As far as you want to introduce new attributes and classes in Active Directory, you can do it using one of these 3 ways :
Manualy using the common ADSIEDIT.MSC (I never do it like that) :
Manualy using the specific Active Directory Scema Editor MMC (Microsoft Management Console)
This is the way I use in the development phase.
MMC.EXE -> File -> Open Component -> Active Directory Schema
If you are using an old server this MMC is only available recording an Active X component :
Regsvr32 c:\windows\system32\schmmgmt.msc
This way is much easier, it's the way I use on a development VM to create my attributes, I Then export the LDIF description with LDIFDE.EXE tool in order to clean it (remove system attributes) and import it on the production servers.
Programaticaly using LDIF
Here is an example of the LDIF syntax of an attribute
dn: CN=SlxChapitres,CN=Schema,CN=Configuration,DC=XXXX
changetype: add
objectClass: top
objectClass: attributeSchema
cn: SlxChapitres
distinguishedName: CN=SlxChapitres,CN=Schema,CN=Configuration,DC=XXXX
instanceType: 4
attributeID: 1.3.6.1.4.1.10558.2.1.6
attributeSyntax: 2.5.5.4
isSingleValued: FALSE
showInAdvancedViewOnly: TRUE
adminDisplayName: SlxChapitres
oMSyntax: 20
lDAPDisplayName: SlxChapitres
name: SlxChapitres
objectCategory: CN=Attribute-Schema,CN=Schema,CN=Configuration,DC=XXXX
This code is LDIF, I can inject it using LDIDE.EXE program the DC=XXXX syntax allowing me to use the -c DNSrc DNTarget of the LDIFFDE.EXE program option to locate it to the right DN.
As far as the Syntax and the matching rules are concerned, In my opinion Active-Directory is not so standard. Microsoft use a kind of combination between these to concepts to give one thing they call Syntax. Whenever you create a new attribute, you must specify its syntax. To uniquely identify the syntax among the total set of 21 syntaxes, you must specify 2 pieces of information: the OID of the syntax and a so-called OM syntax. This pair of values must be set together and correctly correlate with Mictosoft documention.