Puppet - build array from hash titles to check contents in exec - arrays

I am trying to apply the thoward-windows_firewall module to Windows 10 and hit a snag whereby most of the built in rules can be purged with the module's purge function, but some others (such as cortana) cannot. This problem didn't occur in earlier Windows builds. The workaround to this seems to be to run an exec with 'netsh advfirewall reset' which purges them and stops them reappearing.
However, I only want to apply the 'netsh advfirewall reset' exec if firewall rules other than what I have specified in Hiera exist.
My approach is to build an array (or list) of only the titles in hiera and then iterate over them in a PowerShell 'only if' or 'unless'.
Code so far (which is not working) is:
Hiera snippet:
harden::firewall:
'Remote Desktop - User Mode (UDP-In)':
description: 'Inbound rule for the Remote Desktop service to allow RDP traffic. [UDP 3389]'
application_name: 'C:\Windows\system32\svchost.exe'
service_name: 'termservice'
protocol: 17
local_ports: '3389'
remote_ports: '*'
local_addresses: '%{facts.networking.ip}'
remote_addresses: '*'
direction: 1
interface_types: 'All'
enabled: true
grouping: '#FirewallAPI.dll,-28752'
profiles: 3
action: 1
'File and Printer Sharing (Echo Request - ICMPv4-In) 1':
description: 'Echo Request messages are sent as ping requests to other nodes.'
protocol: 1
local_addresses: '%{facts.networking.ip}'
remote_addresses: 'LocalSubnet'
icmp_types_and_codes: '8:*'
direction: 1
interface_types: 'All'
enabled: true
grouping: '#FirewallAPI.dll,-28502'
profiles: 6
action: 1
Manifest (extract):
class harden (
Hash $firewall_rule_names = lookup('harden::firewall'),
){
# reset firewall rules
exec { 'reset_firewall':
command => 'netsh advfirewall reset',
onlyif => 'if (Get-NetFirewallRule | where {\$_.DisplayName -notmatch $firewall_rule_names}) { exit 0 } else { exit 1 }',
provider => powershell,
}
Class { 'windows_firewall':
profile_state => 'on',
in_policy => 'BlockInbound',
out_policy => 'BlockOutbound',
rule_key => 'harden::firewall',
purge_rules => true,
}
I know I need to have a .each look in there somewhere, and also tidy up the powershell 'only if' so that it looks at just the titles in the hash (perhaps re-written to an array of just hash titles) and runs the exec if there are rules on the host that aren't in Hiera, but am getting a bit lost.
Any help sincerely appreciated.

You wrote:
My approach is to build an array (or list) of only the titles in hiera
and then iterate over them in a PowerShell 'only if' or 'unless'.
Apparently, you attempt to build that array in class parameter $firewall_rule_names, with this declaration:
Hash $firewall_rule_names = lookup('harden::firewall'),
and then you attempt to use that list in the onlyif parameter of your Exec:
onlyif => 'if (Get-NetFirewallRule | where {\$_.DisplayName -notmatch $firewall_rule_names}) { exit 0 } else { exit 1 }',
There are multiple problems with that.
First, if you want to interpolate a puppet variable into a string, then that string needs to be quoted with double quotes ("); quoting with single quotes (') suppresses interpolation (and also treats \$ as two literal characters, not an escape sequence).
Second, and where you seem to be at loss, as how to extract the keys of your $firewall_rule_names hash and format them appropriately. I'm uncertain exactly what Powershell requires here, but some of the best tools to use to get it would be the keys() and join() functions provided by the puppetlabs/stdlib module or by Puppet itself if you're using version 5.5 or later. For example, if all you need is a comma-delimited list of the names then something like this would do it:
$really_the_rule_names = join(keys($firewall_rule_names), ', ')
I'm suspicious, though, that you may need to quote the keys. You could get most of that by being clever with the delimiter you specify to join, but you might also want to consider processing the key array with the built-in regsubst() function before joining results together into a string.

I can see a problem in the Only IF statement.
$_DisplayName should be $_.DisplayName , you need to use Dot operator to cherry pick the properties of an Object and \ is also not required.

#John-Bollinger - Puppet and PowerShell = PuppetHell! :)
I finally managed to solve this problem but feel that my code blocks are way to long with too many substitutions. I've tried shorter methods to no avail, so if anyone can suggest working shorter code that would be appreciated.
In short, I ended up having to
get the hiera hash keys (titles) into suitable array syntax for Powershell comparison by:
a. adding single quotes and commas as separators
b. adding a single quote and double quote at the beginning and end of array
c. removing parenthesis from anywhere in the array as PowerShell spat it on them
$firewall_hiera = regsubst(regsubst(regsubst(join(keys($firewall_rules), "', '"), '^', '"\''), '$', '\'"'), '[\(\)]', '', 'G')
produces
"'Core Networking - DNS UDP-Out', 'Core Networking - Dynamic Host Configuration Protocol DHCP-Out', 'File and Printer Sharing Echo Request - ICMPv4-Out', 'Internet Browsing HTTP-Out', 'Internet Browsing HTTPS-Out'"
…
Use a Powershell exec to clean out the Firewall if undefined in Puppet hiera by:
a. using an exec command to purge the host firewall rules with:
command => 'netsh advfirewall firewall delete rule name=all; reg delete "HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules" /f',
b. ensuring idempotency with a very long unless statement:
unless => "if (#(Compare-Object((Get-NetFirewallRule | foreach {\"'\"+\$_.DisplayName+\"'\"}) -join ', ' -replace '[()]') $firewall_hiera).length -eq 0) {exit 0} else {exit 1}",
which produces for the following to compare-object with the above created $firewall_hiera
'Core Networking - DNS UDP-Out', 'Core Networking - Dynamic Host Configuration Protocol DHCP-Out', 'File and Printer Sharing Echo Request - ICMPv4-Out', 'Internet Browsing HTTP-Out', 'Internet Browsing HTTPS-Out'
Desired rules are then recreated (if a purge occurs) using the thoward-windows_firewall module which iterates over the same source hiera. This module is HEAPS better than the rather cr4ppy puppetlabs-windows_firewall module, but can't handle Windows 10's horrible firewall additives after user logon.
Interestingly, only the puppet created array needs outer double quotes. As the puppetlabs-powershell module doesn't echo the .ps1's to disk, I had to echo them out manually to get an idea of what was actually being created and test them in Powershell ISE (the older Josh Cooper powershell module did create temporary .ps1's which was handy though less secure)
I was able to put some regsubst or replace characters in arrays, but ^ and $ do not go well in an array.
All this was to solve Windows 10's incessant creation of additional pointless firewall rules only after a user logs on. This continues intermitently for about 6 puppet runs and finally stops. Never had this problem with Server 2012 R2 for example.
Deleting the following registry key (with Puppetlabs registry module ensure => absent) provides some additional quietening but its not an end in itself:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\RestrictedServices\AppIso\FirewallRules
Again, ANY assistance with shortening the above code, particularly with regards to the regsubst and -replace commands is appreciated. Keep in mind that the produced arrays need to be comparable using Powershell's 'compare-object' feature or similar.
Thanks to #John-Bollinger for getting me started with creating the array from a hiera hash using the keys and join functions.

Related

<MSDialect_pyodbc, TIMESTAMP> error when comparing tables in DVT - BigQuery and SQL Server

I'm trying to compare the table data between two databases (on-prem SQL Server and BigQuery). I'm currently using Data-Validation-Tool for that (DVT).
Using the instructions from Github (link: https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/docs), I installed and created a connection for DVT.
When tried to compare two table from single source, it works fine and provide correct output. But when checked for two different source, it returns dtype: <MSDialect_pyodbc, TIMESTAMP> error.
Details:
I tried to validate over level count but still same error. Command -
data-validation validate column -sc wc_sql_conn -tc my_bq_conn -tbls EDW.dbo.TableName=project_id.dataset_name.TableName --primary-keys PK_Col_Name --count '*'
Also, When checked for schema level validation -
data-validation validate schema -sc wc_sql_conn -tc my_bq_conn -tbls EDW.dbo.TableName=project_id.dataset_name.TableName
I tried to add only INT/STRING columns in custom query and then comparing it.
Custom Query -
SELECT PK_Col_Name, Category FROM EDW.dbo.TableName
Similar custom query prepared for BQ and command for custom query comparison -
data-validation validate custom-query -sc wc_sql_conn -tc my_bq_conn -cqt 'column' -sqf sql_query.txt -tqf bq_query.txt -pk PK_Col_Name --count '*'
data-validation validate custom-query -sc wc_sql_conn -tc my_bq_conn -cqt 'row' -sqf sql_query.txt -tqf bq_query.txt -pk PK_Col_Name --count '*'
Even when using multiple approaches and the scenario doesn't involve a datetime/timestamp column, I typically only get one error -
NotImplementedError: Could not find signature for dtype: <MSDialect_pyodbc, TIMESTAMP>
I tried to google down the error, but no luck. Could someone please help me identify the error?
Additionally, there are no data-validation-tool or google-pso-data-validator tags available. If someone could add that, it may be used in the future and reach the right people.

Loop and Array of Arrays in powershell

I was hoping someone could help me. I am new to powershell and struggling with trying to find the right way to approach something in my script. The script is to do a restore check on our backups using the veeam powershell commands. We have 7 backup jobs with various servers in each job and various drives being backed up in each job. Sometimes the servers are in multiple jobs as we have to split the drives across jobs as we copy the backup jobs to USB every day (so we have to balance out the amount of data we backup in each job so that we can copy the drives to the USB drives). I can write the entire script and get it to work but its around 800 lines long and is very inefficient although easy to understand for my team (and me later on!). My current approach pretty much revolves around performing certain actions for each backup job then each server within that job on each disk. I would like to cut it down using an array/loop. I have figured out how to use basic loops and arrays but I am struggling with being able to link the drives and servers in an array for example:
$Backupjob1 = "Backup Job 1E"
$Backupjob1Servers =#('Server1','Server2')
Somekind of array that allows different numbers of multiple drives for Server1 and Server 2.
$Backupjob1ServeDrives =#(Server1.Drive1 ='C', Server1.Drive2 ='F', Server2.Drive1 = 'C')
and then I need to loop through it so that on each loop, it performs an action on server1 and its first drive then does the next drive. Once server1 has finished, it performs the action on server2 on its first drive and then repeats on the second drive.
I understand the basics of looping through an array but I am struggling to understand how i would create an array of arrays that can deal with the above and then loop through it correctly.
I am stuck as not sure what array works and what options there are.
One way would be to externalise your config into a json file or similar and feed that to your script. Here is an example using a json string:
$serverConfig = #"
[{
"serverName": "server1",
"backupDrives": [
"D","F"
]
},
{
"serverName": "server2",
"backupDrives": [
"D","G","H"
]
}]
"#
$serverList = $serverConfig | ConvertFrom-Json
foreach($server in $serverList) {
Write-Host "Server: $($server.serverName)"
foreach($drive in $server.backupDrives) {
Write-Host "Backing up drive $drive..."
# Logic for the backup
}
}
This demonstrates handling the array of arrays you reference, you have an array of server objects and each server object has an array of drives, so on each iteration of the servers you iterate the server's drives too.
If you save the string into a config file your script could read that in, and then if you make any changes to the backup configuration you only need to change the config file and not the script.
It sounds like you want a dictionary type - thankfully PowerShell has a builtin unordered dictionary type called a hashtable that can be used.
In order to construct a hashtable literal, use #{...} instead of #(...):
$driveMapping = #{
Server1 = #{
Drive1 = 'C'
Drive2 = 'F'
}
Server2 = #{
Drive1 = 'C'
}
}
To traverse this data structure, use a simple nested loop:
foreach($server in $drivemapping.psbase.Keys){
Write-Host "About to backup the drives on server '$server'"
foreach($drive in $drivemapping[$server].psbase.Keys){
$driveLetter = $drivemapping[$server][$drive]
Write-Host "About to backup drive '$drive' with letter '$driveLetter' on server '$server'"
}
}

Compare Array of IPs with rsyslog

There is a list of IPs (their quantity is changing every few minutes) stored in a log-file. The logfile only contains unique IPs (sort -u):
IP-Pool.log [Example]:
192.168.1.1
192.168.1.2
10.1.1.0
.
.
I would like to filter occuring connection events with rsyslog and sort them by:
Events where $msg contains one IP of Array[IP-Pool-List]
Events where $msg doesn't contain any IP of Array[IP-Pool-List]
Therefore I would have to put the IPs into an array so they can be compared but I haven't found a way to do so.
if $msg contains 'connection-event-id' then {
if $msg contains 'Array[IP-Pool-List] then {
-/var/log/event_contains_ip.log
STOP
}
else{
-/var/log/event_doesnt_contain_ip.log
STOP
}
}
Would I also have to restart the rsyslogd service in order to refresh the array [IP-Pool-List], since the file is changing every few minutes ?
Is there a way to solve this with rsyslog, or is there an easier/better way to do this ?
EDIT
I used crontab and inotifywait to keep the IP-Pool-File up to date.
#reboot inotifywait -q -m -e modify /var/log/client-ips.log |
while read -r filename events; do sort -u /var/log/client-ips.log
-o /var/log/ips-unique.log ;done
I also asked this question via mail to the "rsyslog-community" their contact mail is listed at the bottom of the rsyslog documentation (https://lists.adiscon.net/mailman/listinfo/rsyslog)
They gave me the advise to use "Lookup-Tables" to solve this. Currently reading how to use them correctly. Since I am a slow learner, this should take a while.
EDIT 2
A new plan is starting to form:
To get Lookup-Table going, I need to create a JSON-File, which
contains the values I am going to build my filter with
Update the JSON File with "reload_lookup_table" or "JQ", whenever a new IP is added to the
IP-Pool-List.
Use RSYSLOG Lookup-Table to filter for the given IPs and split the
results into two seperate log files | Match.log | NoMatch.log

Determine the port that SQL is using or set it if needed via program

1) I'm looking for a way within a VB.Net/ADO.Net program to determine if and what port an named instance of Microsoft SQL is using without resorting to a registry key (too many different places to look), hacked event log (unreliable), or opening SQL Configuration Manager (again, I need to do this is a program).
2) If I determine that SQL is not configured to use TCP then is there a reliable way through VB.Net/ADO.Net to enable TCP/IP and set a port?
In both cases these functions would be running with elevated privileges during a setup.
I ply my trade in powershell, but you should be able to translate this pretty well. NB: in my testing, the hash keys (e.g. 'MSSQLSERVER', 'Tcp') were case-sensitive.
$machine = new-object Microsoft.SqlServer.Management.Smo.Wmi.ManagedComputer '.'
$t = $machine.ServerInstances['MSSQLSERVER'].ServerProtocols['Tcp']
$t.IsEnabled = $true; #enable TCP
$port = $t.IPAddresses['IPAll'].IPAddressProperties['TcpPort'];
if ($port.Value -ne '1433') {
$port.Value = '1434'; #or whatever port you want to set it to
}
$t.alter();

CakePHP ODBC driver connecting to Microsoft SQL Server; how to remove backticks?

I've got a CakePHP application connecting to a remote MSSQL server through ODBC, but it's not working as planned. Every query dies becasue it is trying to put backticks around identifiers, which is not correct for MSSQL.
As an example, I have a model called Item for a table called items, and when I call
$this->Item->find('all')
it tries to use the query
SELECT `Item`.`id`, `Item`.`name`, `Item`.`descrtiption` FROM `items` AS `Item` WHERE 1
...and I get an error about invalid syntax near ` at line 1.
Is there anyway to prevent this behaviour and remove the backticks? Or else use square brackets like SQL Server seems to like?
I recently took a good look around the odbc driver with the intention of using it against MSSQL 2008 in CakePHP 1.3. Unless you are prepared to put a considerable amount of work in then it's not feasible at present.
Your immediate problem is that you need to override the default quotes with [ and ]. These are set at the top of the dbo_odbc.php file here
var $startQuote = "[";
var $endQuote = "]";
Once you do this the next issue you will run into is the default use of LIMIT, so you'll need to provide your own limiting function copied from dbo_mssq to override
/**
* Returns a limit statement in the correct format for the particular database.
*
* #param integer $limit Limit of results returned
* #param integer $offset Offset from which to start results
* #return string SQL limit/offset statement
*/
function limit($limit, $offset = null) {
if ($limit) {
$rt = '';
if (!strpos(strtolower($limit), 'top') || strpos(strtolower($limit), 'top') === 0) {
$rt = ' TOP';
}
$rt .= ' ' . $limit;
if (is_int($offset) && $offset > 0) {
$rt .= ' OFFSET ' . $offset;
}
return $rt;
}
return null;
}
You'll then run into two problems, neither of which I solved.
In the describe function the odbc_field_type call is not returning a
field type. I'm not sure how critical this is if you describe the fields in the model, but it doesn't sound promising.
More crucially, in the fields function that's used to generate a field list cake works by recursively exploding the . syntax to generate a series of AS aliases. This is fine if you're recursion level is zero, but with deeper recursion you end up with a field list that looks something like 'this.that.other AS this_dot_that.other AS this_dot_that_dot_other', which is invalid MSSQL syntax.
Neither of these are unsolvable, but at this point I decided it was simpler to reload my server and use the MSSQL driver than continue to chase prblems with the ODBC driver, but YMMV
ADDED: This question seems to be getting a bit of attention: so anyone who takes this further could they append their code to this answer - and hopefully we can assemble a solution between us.
why dont you just use the mssql dbo https://github.com/cakephp/cakephp/blob/master/cake/libs/model/datasources/dbo/dbo_mssql.php

Resources