Loop and Array of Arrays in powershell - arrays

I was hoping someone could help me. I am new to powershell and struggling with trying to find the right way to approach something in my script. The script is to do a restore check on our backups using the veeam powershell commands. We have 7 backup jobs with various servers in each job and various drives being backed up in each job. Sometimes the servers are in multiple jobs as we have to split the drives across jobs as we copy the backup jobs to USB every day (so we have to balance out the amount of data we backup in each job so that we can copy the drives to the USB drives). I can write the entire script and get it to work but its around 800 lines long and is very inefficient although easy to understand for my team (and me later on!). My current approach pretty much revolves around performing certain actions for each backup job then each server within that job on each disk. I would like to cut it down using an array/loop. I have figured out how to use basic loops and arrays but I am struggling with being able to link the drives and servers in an array for example:
$Backupjob1 = "Backup Job 1E"
$Backupjob1Servers =#('Server1','Server2')
Somekind of array that allows different numbers of multiple drives for Server1 and Server 2.
$Backupjob1ServeDrives =#(Server1.Drive1 ='C', Server1.Drive2 ='F', Server2.Drive1 = 'C')
and then I need to loop through it so that on each loop, it performs an action on server1 and its first drive then does the next drive. Once server1 has finished, it performs the action on server2 on its first drive and then repeats on the second drive.
I understand the basics of looping through an array but I am struggling to understand how i would create an array of arrays that can deal with the above and then loop through it correctly.
I am stuck as not sure what array works and what options there are.

One way would be to externalise your config into a json file or similar and feed that to your script. Here is an example using a json string:
$serverConfig = #"
[{
"serverName": "server1",
"backupDrives": [
"D","F"
]
},
{
"serverName": "server2",
"backupDrives": [
"D","G","H"
]
}]
"#
$serverList = $serverConfig | ConvertFrom-Json
foreach($server in $serverList) {
Write-Host "Server: $($server.serverName)"
foreach($drive in $server.backupDrives) {
Write-Host "Backing up drive $drive..."
# Logic for the backup
}
}
This demonstrates handling the array of arrays you reference, you have an array of server objects and each server object has an array of drives, so on each iteration of the servers you iterate the server's drives too.
If you save the string into a config file your script could read that in, and then if you make any changes to the backup configuration you only need to change the config file and not the script.

It sounds like you want a dictionary type - thankfully PowerShell has a builtin unordered dictionary type called a hashtable that can be used.
In order to construct a hashtable literal, use #{...} instead of #(...):
$driveMapping = #{
Server1 = #{
Drive1 = 'C'
Drive2 = 'F'
}
Server2 = #{
Drive1 = 'C'
}
}
To traverse this data structure, use a simple nested loop:
foreach($server in $drivemapping.psbase.Keys){
Write-Host "About to backup the drives on server '$server'"
foreach($drive in $drivemapping[$server].psbase.Keys){
$driveLetter = $drivemapping[$server][$drive]
Write-Host "About to backup drive '$drive' with letter '$driveLetter' on server '$server'"
}
}

Related

Create a Powershell script to disable most of SQL server logins

First, and newbie question. Sorry in advance ;)
I'm trying to create a two step script in powershell to change all sql server logins on different servers.
My goal is to be able to run it in a few seconds and not having to access the management studio for this operations. This has to be done in about 1000 computers running different versions of SQL Express.
1st step: Change the SA password.
I've tested this one and it's ok. I'm using a txt file with the server name list and two string for the password and username. Works perfect.
2nd step: Disable all sql logins (including windows auth ones)that are not on text file with the list of logins to keep.
Here's where I'm stuck.
I can use a list of users and disable them but I would like to do the opposite. I want to disable login names that are not on the list but are enabled on the sql server.
I need it to work this way because I have no chance to know the hostnames and usernames without accessing our clients computers / servers and I need it to be done really quick without manual procedures.
This is what I've done to be able to disable the ones on the list. It works fine but... I need the opposite.
Thank you very much and sorry if it's a stupid question.
Regards,
Luís
#DISABLE LOGINS ON THE LIST
[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO') | out-null
$Error.Clear()
cls
$servers = Get-Content C:\example\servers.txt
$logins = Get-Content C:\example\users.txt
foreach($server in $servers)
{
$srv = New-Object ('Microsoft.SqlServer.Management.Smo.Server') $server
foreach($login in $logins)
{
if ($srv.Logins.Contains($login))
{
$srv.Logins[$login].disable();
}
}
}
$srv.Logins |
Where-Object Name -notin $logins |
ForEach-Object Disable
Note that the lookup isn't efficient, because a linear search in array $logins is performed for each login, but that probably won't matter in practice.
The Where-Object and ForEach-Object calls use simplified syntax.

Compare Array of IPs with rsyslog

There is a list of IPs (their quantity is changing every few minutes) stored in a log-file. The logfile only contains unique IPs (sort -u):
IP-Pool.log [Example]:
192.168.1.1
192.168.1.2
10.1.1.0
.
.
I would like to filter occuring connection events with rsyslog and sort them by:
Events where $msg contains one IP of Array[IP-Pool-List]
Events where $msg doesn't contain any IP of Array[IP-Pool-List]
Therefore I would have to put the IPs into an array so they can be compared but I haven't found a way to do so.
if $msg contains 'connection-event-id' then {
if $msg contains 'Array[IP-Pool-List] then {
-/var/log/event_contains_ip.log
STOP
}
else{
-/var/log/event_doesnt_contain_ip.log
STOP
}
}
Would I also have to restart the rsyslogd service in order to refresh the array [IP-Pool-List], since the file is changing every few minutes ?
Is there a way to solve this with rsyslog, or is there an easier/better way to do this ?
EDIT
I used crontab and inotifywait to keep the IP-Pool-File up to date.
#reboot inotifywait -q -m -e modify /var/log/client-ips.log |
while read -r filename events; do sort -u /var/log/client-ips.log
-o /var/log/ips-unique.log ;done
I also asked this question via mail to the "rsyslog-community" their contact mail is listed at the bottom of the rsyslog documentation (https://lists.adiscon.net/mailman/listinfo/rsyslog)
They gave me the advise to use "Lookup-Tables" to solve this. Currently reading how to use them correctly. Since I am a slow learner, this should take a while.
EDIT 2
A new plan is starting to form:
To get Lookup-Table going, I need to create a JSON-File, which
contains the values I am going to build my filter with
Update the JSON File with "reload_lookup_table" or "JQ", whenever a new IP is added to the
IP-Pool-List.
Use RSYSLOG Lookup-Table to filter for the given IPs and split the
results into two seperate log files | Match.log | NoMatch.log

Puppet - build array from hash titles to check contents in exec

I am trying to apply the thoward-windows_firewall module to Windows 10 and hit a snag whereby most of the built in rules can be purged with the module's purge function, but some others (such as cortana) cannot. This problem didn't occur in earlier Windows builds. The workaround to this seems to be to run an exec with 'netsh advfirewall reset' which purges them and stops them reappearing.
However, I only want to apply the 'netsh advfirewall reset' exec if firewall rules other than what I have specified in Hiera exist.
My approach is to build an array (or list) of only the titles in hiera and then iterate over them in a PowerShell 'only if' or 'unless'.
Code so far (which is not working) is:
Hiera snippet:
harden::firewall:
'Remote Desktop - User Mode (UDP-In)':
description: 'Inbound rule for the Remote Desktop service to allow RDP traffic. [UDP 3389]'
application_name: 'C:\Windows\system32\svchost.exe'
service_name: 'termservice'
protocol: 17
local_ports: '3389'
remote_ports: '*'
local_addresses: '%{facts.networking.ip}'
remote_addresses: '*'
direction: 1
interface_types: 'All'
enabled: true
grouping: '#FirewallAPI.dll,-28752'
profiles: 3
action: 1
'File and Printer Sharing (Echo Request - ICMPv4-In) 1':
description: 'Echo Request messages are sent as ping requests to other nodes.'
protocol: 1
local_addresses: '%{facts.networking.ip}'
remote_addresses: 'LocalSubnet'
icmp_types_and_codes: '8:*'
direction: 1
interface_types: 'All'
enabled: true
grouping: '#FirewallAPI.dll,-28502'
profiles: 6
action: 1
Manifest (extract):
class harden (
Hash $firewall_rule_names = lookup('harden::firewall'),
){
# reset firewall rules
exec { 'reset_firewall':
command => 'netsh advfirewall reset',
onlyif => 'if (Get-NetFirewallRule | where {\$_.DisplayName -notmatch $firewall_rule_names}) { exit 0 } else { exit 1 }',
provider => powershell,
}
Class { 'windows_firewall':
profile_state => 'on',
in_policy => 'BlockInbound',
out_policy => 'BlockOutbound',
rule_key => 'harden::firewall',
purge_rules => true,
}
I know I need to have a .each look in there somewhere, and also tidy up the powershell 'only if' so that it looks at just the titles in the hash (perhaps re-written to an array of just hash titles) and runs the exec if there are rules on the host that aren't in Hiera, but am getting a bit lost.
Any help sincerely appreciated.
You wrote:
My approach is to build an array (or list) of only the titles in hiera
and then iterate over them in a PowerShell 'only if' or 'unless'.
Apparently, you attempt to build that array in class parameter $firewall_rule_names, with this declaration:
Hash $firewall_rule_names = lookup('harden::firewall'),
and then you attempt to use that list in the onlyif parameter of your Exec:
onlyif => 'if (Get-NetFirewallRule | where {\$_.DisplayName -notmatch $firewall_rule_names}) { exit 0 } else { exit 1 }',
There are multiple problems with that.
First, if you want to interpolate a puppet variable into a string, then that string needs to be quoted with double quotes ("); quoting with single quotes (') suppresses interpolation (and also treats \$ as two literal characters, not an escape sequence).
Second, and where you seem to be at loss, as how to extract the keys of your $firewall_rule_names hash and format them appropriately. I'm uncertain exactly what Powershell requires here, but some of the best tools to use to get it would be the keys() and join() functions provided by the puppetlabs/stdlib module or by Puppet itself if you're using version 5.5 or later. For example, if all you need is a comma-delimited list of the names then something like this would do it:
$really_the_rule_names = join(keys($firewall_rule_names), ', ')
I'm suspicious, though, that you may need to quote the keys. You could get most of that by being clever with the delimiter you specify to join, but you might also want to consider processing the key array with the built-in regsubst() function before joining results together into a string.
I can see a problem in the Only IF statement.
$_DisplayName should be $_.DisplayName , you need to use Dot operator to cherry pick the properties of an Object and \ is also not required.
#John-Bollinger - Puppet and PowerShell = PuppetHell! :)
I finally managed to solve this problem but feel that my code blocks are way to long with too many substitutions. I've tried shorter methods to no avail, so if anyone can suggest working shorter code that would be appreciated.
In short, I ended up having to
get the hiera hash keys (titles) into suitable array syntax for Powershell comparison by:
a. adding single quotes and commas as separators
b. adding a single quote and double quote at the beginning and end of array
c. removing parenthesis from anywhere in the array as PowerShell spat it on them
$firewall_hiera = regsubst(regsubst(regsubst(join(keys($firewall_rules), "', '"), '^', '"\''), '$', '\'"'), '[\(\)]', '', 'G')
produces
"'Core Networking - DNS UDP-Out', 'Core Networking - Dynamic Host Configuration Protocol DHCP-Out', 'File and Printer Sharing Echo Request - ICMPv4-Out', 'Internet Browsing HTTP-Out', 'Internet Browsing HTTPS-Out'"
…
Use a Powershell exec to clean out the Firewall if undefined in Puppet hiera by:
a. using an exec command to purge the host firewall rules with:
command => 'netsh advfirewall firewall delete rule name=all; reg delete "HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules" /f',
b. ensuring idempotency with a very long unless statement:
unless => "if (#(Compare-Object((Get-NetFirewallRule | foreach {\"'\"+\$_.DisplayName+\"'\"}) -join ', ' -replace '[()]') $firewall_hiera).length -eq 0) {exit 0} else {exit 1}",
which produces for the following to compare-object with the above created $firewall_hiera
'Core Networking - DNS UDP-Out', 'Core Networking - Dynamic Host Configuration Protocol DHCP-Out', 'File and Printer Sharing Echo Request - ICMPv4-Out', 'Internet Browsing HTTP-Out', 'Internet Browsing HTTPS-Out'
Desired rules are then recreated (if a purge occurs) using the thoward-windows_firewall module which iterates over the same source hiera. This module is HEAPS better than the rather cr4ppy puppetlabs-windows_firewall module, but can't handle Windows 10's horrible firewall additives after user logon.
Interestingly, only the puppet created array needs outer double quotes. As the puppetlabs-powershell module doesn't echo the .ps1's to disk, I had to echo them out manually to get an idea of what was actually being created and test them in Powershell ISE (the older Josh Cooper powershell module did create temporary .ps1's which was handy though less secure)
I was able to put some regsubst or replace characters in arrays, but ^ and $ do not go well in an array.
All this was to solve Windows 10's incessant creation of additional pointless firewall rules only after a user logs on. This continues intermitently for about 6 puppet runs and finally stops. Never had this problem with Server 2012 R2 for example.
Deleting the following registry key (with Puppetlabs registry module ensure => absent) provides some additional quietening but its not an end in itself:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\RestrictedServices\AppIso\FirewallRules
Again, ANY assistance with shortening the above code, particularly with regards to the regsubst and -replace commands is appreciated. Keep in mind that the produced arrays need to be comparable using Powershell's 'compare-object' feature or similar.
Thanks to #John-Bollinger for getting me started with creating the array from a hiera hash using the keys and join functions.

How to process SSAS cube structures concurrently from PowerShell script using Mircrosoft.AnalysisServices namespace

TL;DR: Is there a way for a PowerShell script calling Microsoft.AnalysisServices functions to process multiple cube structures concurrently?
I have a Microsoft SSAS cube that needs several measure groups processed before the rest of the cube is processed later in the job plan. I have created a PowerShell script that enumerates the measure groups to process and calls measureGroup.Process('ProcessFull') from the Microsoft.AnalysisServices namespace. This works to process the measure group, dimension, et.al.
However, processing the measure groups in this manner doesn't allows SQL Server 2014 to parallelize the processing. A cube that takes on average 2 hours to fully process was running for 7 hours before we killed it.
Is there a way in PowerShell to batch the processes so that they are sent at the same time to the cube? If this could be done, it would allow the server to do concurrent processing instead of one object at a time.
I have taken a look through the documentation on the MSDN as well as consulted Google, but was unable to find an answer. Thanks!
You should check this method in PowerShell: http://ssas-info.com/analysis-services-scripts/1238-powershell-script-to-process-all-dimensions-and-cubes-in-one-db-limiting-workload
Have a look at powershell jobs:
Background Jobs
Here's a quick example that you could adapt to run your measuregroup processing:
$cmd = {
param($a)
Write-Host $a
}
$list = #("a","b","c")
$list | ForEach-Object {
Start-Job -ScriptBlock $cmd -ArgumentList $_
}
It's quite simple, you define your script block in the $cmd variable, this is where you would put your logic around processing the measure group. The $list variable could contain a list of the measure groups to process.
You then start a job for each item in the list, which executes the code in the script block, passing through the item in the list as a parameter. In this example it simply prints out the parameter that you passed in. You can of course pass in as many parameters as you like.
To get the results, you can use the Get-Job cmdlet to check the status of the jobs and Receive-Job to get the output. Remove-Job can then be used to clear finished jobs from the queue.
The following command run after the code above will get the results of all the jobs (in this case just the a,b,c that we passed in and then will remove it from the queue:
Get-Job | ForEach-Object { Receive-Job $_.Id; Remove-Job $_.Id }

Biztalk suspended messages in database

I was wondering if someone knows where I can see the data of a suspended message in the biztalk database.
I need this because about 900 messages have been suspended because of a validation and I need to edit all of them, resuming isn't possible.
I know that info of suspended messages are shown in BizTalkMsgBoxDb in the table InstancesSuspended and that the different parts of each message are shown in the table MessageParts. However I can't find the table where the actual data is stored.
Does anyone have any idea where this can be done?
I found a way to do this, there's no screwing up my system when I just want to read them.
How I did it is using the method "CompressionStreams" using Microsoft.Biztalk.Pipeline.dll.
The method to do this:
public static Stream getMsgStrm(Stream stream)
{
Assembly pipelineAssembly = Assembly.LoadFrom(string.Concat(#"<path to dll>", #"\Microsoft.BizTalk.Pipeline.dll"));
Type compressionStreamsType = pipelineAssembly.GetType("Microsoft.BizTalk.Message.Interop.CompressionStreams", true);
return (Stream)compressionStreamsType.InvokeMember("Decompress", BindingFlags.Public | BindingFlags.InvokeMethod | BindingFlags.Static, null, null, new object[] { (object)stream });
}
Then I connect with my database, fill in a dataset and stream out the data to string, code:
String SelectCmdString = "select * from dbo.Parts";
SqlDataAdapter mySqlDataAdapter = new SqlDataAdapter(SelectCmdString, "<your connectionstring">);
DataSet myDataSet = new DataSet();
mySqlDataAdapter.Fill(myDataSet, "BodyParts");
foreach (DataRow row in myDataSet.Tables["BodyParts"].Rows)
{
if (row["imgPart"].GetType() != typeof(DBNull))
{
SqlBinary binData = new SqlBinary((byte[])row["imgPart"]);
MemoryStream stm = new MemoryStream(binData.Value);
Stream aStream = getMsgStrm(stm);
StreamReader aReader = new StreamReader(aStream);
string aMessage = aReader.ReadToEnd();
//filter msg
//write msg
}
}
I then write each string to an appropriate "txt" or "xml" depending on what u want, you can also filter out certain messages with regular expression, etc.
Hope this helps anyone, it sure as hell helped me.
Greetings
Extract Messages from suspended instances
Scenario:
BizTalk 2010 and SQL 2008 R2 is the environment we have used fore this scenario.
You have problem with some integrations, 1500 suspended instances inside BizTalk and you need to send the actual messages to a customer, and then you properly do not want to manually save out this from BizTalk Administrator.
There are a lot of blogs and Internet resources pointing out vbs, powershell scripts how to do this, but I have used BizTalk Terminator to solve this kind of scenarios.
As you now BizTalk terminator is asking you 3 questions when the tool starts
I.1.All BizTalk databases are backed up?
II.2.All Host Instances is stopped?
III.3.All BizTalk SQL Agents is stopped?
This is ok when you are going to actually change something inside BizTalk databases but this is not what you are going to do in this scenario you are only using the tool to read from BizTalk databases. But you should always have backups off BizTalk databases.
You are always responsible for what you are doing, but when we have used this tools in the way I describe we have not have any problem with this scenario.
So after you have start Terminator tool please click yes to the 3 questions(you dont need to stop anything in this scenario) then connect to the correct environment please do this in your test environment first so you feel comfortable with this scenario, the next step is to choose a terminator task choose Count Instances(and save messages) after this you have to fill in the parameter TAB with correct serviceClass and Hostname and set SaveMessages to True and last set FilesaveFullPath to the correct folder you want to save the messages to.
Then you can choose to click on the Execute Button and depending of the size and how many it can take some time, after this disconnect Terminator do NOT do anything else.
You should now if you have filled in the correct values in the parameter TAB have the saved messages inside the FilesaveFullPath folder.
Download BizTalk terminator from this address:
http://www.microsoft.com/en-us/download/details.aspx?id=2846
This is more than likely not supported by Microsoft. Don't risk screwing up your system. If you have a need to have a edit and resubmit, it needs to be built into the orchestration. Otherwise, your best bet is to use WMI to write a script to:
pull out all of the suspended messages
terminate them
edit them
resubmit them
you can find it through the HAT tool you just need to specify the schema ,port and the exact date
with the exact time and it will show you the messages right click on the desired one and save .

Resources