Array not creating correctly in PowerShell - arrays

I have a script where I get all of the folders in d:\folder\*\*\ where the name is -like "*\Log". I then split the folder paths apart to run through wmi to get the corresponding services. After that I'm wanting to split apart the PathName property from $Services so I get everything before the \xxxxx.exe and add \log to the end of the result. Eventually I'll then use those paths to do some compression and archiving of files via a gci.
For whatever reason when I run the script below I the previous loops $LocalLogVar without "log" appended and the current loops LocalLogVar with log appended. I'm sure I'm doing something wrong that's blatantly obvious to somebody out there. If somebody could point me in the right direction on this it'd be much appreciated! I also apologize for the word vomit here, I've been looking at this script all day and my brain's pretty much used up.
A couple of notes:
The number of words in the paths vary which is why I can't manually do $LocalLogVar = "$Var1\$Var2\$Var3\Log"
If I don't have the [array] in front of $LogFolders the object type becomes a string and I get the previous loop's $LocalLogVar without "log" appended combined with the current loop's $LocalLogVar
I tried doing [collections.arraylist]$LogFolders=#() with no success
c:\folder is a shortcut to d:\folder, which is why there's c:\folder\xxx and d:\folder\xxx in the list below
SplitCount is -1 because I don't want the .exe from the path, I just want the folder structure
The naming convention for the string before .exe varies so I can't use an enumerated counter.
Example of first bullet:
word7-word8 #This is the previous loop's $LocalLogVar w/o "log" appended
C:\folder\word5\word6\word9-word8\log #This is the current loop's $LocalLogVar w/ "log" appended.
Example of the second bullet:
word7-word8C:\folder\word5\word6\word9-word8\log
What I should be getting:
D:\folder\word-anotherword\word7-word8\log
D:\folder\word-anotherword\word9-word8\log
C:\folder\word1\word7-word8\log
C:\folder\word1\word9-word8\log
C:\folder\word2\word7-word8\log
C:\folder\word2\word9-word8\log
D:\folder\word2\word10-word11\log
D:\folder\word2\word12-word8\log
C:\folder\word3\word7-word8\log
C:\folder\word3\word9-word8\log
D:\folder\word4\word7-word8\log
C:\folder\word4\word9-word8\log
C:\folder\word5\word6\word7-word8\log
C:\folder\word5\word6\word9-word8\log
C:\folder\word5\word6\word7-word8\log
C:\folder\word5\word6\word9-word8\log
$Folders = Get-ChildItem D:\folder\*\*\ -Directory -Recurse -Verbose `
| Where-Object { $_.fullname -like "*\Log" }
$2 = #()
$LogFolders = #()
foreach ($folder in $folders) {
$ServName = $folder.fullname.split('\')[2]
$ServType = $folder.fullname.split('\')[3]
$ServNameCheck = "*$($ServName.replace('-',' '))*"
$ServTypeCheck = "*$($ServType.replace('-',' '))*"
$PathName = Get-WmiObject -ClassName Win32_Service `
| Where-Object { $_.caption -like "$ServNameCheck" -and $_.caption -like "$ServTypeCheck" } `
| Select-Object Name, Caption, #{n = 'PathName'; e = { ($_.PathName).trim('"') } }
$2 += $PathName
}
$Services = $2 | Sort-Object pathname | Get-Unique -AsString
foreach ($ServPath in $services.pathname) {
$LocalLogVar = #()
if (Get-Variable `
| Where-Object { $_.name -match "^Split([0-9]|10)$" }) {
Get-Variable `
| Where-Object { $_.name -match "^Split([0-9]|10)$" } | Remove-Variable -Force
}
[int]$SplitCount = $ServPath.split('\').count
[int]$SplitCountCheck = $SplitCount - 1
$x = 0
do {
New-Variable -Name "Split$x" -Value "$($ServPath.split('\')[$x])"
$RegEx = "Split$x"
$LogFolderName = Get-Variable | Where-Object { $_.name -match $RegEx } | Select-Object value
[string]$LogFolders = $LogFolderName.value.ToString()
$LocalLogVar += $LogFolders + '\'
$x++
} until ($x -eq $SplitCountCheck)
$LocalLogVar = $LocalLogVar
$LocalLogVar = $LocalLogVar + "log"
[array]$LogFolders += $LocalLogVar
}

Wow, so that's a script. Kind of hard to follow, since some of it seems needlessly complex. I'm not sure if it will accomplish what you're looking for, but that's because you were super vague with your folder descriptions. Do the folders always start like this:
D:\folder<Service Short Name><Service Long Name>...\logs
If not you could be in trouble. The last four items on your example list of what you expect to see don't look like they are like that. I think the way your folders are laid out are like this:
D:\folder...<Service Short Name><Service Long Name>\logs
The difference being where the extra folders are located. If they're before the service like I think they are your script will miss things.
Now, on to getting your list that you want. What I see from looking at your script is that you get a folder list for all folders under D:\folder\*\*\ named 'log'. Then you split out the 3rd and 4th folders to get a service's short name, and long name respectively. Then one by one you pull a list of all services from WMI, and filter for just the service that matches the name and caption (short name, and long name) referred to by the folders. After that you make sure you only have one listing of any given service.
Regarding this first part of the script, you can make it faster by letting the file system provider filter things for you. Instead of pulling a folder list of everything and then filtering for paths that end in '\log', you should use the -filter parameter of the Get-ChildItem cmdlet like this:
$Folders = Get-ChildItem C:\temp\*\*\ -Directory -Recurse -Verbose -Filter 'log'
Then you should query WMI one time, save the result, then pick and choose from there based on your folders. Something like:
[array]$2 = foreach ($folder in $folders) {
$ServName,$ServType = $folder.fullname.split('\')[2,3] -replace '-',' '
$PathName = $AllServices |
Where-Object { $_.caption -like "*$ServName*" -and $_.caption -like "*$ServType*" } |
Select-Object Name, Caption, #{n = 'PathName'; e = { $_.PathName -replace '^(\w\S+) .*','$1' -replace '^([''"])([^\1]+)\1.*','$2' } }
}
$Services = $2 | Sort-Object pathname | Get-Unique -AsString
I did a little regex magic to clean up the pathname instead of just .trim('"') since this gets rid of parameters in the service execution, and cleans paths that are enclosed in single quotes not just double quotes. If what you have works for you feel free to keep it, but this is a little more capable. It may be worth noting that Get-Unique is case sensitive, so 'C:\folder\word3\word9-word8' and 'C:\folder\word3\word9-Word8' are different. You might want to do a .ToUpper() on your paths before you look for unique ones.
Once you have your array of services you loop through them, splitting the file path, reassembling it, and finally adding 'log' to the end of it. That was your way to remove the executable from the path. There's a cmdlet that was designed to do just that: split-path. Use that with Join-Path and that whole last loop gets much simpler:
[array]$LogFolders = foreach ($ServPath in $services.pathname) {
Join-Path (Split-Path $ServPath) 'log'
}
Lastly, try not to use +=, since PowerShell has to rebuild the whole array each time you do that. You'll notice I moved the $Variable = bit outside the loop in places that you do that.

Related

PowerShell ForEach hash comparison between directories

I'm working on a script to compare two directories. There are two main things I want the script to show in the output--which files exist on one directory but not the other, and which files appear in both directories but have differences in them. Matching files don't need to show up.
I got some advice before on how to achieve this, but since I'm still pretty new to PS I'm having trouble executing it. What I'm trying to do is this:
I have Path #1. For each file in that path, I want to test for their existence on Path #2.
If the file exists in both paths, do a hash comparison between them. If there are differences, add the files to List A.
If the file appears in Path 1 but not Path 2, put them in List B.
This isn't as important, but would it also be possible to find files that exist in Path 2 but not Path 1? For work purposes that probably won't matter, but it will still be nice just in case.
Take the output and format it so that it can show something like: "The following files exist in Path 1 and not Path 2," and "The following files exist in both paths but have differences."
Basically, I don't just want an info dump of files to be the output and people end up having to puzzle through it. And like I said, I think the advice I received on how to do it will be good, I'm just having trouble making it work.
Here's the code I have so far:
$Source = #(Get-ChildItem -Recurse -Path \\SERVER\D$\PSTest)
foreach ($file in $Source){
If ($Target = Test-Path #(Get-ChildItem -Recurse -Path \\SERVER\D$\PSTest))
{
$HashResult = (Compare-Object -ReferenceObject $file -DifferenceObject
$Target -Property hash -PassThru).Path
}
else {
$Missing += $file
}
}
Write-Host 'These files have differences.' -ForegroundColor Green
$HashResult
Write-Host 'These files are missing from the target path.' -ForegroundColor
Green
$Missing
When I run that, I don't get any results (other than the text output). Where am I going wrong with this?
Made a few assumptions about the file names and their uniqueness down through the various depths of the source/target folders:
$SourceDir = "C:\\temptest";
$DestDir = "D:\\temptest";
$SourceFiles = #(Get-ChildItem -Recurse -Path $SourceDir);
$DestFiles = #(Get-ChildItem -Recurse -Path $DestDir);
$SourceFileNames = $SourceFiles | % { $_.Name };
$DestFileNames = $DestFiles | % { $_.Name };
$MissingFromDestination = #();
$MissingFromSource = #();
$DifferentFiles = #();
foreach($f in $SourceFiles) {
if (!$DestFileNames.Contains($f.Name)) {
$MissingFromDestination += $f;
} else {
$t = $DestFiles | Where { $_.Name -eq $f.Name };
if ((Get-FileHash $f.FullName).hash -ne (Get-FileHash $t.FullName).hash) {
$DifferentFiles += $f;
}
}
}
foreach($f in $DestFiles) {
if (!$SourceFileNames.Contains($f.Name)) {
$MissingFromSource += $f;
}
}
"
Missing from Destination: "
$MissingFromDestination | % { $_.FullName };
"
Missing from Source: "
$MissingFromSource | % { $_.FullName };
"
Source is Different: "
$DifferentFiles | % { $_.FullName };
This is a bit naive in its approach insofar as it is really only checking file names and ignoring subfolder tree structures. But, hopefully, it will give you enough of a leaping off point.

Use txt file as list in PowerShell array/variable

I've got a script that searches for a string ("End program" in this case). It then goes through each file within the folder and outputs any files not containing the string.
It works perfectly when the phrase is hard coded, but I want to make it more dynamic by creating a text file to hold the string. In the future, I want to be able to add to the list of string in the text file. I can't find this online anywhere, so any help is appreciated.
Current code:
$Folder = "\\test path"
$Files = Get-ChildItem $Folder -Filter "*.log" |
? {$_.LastWriteTime -gt (Get-Date).AddDays(-31)}
# String to search for within the file
$SearchTerm = "*End program*"
foreach ($File in $Files) {
$Text = Get-Content "$Folder\$File" | select -Last 1
if ($Text | WHERE {$Text -inotlike $SearchTerm}) {
$Arr += $File
}
}
if ($Arr.Count -eq 0) {
break
}
This is a simplified version of the code displaying only the problematic area. I'd like to put "End program" and another string "End" in a text file.
The following is what the contents of the file look like:
*End program*,*Start*
If you want to check whether a file contains (or doesn't contain) a number of given terms you're better off using a regular expression. Read the terms from a file, escape them, and join them to an alternation:
$terms = Get-Content 'C:\path\to\terms.txt' |
ForEach-Object { [regex]::Escape($_) }
$pattern = $terms -join '|'
Each term in the file should be in a separate line with no leading or trailing wildcard characters. Like this:
End program
Start
With that you can check if the files in a folder don't contain any of the terms like this:
Get-ChildItem $folder | Where-Object {
-not $_.PSIsContainer -and
(Get-Content $_.FullName | Select-Object -Last 1) -notmatch $pattern
}
If you want to check the entire files instead of just their last line change
Get-Content $_.FullName | Select-Object -Last 1
to
Get-Content $_.FullName | Out-String

Recursing Through Multiple Text Files with References

I have hundreds of text files in a folder which can often reference each other, and go serveral levels deep. Not sure if I am explaining this well, so I will explain with an example.
Let's say folder "A" contains 500 .txt files. The first one could be called A.txt and somewhere in there it mentions B.txt, which in turn mentions C.txt and so on. I believe the number of levels down is no more than 10.
Now, I want to find a certain text strings which relate to A.txt by programmitically going through that file, then if it sees references to other .txt files go through them as well and so on. The resulting output would be something like A_out.txt which contains everything it found based on a regex.
I started out with this using Powershell but am now a little stuck:
$files = Get-ChildItem "C:\TEST\" -Filter *.txt
$regex = ‘PCB.*;’
for ($i=0; $i -lt $files.Count; $i++) {
$infile = $files[$i].FullName
$outfile = $files[$i].BaseName + "_out.txt"
select-string $infile -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Value } > $outfile
}
It goes through every .txt file and outputs everything that matches the PCB.*; expression to its corresponding _out.txt file.
I have absolutely no idea how to now expand this to include references to the other files. I'm not even sure if this is possible in PowerShell or whether I need to use another language to achieve what I want.
I could get some office monkey's to do all this manually but if this is relatively simple to code then it would save us a lot of time. Any help would be greatly appreciated :)
/Edit
Whilst running through this in my head, I thought I could build up an array for every time another one of the files is mentioned, and then repeat the process for those as well. However, back to my original problem, I have no idea how I would go about this.
/Edit 2:
Sorry, had been away for a few days and am only just picking this up. I have been using what I've learnt from this question and a few others to come up with the following:
function Get-FileReference
{
Param($FileName, $OutputFileName='')
if ($OutputFileName -eq '')
{
Get-FileReference $FileName ($FileName -replace '.xml$', '_out.xml')
}
else
{
Select-String $FileName -Pattern 'BusinessObject.[^"rns][w.]*' -AllMatches | % { $_.Matches } | % { $_.Value } | Add-Content $OutputFileName
Set-Location C:\TEST
$References = (Select-String -Pattern '(?<=resid=")d+' -AllMatches -path $FileName | % { $_.Matches } | % { $_.Value })
Write "SC References: $References" | Out-File OUTPUT.txt -Append
foreach ($Ref in $References)
{
$count
Write "$count" | Out-File OUTPUT.txt -Append
$count++
Write "SC Reference: $Ref" | Out-File OUTPUT.txt -Append
$xml = [xml](Get-Content 'C:\TEST\package.xml')
$res = $xml.SelectSingleNode('//res[#id = $Ref]/child::resver[last()]')
$resource = $res.id + ".xml"
Write "File to Check $resource" | Out-File OUTPUT.txt -Append
Get-FileReference $resource $OutputFileName
}
}
}
$files = gci "C:\TEST" *.xml
ForEach ($file in $files) {
Get-FileReference $file.FullName
}
Following my original question, I realised that this was a little bit more extensive than I originally thought and therefore had to tinker.
These are the noteable points:
All the parent files are .xml and code that matches on
"BusinessObject" etc works as expected.
The references to other
files are not simply .txt but require a pattern match of
'(?<=resid=")d+'.
This pattern match needs to be cross referenced with another file package.xml and based on the value
it returns, the file it next needs to look into is [newname].xml
As before, those child .xml files could reference some of the
other .xml files
The code I have pasted above seems to be getting stuck in endless loops (hence why I have debugging in there at the moment) and it is not liking the use of $Ref in:
$res = $xml.SelectSingleNode('//res[#id = $Ref]/child::resver[last()]')
That results in the following error:
Exception calling "SelectSingleNode" with "1" argument(s): "Namespace Manager or XsltContext needed. This query has a prefix, variable, or user-defined function."
Since there could be hundreds of files it dies when it gets over 1000+.
A recursive function which tries to do what you want.
function Get-FileReference
{
Param($FileName, $OutputFileName='')
if ($OutputFileName -eq '')
{
Get-FileReference $FileName ($FileName -replace '\.txt$', '_out.txt')
}
else
{
Select-String -Pattern 'PCB.*;' -Path $FileName -AllMatches | Add-Content $OutputFileName
$References = (Select-String -Pattern '^.*\.txt' -AllMatches -path $FileName).Matches.Value
foreach ($Ref in $References)
{
Get-FileReference $Ref $OutputFileName
}
}
}
$files = gci *.txt
ForEach ($file in $files) { Get-FileReference $file.FullName }
It takes two parameters - a filename and an output filename. If called without an output filename, it assumes it's at the top of a new recursion tree and generates an output filename to append to.
If called with an output filename (i.e. by itself) it searches for PCB patterns, appends to the output, then calls itself on any file references, with the same output filename.
Assuming that file references are lines on their own with no spaces xyz.txt.

Script/tool to delete specified filename

I need a batch file /script/tool to delete specified files in folder.
I have a folder with a lot of .xml files. It can contain files named difference of only a few characters (indicating the date).
aa_bb_000000001_2015_9_1.xml
aa_bb_000000001_2015_9_15.xml
aa_bb_000000001_2015_10_1.xml
aa_bb_000000002_2015_5_5.xml
aa_bb_000000002_2015_8_14.xml
aa_bb_000000002_2015_10_1.xml
aa_bb_000000005_2015_7_7.xml
.
.
The length of this part is 15 string
aa_bb_000000001
This part represents a date
2015_10_1
I need to delete all the files that part of the name with a date is earliest.
As a result batch should stay only files:
aa_bb_000000001_2015_10_1.xml
aa_bb_000000002_2015_10_1.xml
aa_bb_000000005_2015_7_7.xml
.
.
Here's one solution that's fairly short. To understand how the code works, it would be best to focus on what the Group-Object command does, what regular expressions are, and how they interact with the -match operator:
$Groups = Get-ChildItem "C:\XMLFiles\*.xml" | Group-Object {$_.Name.Substring(0, 15)}
$FilesToKeep = #{}
foreach ($Group in $Groups) {
$MaxDate = "00000000"
foreach ($FileInfo in $Group.Group) {
$FileInfo.name -match "(\d{4})_(\d{1,2})_(\d{1,2}).xml$" | Out-Null
$Date = $Matches[1]+([int]$Matches[2]).ToString("00")+([int]$Matches[3]).ToString("00")
if ($Date -gt $MaxDate) {
$MaxDate = $Date
$FilesToKeep[$Group.Name] = $FileInfo.FullName
}
}
}
Get-ChildItem "C:\XMLFiles\*.xml" | Where-Object {-not $FilesToKeep.ContainsValue($_.FullName)} | Remove-Item

Filtering Search Results Against a CSV in Powershell

Im interested in some ideas on how one would approach coding a search of a filesystem for files that match any entries contained in a master CSV file. I have a function to search the filesystem, but filtering against the CSV is proving harder than I expect. I have a csv with headers in it for Name & IPaddr:
#create CSV object
$csv = import-csv filename.csv
#create filter object containing only Name column
$filter = $csv | select-object Name
#Now run the search function
SearchSubfolders | where {$_.name -match $filter} #returns no results
I guess my question is this: Can I filter against an array within a pipeline like this???
You need a pair of loops:
#create CSV object
$csv = import-csv filename.csv
#Now run the search function
#loop through the folders
foreach ($folder in (SearchSubfolders)) {
#check that folder against each item in the csv filter list
#this sets up the loop
foreach ($Filter in $csv.Name) {
#and this does the checking and outputs anything that is matched
If ($folder.name -match $Filter) { "$filter" }
}
}
Usually CSVs are 2-dimensional data structures, so you can't use them directly for filtering. You can convert the 2-dimensional array into a 1-dimensional array, though:
$filter = Import-Csv 'C:\path\to\some.csv' | % {
$_.PSObject.Properties | % { $_.Value }
}
If the CSV has just a single column, the "mangling" can be simplified to this (replace Name with the actual column name):
$filter = Import-Csv 'C:\path\to\some.csv' | % { $_.Name }
or this:
$filter = Import-Csv 'C:\path\to\some.csv' | select -Expand Name
Of course, if the CSV has just a single column, it would've been better to make it a flat list right away, so it could've been imported like this:
$filter = Get-Content 'C:\path\to\some.txt'
Either way, with the $filter prepared, you can apply it to your input data like this:
SearchSubFolders | ? { $filter -contains $_.Name } # ARRAY -contains VALUE
The -match operator won't work, because it compares a value (left operand) against a regular expression (right operand).
See Get-Help about_Comparison_Operators for more information.
Another option is to create a regex from the filename collection and use that to filter for all the filenames at once:
$filenames = import-csv filename.csv |
foreach { $_.name }
[regex]$filename_regex = ‘(?i)^(‘ + (($filenames | foreach {[regex]::escape($_)}) –join “|”) + ‘)$’
$SearchSubfolders |
where { $_.name -match $filename_regex }
You can use Compare-Object to do this pretty easily if you are matching the actual Names of the files to names in the list. An example:
$filter = import-csv files.csv
ls | Compare-Object -ReferenceObject $filter -IncludeEqual -ExcludeDifferent -Property Name
This will print the files in the current directory that match the any Name in files.csv. You could also print only the different ones by dropping -IncludeEqual and -ExcludeDifferent flags. If you need full regex matching you will have to loop through each regex in the csv and see if it is a match.
Here's any alternate solution that uses regular expression filters. Note that we will create and cache the regex instances so we don't have to rely on the runtime's internal cache (which defaults to 15 items). First we have a useful helper function, Test-Any that will loop through an array of items and stop if any of them satisfies a criteria:
function Test-Any() {
param(
[Parameter(Mandatory=$True,ValueFromPipeline=$True)]
[object[]]$Items,
[Parameter(Mandatory=$True,Position=2)]
[ScriptBlock]$Predicate)
begin {
$any = $false
}
process {
foreach($item in $items) {
if ($predicate.Invoke($item)) {
$any = $true
break
}
}
}
end { $any }
}
With this, the implementation is relatively simple:
$filters = import-csv files.csv | foreach { [regex]$_.Name }
ls -recurse | where { $name = $_.Name; $filters | Test-Any { $_.IsMatch($name) } }
I ended up using a 'loop within a loop' construct to get this done after much trial and error:
#the SearchSubFolders function was amended to force results in a variable, SearchResults
$SearchResults2 = #()
foreach ($result in $SearchResults){
foreach ($line in $filter){
if ($result -match $line){
$SearchResults2 += $result
}
}
}
This works great after collapsing my CSV file down to a text-based array containing only the necessary column data from that CSV. Much thanks to Ansgar Wiechers for assisting me with that particular thing!!!
All of you presented viable solutions, some more complex than I cared for, nevertheless if I could mark multiple answers as correct, I would!! I chose the correct answer based on not only correctness but also simplicity.....

Resources