Slow array operations after DB resultset - arrays

I need help optimizing my PowerShell script.
$sorted = #()
$firsttime = 0
$j = 0
$zaehler = $results.Count-1
for ($i=0; $i -le $results.Count-1; $i++) {
$j = $i+1
while ($results.GUID[$i] -eq $results.GUID[$j]) {
$klassen = ""
$rec = $results | where {$_.GUID -eq $results.GUID[$i]}
if ($firsttime -eq 0 -or !$sorted.GUID.contains($rec[0].GUID)) {
$firsttime = 1
foreach ($item in $rec.Klasse) {
if ($klassen -eq "") {
$klassen += $item
} else {
if (!$klassen.Contains($item)) {
$klassen += "," + $item
}
}
}
$rec[0].Klasse = $klassen
$sorted += $rec[0]
}
$j = $j+1
}
Write-Host ($i/$zaehler).ToString("P") "von Schule" $schule
}
if (!$sorted) {
$results
} else {
$sorted
}
Basically in my resultset ($results) I got duplicate lines of teachers and the only difference is the class ("Klasse/Klassen") they are teaching at.
To minimize the output I am checking if the first GUID is the same as the second and then the script appends the second class to the first one. So the $sorted array has just one line per teacher with a comma-seperated string which shows all classes.
Sample line of $results:
#{
GUID={1234567-1234-1234-1234-1234567};
userrole=teacher;
Vorname=Max;
Nachname=Mustermann;
Geburtstag=01.01.2000;
Klasse=9A;
Schule=123456
}
#{
GUID={1234567-1234-1234-1234-1234567};
userrole=teacher;
Vorname=Max;
Nachname=Mustermann;
Geburtstag=01.01.2000;
Klasse=9B;
Schule=123456
}
Sample line of $sorted[0]:
#{
GUID={1234567-1234-1234-1234-1234567};
userrole=teacher;
Vorname=Max;
Nachname=Mustermann;
Geburtstag=01.01.2000;
Klasse=9A,9B,9C,5A;
Schule=123456
}
The sorting process (check if contains, foreach $item, add to $klassen) is pretty slow.
I would be very grateful for any kind of ideas how to optimize the script.

Maybe something like this would work:
$results | ForEach-Object {
New-Object -Type PSObject -Property $_
} | Group-Object GUID | ForEach-Object {
$teacher = $_.Group
$teacher | Select-Object -First 1 -Exclude Klasse -Property *, #{n='Klasse';e={
$teacher | Select-Object -Expand Klasse
}}
}
Convert your hashtables into custom objects, group them by GUID, then replace the original "Klasse" property with a new one containing an array of the values from all objects in that group, and select the first result.
Should $results already be a list of objects you can skip the first ForEach-Object:
$results | Group-Object GUID | ForEach-Object {
$teacher = $_.Group
$teacher | Select-Object -First 1 -Exclude Klasse -Property *, #{n='Klasse';e={
$teacher | Select-Object -Expand Klasse
}}
}

Related

Find duplicate values in an array

i have the following code:
cls
Get-Module -ListAvailable | Import-Module
Import-Module ActiveDirectory
$Groups = Get-ADGroup -Filter {name -like 'XYZ'} | select name -
ExpandProperty name
$i=0
$tot = $Groups.count
$Table = #()
$Record = #{
"Group Name" = ""
"Name" = ""
"username" = ""
}
Foreach ($Group in $Groups) {
#// Set up progress bar
$i++
$status = "{0:N0}" -f ($i / $tot * 100)
Write-Progress -Activity "Exporting AD Groups" -status "Processing
Group $i of $tot : $status% Completed" -PercentComplete ($i / $tot *
100)
$Arrayofmembers = Get-ADGroupMember -identity $Group -recursive |
select name, SamAccountName
foreach ($Member in $Arrayofmembers) {
$Record."Group Name" = $Group
$Record."Name" = $Member.name
$Record."username" = $Member.SamAccountName
$objRecord = New-Object PSObject -property $Record
$Table += $objrecord
}
}
Write-Host $Table
which works perfectly but i want to list all duplicates in the $Record."Name" = $Member.name with specific group so for example:
username=barry is duplicated in GROUP XYZ
i tried already the following:
ForEach ($Element in $Table)
{
If (($Table -match $Element).count -gt 1)
{
"Duplicates detected"
}
}
The simplest answer is to just pipe $Table to Group-Object and filter for groups with a count greater than one at the end as such:
$Table | Group 'Group Name','Name' | Where{$_.Count -gt 1}
If you are looking to do this in the middle of your loop you could do so by grouping the results of Get-ADGroupMember, but I think it'll probably be faster to do it all at the end.
You could simply keep track of the members using a hashtable:
$seen = #{};
foreach ($Member in $Arrayofmembers)
{
if($seen[$Member.Name])
{
Write-Host "$($Member.Name) is duplicated in group $Group";
}
else
{
$seen.Add($Member.Name, $true);
}
# ... rest of the loop
}

Output arrays to CSV

I have two CSVs:
Total 19_01_16.csv:
hostname,user,path,size,creation,LastAccess,Copied,NewName,Completed
comp1,user1,\\comp1\users1\file.pst,100,17/02/2015,17/01/2016,Yes,file_user1_.pst,
comp1,user1,\\comp1\users1\file2.pst,200,17/02/2015,17/01/2016,Yes,file2_user1_.pst,
comp2,user2,\\comp2\users2\file.pst,100,17/02/2015,17/01/2016,Yes,file_user2_.pst,
PST Passwords.csv:
user,Path,Password1,Password2,Password3,Error
user1,\\comp1\users1\file.pst,openme,openme,openme,
I'm trying to merge the two with different headers and additional content.
This is what I have so far:
$a = Import-Csv "$PST_PARENT\DailyReports\Total 19_01_16.csv"
"Hostname,User,PST_Name,Original_Path,New_Path,Size,AcceptableLoss,Password1,Password2,Password3" |
Set-Content "$PST_PARENT\DailyReports\New Build.csv"
$a | foreach {
$HOSTNAME = $_.hostname
$USER = $_.User
$PATH = $_.path
$NEW_NAME = $_.NewName
$NEWPATH = "$PST_SHARE\$USER\$NEW_NAME"
$SIZE = $_.Size
$SIZE_FAIL = ( [convert]::ToSingle( $SIZE ) / 2 )
$b = Import-Csv "$PST_PARENT\DailyReports\PST Passwords.csv"
$b | foreach {
if ( $USER -like $b.user ) {
$PASSWORD1 = $b.password1
$PASSWORD2 = $b.password2
$PASSWORD3 = $b.password3
} else {
$PASSWORD1 = "none"
$PASSWORD2 = "none"
$PASSWORD3 = "none"
}
}
$HOSTNAME,$USER,$NEW_NAME,$PATH,$NEWPATH,$SIZE,$SIZE_FAIL,$PASSWORD1,$PASSWORD2,$PASSWORD3 |
Add-Content "$PST_PARENT\DailyReports\New Build.csv"
}
The output of New Build.csv looks like this:
Hostname,User,PST_Name,Original_Path,New_Path,Size,AcceptableLoss,Password1,Password2,Password3
comp1
user1
file.pst
\\comp1\users1\file.pst
\\share\PST_Storage\file_user1_.pst
100
5
none
none
none
In essence the output is working, it's just not scrolling for each line, it's putting each array onto a new line.
I tried adding | ConvertTo-Csv -NoTypeInformation but all that did was convert the arrays to numbers, they still went down not across.
Any ideas? Am I on the right line or doing the whole thing so very wrong?
$HOSTNAME,$USER,... defines an array, which is written to the output file one element per line. You need to put the list in double quotes to turn it into a comma-separated string that you can write to the output file as a single line.
"Hostname,User,PST_Name,Original_Path,New_Path,Size,AcceptableLoss,Password1,Password2,Password3" |
Set-Content "$PST_PARENT\DailyReports\New Build.csv"
$a | foreach {
...
"$HOSTNAME,$USER,$NEW_NAME,$PATH,$NEWPATH,$SIZE,$SIZE_FAIL,$PASSWORD1,$PASSWORD2,$PASSWORD3"
} | Add-Content "$PST_PARENT\DailyReports\New Build.csv"
or you construct a custom object from the elements that you can export via Export-Csv.
$a | foreach {
...
New-Object -Type PSCustomObject -Property #{
'Hostname' = $HOSTNAME
'User' = $USER
'NewName' = $NEW_NAME
'Path' = $PATH
'NewPath' = $NEWPATH
'Size' = $SIZE
'SizeFail' = $SIZE_FAIL
'Password1' = $PASSWORD1
'Password2' = $PASSWORD2
'Password3' = $PASSWORD3
}
} | Export-Csv "$PST_PARENT\DailyReports\New Build.csv" -NoType

How to add a number at end of username

I need to generate usernames that start with a letter of the alphabet, followed by the date and that ends with numbers from 1 to 20. For a total of 520 names.
Here's the variables I came up with:
$letter = 97..122 | foreach {[char]$_}
$date = get-date -UFormat %a%y%d
$name = $letter | ForEach-Object { $_ + $date }
So I managed to put my letters and my dates together, to form names like aWed1513, which is exactly what I need, but when it comes to make each to end with numbers, I'm stuck. Tried with a For loop by adding incrementing $i to $name, but doesn't work..
It seems like you're just about there. Your for loop should look like this:
$names = #()
$letter = 97..122 | foreach {[char]$_}
$date = get-date -UFormat %a%y%d
for($i = 1; $i -le 20; $i++) {
$names += $letter | ForEach-Object { '{0}{1}{2}' -f $_, $date, $i }
}
$names
An alternate approach to the answer given that just adds another ForEach-Object to the OP's original code:
$letter = 97..122 | foreach {[char]$_}
$date = get-date -UFormat %a%y%d
$names = $letter | ForEach-Object { $_ + $date } | ForEach-Object { for($i=1;$i -le 20;$i++) { $_ + $i } }
And to shorten to get same result:
$letter = 97..122 | foreach {[char]$_}
$date = get-date -UFormat %a%y%d
$names = $letter | ForEach-Object { for($i=1;$i -le 20;$i++) { $_ + $date + $i } }

Get all combinations of an array

I'm currently trying to make a function that gets all possible combinations of array values.
I have come up with a non function version but it's limited to 3 values so i'm trying to make a function out of it to become more Dynamic
I tried searching SO but could not find a powershell example of what i was trying to do, i could find a PHP version but i'm very limited in my PHP
PHP: How to get all possible combinations of 1D array?
Non-function Script
$name = 'First','Middle','Last'
$list = #()
foreach ($c1 in $name) {
foreach ($c2 in $name) {
foreach ($c3 in $name) {
if (($c1 -ne $c2) -and ($c2 -ne $c3) -and ($c3 -ne $c1))
{
$list += "$c1 $c2 $c3"
}
}
}
}
This gives me the result
First Middle Last
First Last Middle
Middle First Last
Middle Last First
Last First Middle
Last Middle First
I'm not sure how i would rearrange the values when i'm recursing the function, this is what i have so far:
<#
.Synopsis
Short description
.DESCRIPTION
Long description
.EXAMPLE
Example of how to use this cmdlet
.EXAMPLE
Another example of how to use this cmdlet
#>
function Get-Combinations
{
[CmdletBinding()]
[OutputType([int])]
Param
(
# Param1 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
[string[]]$Array,
# Param1 help description
[Parameter(Mandatory=$false,
ValueFromPipelineByPropertyName=$false,
Position=1)]
[string]$Temp,
# Param1 help description
[Parameter(Mandatory=$false,
ValueFromPipelineByPropertyName=$true,
Position=2)]
[string[]]$Return
)
Begin
{
Write-Verbose "Starting Function Get-Combinations with parameters `n`n$($Array | Out-String)`n$temp`n`n$($Return | Out-String)"
If ($Temp)
{
$Return = $Temp
}
$newArray = new-object system.collections.arraylist
}
Process
{
Write-Verbose ($return | Out-String)
For($i=0; $i -lt $Array.Length; $i++)
{
#Write-Verbose $i
$Array | ForEach-Object {$newArray.Add($_)}
$newArray.RemoveAt($i)
Write-Verbose ($newArray | Out-String)
if ($newArray.Count -le 1)
{
Get-Combinations -Array $newArray -Temp $Temp -Return $Return
}
else
{
$Return = $Temp
}
}
$newArray
}
End
{
Write-Verbose "Exiting Function Get-Combinations"
}
}
$combinations = #("First","First2","Middle","Last")
$Combos = Get-Combinations -Array $combinations
$Combos
But the output i'm getting is all over the place
First2
Last
First2
Last
First
First2
Middle
Last
First
First2
Middle
Last
28/08 Update
Getting closer but still getting weird output
<#
.Synopsis
Short description
.DESCRIPTION
Long description
.EXAMPLE
Example of how to use this cmdlet
.EXAMPLE
Another example of how to use this cmdlet
#>
function Get-Combinations
{
[CmdletBinding()]
[OutputType([int])]
Param
(
# Param1 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
[string[]]$Array,
# Param1 help description
[Parameter(Mandatory=$false,
ValueFromPipelineByPropertyName=$false,
Position=1)]
[string]$Temp,
# Param1 help description
[Parameter(Mandatory=$false,
ValueFromPipelineByPropertyName=$true,
Position=2)]
[string[]]$Return
)
Begin
{
Write-Verbose "Starting Function Get-Combinations with parameters `n`n$($Array | Out-String)`n$temp`n`n$($Return | Out-String)"
If ($Temp)
{
$Return += $Temp
}
#$newArray = new-object [System.Collections.ArrayList]
#$Array | ForEach-Object {$newArray.Add($_) | Out-Null}
[System.Collections.ArrayList]$newArray = $Array
}
Process
{
Write-Verbose "return -> $return"
For($i=0; $i -lt $Array.Length; $i++)
{
Write-Verbose "`$i -> $i"
$element = $newArray[0]
$newArray.RemoveAt(0)
Write-Verbose "`$newArray -> $newArray"
Write-Verbose "Element -> $element"
if ($newArray.Count -gt 0)
{
Get-Combinations -Array $newArray -Temp (($temp + " " +$element).Trim()) -Return $Return
}
else
{
$Return = $Temp + " " + $element
}
}
$return
}
End
{
Write-Verbose "Exiting Function Get-Combinations"
}
}
$combinations = #("First","First2","Middle","Last")
$return = #()
$Combos = Get-Combinations -Array $combinations -Return $return
$Combos
New output (Yes there is a space before the 'Last' value, no i have no idea why)
First First2 Middle Last
First First2 Last
First Middle Last
First Last
First2 Middle Last
First2 Last
Middle Last
Last
Here is my solution:
function Remove ($element, $list)
{
$newList = #()
$list | % { if ($_ -ne $element) { $newList += $_} }
return $newList
}
function Append ($head, $tail)
{
if ($tail.Count -eq 0)
{ return ,$head }
$result = #()
$tail | %{
$newList = ,$head
$_ | %{ $newList += $_ }
$result += ,$newList
}
return $result
}
function Permute ($list)
{
if ($list.Count -eq 0)
{ return #() }
$list | %{
$permutations = Permute (Remove $_ $list)
return Append $_ $permutations
}
}
cls
$list = "x", "y", "z", "t", "v"
$permutations = Permute $list
$permutations | %{
Write-Host ([string]::Join(", ", $_))
}
EDIT: the same in one function (Permute). This is cheating a bit, however since I replaced plain functions whith lambdas. You could replace recursive calls with a stack you handle yourself, but that would make the code unecessarily complex ...
function Permute ($list)
{
$global:remove = {
param ($element, $list)
$newList = #()
$list | % { if ($_ -ne $element) { $newList += $_} }
return $newList
}
$global:append = {
param ($head, $tail)
if ($tail.Count -eq 0)
{ return ,$head }
$result = #()
$tail | %{
$newList = ,$head
$_ | %{ $newList += $_ }
$result += ,$newList
}
return $result
}
if ($list.Count -eq 0)
{ return #() }
$list | %{
$permutations = Permute ($remove.Invoke($_, $list))
return $append.Invoke($_, $permutations)
}
}
cls
$list = "x", "y", "z", "t"
$permutations = Permute $list
$permutations | %{
Write-Host ([string]::Join(", ", $_))
}
I tried to learn something new and help you out but Im stuck. maybe this will help you get in the right direction but I dont know enough about Powershell recursion to figure this out. I converted the php to powershell and in theory it should work but it doesnt.
$array = #('Alpha', 'Beta', 'Gamma', 'Sigma')
function depth_picker([system.collections.arraylist]$arr,$temp_string, $collect)
{
if($temp_string -ne ""){$collect += $temp_string}
for($i = 0; $i -lt $arr.count;$i++)
{
[system.collections.arraylist]$arrCopy = $arr
$elem = $arrCopy[$i]
$arrCopy.removeRange($i,1)
if($arrCopy.count -gt 0){
depth_picker -arr $arrCopy -temp_string "$temp_string $elem" -collect $collect}
else{$collect += "$temp_string $elem"}
}
}
$collect = #()
depth_picker -arr $array -temp_string "" -collect $collect
$collect
It seems to work and will get you the first set of possibles:
Alpha
Alpha Beta
Alpha Beta Gamma
Alpha Beta Gamma Sigma
But for some reason that I cant figure out when it gets back to the previous functions and does $i++ then checks ($i -lt $arr.count) $arr.count it always 0 so it never goes to the next iteration to continue finding the possibilities.
Hopefully someone else can fix what I cant seem to figure out as I dont know enough about recursion. But it seems that with each level of depth called the previous depth level $arr variable and values is lost.
Here is my solution with a recursive function. It generates space separated strings but it's quite simple to split each element with $list[$i].split(" "):
function Get-Permutations
{
param ($array, $cur, $depth, $list)
$depth ++
for ($i = 0; $i -lt $array.Count; $i++)
{
$list += $cur+" "+$array[$i]
if ($depth -lt $array.Count)
{
$list = Get-Permutations $array ($cur+" "+$array[$i]) $depth $list
}
}
$list
}
$array = #("first","second","third","fourth")
$list = #()
$list = Get-Permutations $array "" 0 $list
$list
The solution posted by Micky Balladelli almost worked for me. Here is a version that does not duplicate values:
Function Get-Permutations
{
param ($array_in, $current, $depth, $array_out)
$depth++
$array_in = $array_in | select -Unique
for ($i = 0; $i -lt $array_in.Count; $i++)
{
$array_out += ($current+" "+$array_in[$i]).Trim()
if ($depth -lt $array_in.Count)
{
$array_out = Get-Permutations $array_in ($current+" "+$array_in[$i]) $depth $array_out
}
else {}
}
if(!($array_out -contains ($array_in -Join " "))) {}
for ($i = 0; $i -lt $array_out.Count; $i++)
{
$array_out[$i] = (($array_out[$i].Split(" ")) | select -Unique) -Join " "
}
$array_out | select -Unique
}

Find file extension in folder and sum total size

I am trying to find all file extension in a folder and subfolders and generate a list. I found a oneliner but it do not generate the list as i want. i got mutiple of paths so i do this.
$date = get-date -Format d
$File = "C:\NoBackup\FolderPaths.txt"
foreach ($Folder in (Get-Content $File)) {
Get-ChildItem $Share -Recurse -ErrorAction SilentlyContinue | Group-Object extension | Select-Object #{Name="Folder";Expression={$Folder}}, name, #{n='TotalSize';e={$_.group | ForEach-Object -Begin {$size=0} -Process {$size += ([decimal]::round($_.Length / 1MB))} -End {"$size MB"}}} | Sort-Object -Property 'TotalSize' -Descending | Format-Table -AutoSize
}
This will give a new header foreach folder in folderpaths, and i need the result be like this
.ext1 .ext2 .ext3 .ext4
D:\Folder1 5MB 12MB 20MB 8MB
D:\Folder2 10MB 54MB 12MB 3MB
D:\Folder3 2MB 12MB 20MB 100MB
I cant find out to rewrite the code to get what i need. Hope you can help me out with this.
The script works now. I needed to change
foreach($folder in $folders)
To
foreach($folder in (Get-Content $file))
It is not short or sweet, but try this:
function ConvertTo-Units($decimal)
{
$value = [decimal]::Round($decimal/1mb,2)
$units = "MB"
if($value -eq 0)
{
$value = [decimal]::Round($decimal/1kb,2)
$units = "KB"
}
return "{0} {1}" -f $value,$units
}
$File = "C:\NoBackup\FolderPaths.txt"
$fileData = #{}
foreach ($folder in (Get-Content $file))
{
$files = Get-ChildItem $folder -Recurse -ErrorAction SilentlyContinue -File
$fileData[$folder] = $files | Select-Object extension,length | %{$h = #{}} { $h[$_.extension] += $_.length } { $h}
}
$extensions = $fileData.Keys | % { $fileData[$_].Keys } | % tolower | Select-Object -Unique | ? { $_ }
$properties = #(
#{Name="Folder";Expression={$_}}
)
$extensions | % {$properties += #{Name="$_";Expression=[scriptblock]::Create("ConvertTo-Units `$fileData[`$folder][""$_""]")}}
$result = #()
foreach($folder in $folders)
{
$result += $folder | Select-Object $properties
}
$result | ft * -AutoSize -force

Resources