Powershell Array: HOWTO Dedup Output - arrays

When I run this parser script on my contacts.xml, which shows one line per user, I get multiple instances of the same data. I only want a single entry for the same data. How do I dedup the data before it writes to the CSV?
#https://stackoverflow.com/questions/29999682/powershell-parsing-a-text-file
$input = Get-Content $env:USERPROFILE\Downloads\contacts.xml\Downloads\contacts.xml
$array = #()
$input | % {
$writeobj = $false
$obj = New-Object System.Object
if ($_ -match 'email*') {
$Email = ($_ -split ':')[1]
}
if ($_ -match 'FN*') {
$NAME = ($_ -split ':')[1]
$writeobj = $true
}
if ($writeobj) {
$obj | Add-Member -Type NoteProperty -Name Email -Value $Email
$obj | Add-Member -Type NoteProperty -Name Name -Value $NAME
$array += $obj
}
Write-Host $Name, $email
}
$array | Export-Csv -Path C:\scripts\reports\test.csv -NoTypeInformation
I expect this to produce single entries but I get duplicates (and they don't line up right either).
(And yes I checked the XML file for single entries)

Select the unique objects.
$array |
Select-Object -Property * -Unique |
Export-Csv -Path 'C:\scripts\reports\test.csv' -NoType
As a side note, you may want to avoid appending to an array in a loop, as that is bound to perform poorly. Just pipe your ForEach-Object loop directly into Export-Csv.

I figured it out.
I REVERSED the $obj Add-Member variables *that fixed the order) and added another "$writeobj = $true" line to the FN match, and VOILĂ€ no more dupes.
Is that weird or what?
#https://stackoverflow.com/questions/29999682/powershell-parsing-a-text-file
$input = Get-Content $env:USERPROFILE\Downloads\contacts.xml $array =
#() $input | % {
$writeobj = $false
$obj = New-Object System.Object
If ($_ -match 'email') {
$Email = ($_ -split ':')[1]
$writeobj = $true
}
If ($_ -match 'FN') {
$NAME = ($_ -split ':')[1]
$writeobj = $true # <-- right here
}
If ($writeobj){
$obj | Add-Member -type NoteProperty -name Email -value **$NAME**
$obj | Add-Member -type NoteProperty -name Name -value **$Email**
$array += $obj
}
Write-Host $Name, $email } $array | Export-Csv -path C:\scripts\reports\test.csv -NoTypeInformation

Related

Parsing a text file line by line and extracting string if it matches

We have a text log file similar to below with many users
<user>sandip</user>
something
<time>4:38 PM</time>
anything
<elapsed time> 60 mins </elapsed time>
We want to extract all users and we did the same simply by
Get-Content "C:\LOG\test.txt" | Select-String '(<user>.+</user>)' | ForEach-Object {
$_.Matches[0].Groups[1].Value
}
We want to parse test file line by line, check if it contains
user/time/elapsed time and [insert it in a dynamic variable if required] make a table of the same
Considering that your Log File follows the same format that i tested with:
(i.e something like this one:)
LogFile
This code should work just fine:
*
$Lines = get-content .\log.txt
$array = #()
foreach ($line in $lines)
{
if($line -like "<user>*")
{
$obj = New-Object psobject
Add-Member -InputObject $obj -MemberType NoteProperty -Name "UserName" -value $line.Replace("<user>","").Replace("</user>","")
}
if($line -like "<time>*")
{
Add-Member -InputObject $obj -MemberType NoteProperty -Name "Time" -value $line.Replace("<time>","").Replace("</time>","")
}
if($line -like "<elapsed time>*")
{
Add-Member -InputObject $obj -MemberType NoteProperty -Name "ElapsedTime" -value $line.Replace("<elapsed time>","").Replace("</elapsed time>","")
$array += $obj
}
}
$array | Export-Csv .\test.csv
*
A quite compact solution using two regular expressions,
with a nonconsuming positive
lookahead to
split the source file into sections starting with <user>
inside an if to get -match the current line with an alternation and a backreference to insert the found key and value into the current Row of the new table. See this RegEx live on regex101.com
## Q:\Test\2018\06\22\SO_50988379.ps1
$Table = ForEach ($Section in ((Get-Content .\Test.log -raw) -split '(?=<user>)' -ne '')) {
$Row = New-Object psobject
ForEach ($Line in ($Section -split "`r?`n")) {
if($Line -match "<(user|time|elapsed time)>([^<]+)</\1>"){
Add-Member -InputObject $Row -MemberType NoteProperty `
-Name "$($Matches[1])" -value $Matches[2].Trim()
}
}
$Row
}
$Table #| Export-Csv .\test.csv
Sample output:
> .\SO_50988379.ps1
user time elapsed time
---- ---- ------------
sandip 4:38 PM 60 mins
Joshi 8:15 PM 60 mins

Powershell - foreach to array as job (local / multi-threaded)

I'm trying to parse a site to collect price and product details. The script works in a loop however it's very slow. So I'm trying to run a multi-threaded powershell script as a job.
I've tried a lot of suggestions but I'm struggling to get the results out even though I can see its working (the web-request screen flashing up)
I'm only selecting the last 10 but I'll put in a throttle later. Just can't get it to output. Essentially I'd like all results to flow back into $arr.
#Import Danmurphy Sitelist
[xml] $XmlDocument = (New-Object System.Net.WebClient).DownloadString("http://www.example.com/sites.xml")
#get websites listed
$ImportedProducts = $XmlDocument.DocumentElement.url | select -Last 10
"Killing existing jobs . . ."
Get-Job | Remove-Job -Force
"Done."
#loop through the products
#Create Array
$arr = #()
#$argumentlist
#ScriptBlock
$ScriptBlock = {
Param($product,$arr)
if ($product.loc -like "http://www.example.com/product/*"){
$uri = $product.loc
$WebResponse = Invoke-WebRequest -Uri $uri -SessionVariable WS
#mainpricetest
$mainprice = $WebResponse.AllElements | ? { $_.Class -eq 'price-main' } | select innerText
$MainPriceArray = $mainprice.innerText.Split(' ')
$MainUnitArry = $MainPriceArray[1..10]
$MainDollar = $MainPriceArray[0]
$MainUnit = $MainUnitArry -join ' '
$item = New-Object PSObject
$item | Add-Member -type NoteProperty -Name 'Product Site' -Value $($product.loc)
$item | Add-Member -type NoteProperty -Name 'Main Price' -Value $($MainDollar)
$item | Add-Member -type NoteProperty -Name 'Main Unit' -Value $($MainUnit)
$arr += $item
}
}
foreach ($product in $ImportedProducts){
Start-Job -InputObject $ImportedProducts -ScriptBlock $ScriptBlock -ArgumentList $product,$arr
}
$data = Get-Job * | Receive-Job
#Show Array
$arr
So you would want to use runspaces for that. Runspaces is a pretty complicated thing, luckily we have Posh-RSJob which handles everything for you. https://github.com/proxb/PoshRSJob
You can pass in the script block, so you would need very little adjustments.
Probably something like this:
foreach ($product in $ImportedProducts){
Start-RSJob -ScriptBlock $ScriptBlock
}
Get-RSjob | Receive-RSJob
If you want to get the results into $arr, you can't do it from within the script block as you are attempting to do. Multiple script blocks running in parallel cannot be allowed to access a single copy of a variable without taking additional steps not worth getting into.
The answer to your problem is going to be to write the output of each script block as regular output. That output is buffered until you use Receive-Job to get the results out of the job at which time you capture it into the $arr variable in a single threaded manner. Below is cod which should get you most of the way there.
#Import Danmurphy Sitelist
[xml] $XmlDocument = (New-Object System.Net.WebClient).DownloadString("http://www.example.com/sites.xml")
#get websites listed
$ImportedProducts = $XmlDocument.DocumentElement.url | select -Last 10
"Killing existing jobs . . ."
Get-Job | Remove-Job -Force
"Done."
#loop through the products
#Create Array
$arr = #()
#$argumentlist
#ScriptBlock
$ScriptBlock = {
Param($product)
if ($product.loc -like "http://www.example.com/product/*"){
$uri = $product.loc
$WebResponse = Invoke-WebRequest -Uri $uri -SessionVariable WS
#mainpricetest
$mainprice = $WebResponse.AllElements | ? { $_.Class -eq 'price-main' } | select innerText
$MainPriceArray = $mainprice.innerText.Split(' ')
$MainUnitArry = $MainPriceArray[1..10]
$MainDollar = $MainPriceArray[0]
$MainUnit = $MainUnitArry -join ' '
$item = New-Object PSObject
$item | Add-Member -type NoteProperty -Name 'Product Site' -Value $($product.loc)
$item | Add-Member -type NoteProperty -Name 'Main Price' -Value $($MainDollar)
$item | Add-Member -type NoteProperty -Name 'Main Unit' -Value $($MainUnit)
Write-Output $item
}
}
foreach ($product in $ImportedProducts){
Start-Job -InputObject $ImportedProducts -ScriptBlock $ScriptBlock -ArgumentList $product
}
do {
$arr += Get-Job -State Completed | Receive-Job -AutoRemoveJob
} while (Get-Job -State Running)
#Show Array
$arr

How do I convert an array object to a string object in PowerShell?

While trying to create an CSV file with information about certificates I have an issue to store the userrights on the private key.
The problem is that I want to store multiple values in one attribute so I use an array.
At first I had no errors, however the column in my csv-file remained empty even in the case where the array has a value.
With a simple Write-Host I can see my array has the expected value so this part works okay.
For further investigations I have added the line:
Get-Member $certs.GetValue("UserRights")
This gives an error indicating I have to convert my variable to a string-variable.
So next I have tried to convert this array to a single string.
I have tried several ways but my error doesn't disappear so it doesn't work.
Underneath is my full code with some former attempts commented.
cls $certs = Get-ChildItem cert:\LocalMachine -Recurse | Where-Object {-not $_.PSIsContainer} | Select * Write-Host ("There were {0} certificates" -f ($certs | Measure-Object).Count)
foreach($certificate in $certs) {
if($certificate.HasPrivateKey)
{
Write-Host "Certificate's PSChildName is" $certificate.PSChildName
$rsaFile = $certificate.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName
$fullPath = "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\" + $rsaFile
$acl = Get-Acl -Path $fullPath
foreach($accessrule in $acl.Access)
{
Write-Host "User" $accessrule.IdentityReference "has the following rights:" $accessrule.FileSystemRights
}
Write-Host "------"
$UserRechten = #()
foreach($accessrule in $acl.Access)
{
$UserRechten += "{0}:{1};" -f ($accessrule.IdentityReference,$accessrule.FileSystemRights)
}
Write-Host "================================================================"
# -join $UserRechten
# $Userrechten | out-string
# $UserRechten = [system.String]::Join(" ", $UserRechten)
$separator = ";"
[string]::Join($separator,$UserRechten)
$certs | Add-Member -MemberType NoteProperty -Name "UserRights" -Value $UserRechten -Force
Write-Host "UserRechten has value : "$UserRechten
Get-Member $certs.GetValue("UserRights")
Write-Host "================================================================"
} }
$Certs | Add-Member -MemberType NoteProperty -Name "MachineName" -Value $env:COMPUTERNAME -Force
# $certs | Add-Member -MemberType NoteProperty -Name "Store" -Value 'My' -Force $RunDate = Get-Date -Format 'yyyy-MM-dd' $certs | Add-Member -MemberType NoteProperty -Name "RunDate" -Value $RunDate -Force $certs | Add-Member -MemberType NoteProperty -Name "Owner" -Value $env:USERNAME -Force
$Certs | Select * | Export-Csv c:\Certificaten\LocalCertsAll_$env:COMPUTERNAME.csv
$Certs | Select MachineName, Owner, PSParentPath, DnsNameList, PSChildName, NotBefore, NotAfter, Rundate, EnhancedKeyUsageList, HasPrivateKey, SerialNumber, Issuer, Subject, FriendlyName, UserRigthts |
Export-CSV c:\Certificaten\Localcerts_$env:COMPUTERNAME.csv
As noted in the comments, Get-Member is probably not what you're looking for
You (almost certainly) don't want to add the UserRights member property to the $Certs array, but rather to the individual objects in $Certs.
(I removed a bunch of superfluous Write-Host statements for readability):
$CertsAmended = foreach($Certificate in $certs)
{
if($certificate.HasPrivateKey)
{
$rsaFile = $certificate.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName
$fullPath = "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\" + $rsaFile
$acl = Get-Acl -Path $fullPath
# Create the UserRights value using -join
$UserRechten = #(foreach($accessrule in $acl.Access){
Write-Host "User" $accessrule.IdentityReference "has the following rights:" $accessrule.FileSystemRights
"{0}:{1}" -f ($accessrule.IdentityReference,$accessrule.FileSystemRights)
}) -join ";"
# Add the property to the individual object
$Certificate | Add-Member -MemberType NoteProperty -Name "UserRights" -Value $UserRechten
Write-Host "Userrights: " $UserRechten
# "Drop" the certificate object (now with a UserRights value) back onto the pipeline
$Certificate
}
}
Now you can export the $CertsAmended array to CSV all you want
If you find the $var = #(foreach($item in $collection){}) -join ';' displeasing to the eye, break it into two statements:
$UserRechten = foreach($accessrule in $acl.Access)
{
# Create UserRight string here, without ;
}
$UserRechten = $UserRechten -join ';'
For the $fullPath variable, you may want to use the Join-Path cmdlet:
$fullPath = Join-Path "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\" $rsaFile

Create Powershell Object From Loop Containing Multiple Users

I need to create a Powershell object, array or hash table to store a list of users and assorted details, these are extracted from a CSV file and located using Get-ADUser. As below:
$userList = Import-CSV $CSVInputFile
$users = #{}
Foreach ($csvUser in $userList)
{
$userSearchString = $csvUser | Select -ExpandProperty SamAccountName
$currentUser = (Get-ADUser -Filter {SamAccountName -eq $userSearchString} `
-Properties PasswordExpired,PasswordLastSet,EmailAddress |
Where {$_.Enabled -eq "True"})
If ($currentUser.EmailAddress -ne $null)
{
$currentUserEmailString = $csvUser | Select -ExpandProperty EmailAddress
$currentUserEmailString = ($currentUserEmailString -as [string])
$currentUser.EmailAddress = $currentUserEmailString
}
$Users = New-Object PSObject -Property #{
DistinguishedName = $currentUser.DistinguishedName
EmailAddress = $currentUser.EmailAddress
Enabled = $currentUser.Enabled
GivenName = $currentUser.GivenName
Name = $currentUser.Name
PasswordExpired = $currentUser.PasswordExpired
PasswordLastSet = $currentUser.PasswordLastSet
SamAccountName = $currentUser.SamAccountName
Surname = $currentUser.Surname
}
$Users
}
How can I add the details of each user for each iteration of the loop to the object.
I want to end up with an object containing the details of a number of users, same as the output directly from Get-ADUser:
Name SamAccountName EmailAddress
---- -------------- ------------
User1 user1 user1#domain.com
User2 user2 user2#domain.com
User3 user3 user3#domain.com
Any help would be greatly appreciated!
Not sure if I'm missing the point on this but I see you are building a custom object right in your loop. The only issue I do see is you are not keeping the results after each loop. Rather you are destroying the objects history.
I would change the declaration of $users to an array $users = #() and instead of populating a user hashtable into users add the current object into the array. You will then have an array of hashtables:
$Users += New-Object PSObject -Property #{...
Then you could the $Users output line outside the loop and you will have the whole thing. Then you could just output to a Select to get the output you desire.
$Users | Select-Object name,SamAccountName,EmailAddress
There is a potential major drawback of this approach though. When using += on arrays a new array is created and resized for the new element and the old array is discarded. This has huge performance implications for larger arrays.
An even better way to approach this would be to leverage the pipeline. This would be a performance boost when you have larger user groups.
Import-CSV $CSVInputFile | ForEach-Object{
$userSearchString = $_.SamAccountName
$currentUser = Get-ADUser -Filter {SamAccountName -eq $userSearchString} `
-Properties PasswordExpired,PasswordLastSet,EmailAddress |
Where {$_.Enabled -eq "True"}
If ($currentUser.EmailAddress -ne $null){
$currentUser.EmailAddress = $_.EmailAddress
}
[pscustomobject][ordered]#{
DistinguishedName = $currentUser.DistinguishedName
# ..... truncated
Surname = $currentUser.Surname
}
}
Now you could send that to something like Export-CSV or just save it into a variable. Your options are open now. [pscustomobject][ordered] are type accelerators available in PowerShell v3.0+
Define an $users as Array
$users = #()
and append the New-Object into $Users.
$Users += New-Object
Can't believe both of you guys got in before me! Oh well.
Hope this helps anyway.
$userList = Import-CSV $CSVInputFile
$users = #()
Foreach ($csvUser in $userList)
{
$userSearchString = $csvUser | Select -ExpandProperty SamAccountName
$currentUser = (Get-ADUser -Filter {SamAccountName -eq $userSearchString} `
-Properties PasswordExpired,PasswordLastSet,EmailAddress |
Where {$_.Enabled -eq "True"})
If ($currentUser.EmailAddress -ne $null)
{
$currentUserEmailString = $csvUser | Select -ExpandProperty EmailAddress
$currentUserEmailString = ($currentUserEmailString -as [string])
$currentUser.EmailAddress = $currentUserEmailString
}
#clears the properties of the previous object and starts collecting properties
$UserObj = New-Object PSObject
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "DistinguishedName" -Value $($currentUser.DistinguishedName)
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "EmailAddress" -Value $($currentUser.EmailAddress)
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "Enabled" -Value $($currentUser.Enabled)
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "GivenName" -Value $($currentUser.GivenName)
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "UserName" -Value $($currentUser.Name)
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "PasswordExpired" -Value $($currentUser.PasswordExpired)
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "PasswordLastSet" -Value $($currentUser.PasswordLastSet)
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "SamAccountName" -Value $($currentUser.SamAccountName)
Add-Member -InputObject $UserObj -MemberType NoteProperty -Name "Surname" -Value $($currentUser.Surname)
#saves the properties in an array that exists outside of the loop to preserve information beyond one interation
$users += $UserObj
}
$users | Format-Table -Property UserName,SamAccountName,EmailAddress

Combine two arrays into a hashtable

Is there a way to combine two arrays to a hashtable ?
$Name = Get-ChildItem C:\Users | Select-Object Name
$CreationTime = Get-ChildItem C:\Users | Select-Object CreationTime
$Table = New-Object psobject
foreach ($item in $Name) {
foreach ($item2 in $CreationTime) {
Add-Member -InputObject $Table -MemberType NoteProperty -Name $item -Value $item2 } }
If I simply print out $item and $item2 I get every result multiple times, I know that this is because of the nested foreachs.
The example above is not very good, i acutally would need this to import different csv files and create a hashtable to export them again.
Have a single for loop were you iterate through the first array with an index. Assuming the same index in both arrays gives the correct pair, add the pair to the hashtable. No need for nested loops.
Some Pseudo code:
for ($i = 0; $i < $Name.length; $i++) {
Add-Member -Name $Name[$i] -Value $CreationTime[$i]
}
What I requested would look something like this:
$Name = Get-ChildItem C:\Users | Select-Object Name
$CreationTime = Get-ChildItem C:\Users | Select-Object CreationTime
$Table = New-Object psobject
$i = 0
do {
Add-Member -InputObject $Table -MemberType NoteProperty -Name $Name[$i].Name -Value $CreationTime[$i].CreationTime
$i++ }
while ($i -ne $name.Length)
$Table
However, this solution is more like what I wanted:
$Name = Get-ChildItem C:\Users | Select-Object Name
$CreationTime = Get-ChildItem C:\Users | Select-Object CreationTime
$TableData =#()
$i = 0
do {
$Table = New-Object psobject
Add-Member -InputObject $Table -MemberType NoteProperty -Name "FolderName:" -Value $Name.Name[$i]
Add-Member -InputObject $Table -MemberType NoteProperty -Name "Creation Time:" -Value $CreationTime.CreationTime[$i]
$TableData += $Table
$i++ }
while ($i -ne $name.Length)
$TableData | Export-csv -Delimiter ";" -Path $PSScriptRoot\out.csv -NoTypeInformation
Sorry for being unspecific! Is there an easier way to do this ?

Resources