Update object array over multiple iterations - arrays

I have an array of custom objects:
$report = #()
foreach ($person in $mylist)
{
$objPerson = New-Object System.Object
$objPerson | Add-Member -MemberType NoteProperty -Name Name -Value $person.Name
$objPerson | Add-Member -MemberType NoteProperty -Name EmployeeID
$objPerson | Add-Member -MemberType NoteProperty -Name PhoneNumber
$report += $objPerson
}
Note that I haven't set values for the last two properties. The reason I've done this is because I'm trying to produce a matrix where I'll easily be able to see where these are blanks (although I could just set these to = "" if I have to).
Then, I want to iterate through a second dataset and update these values within this array, before exporting the final report. E.g. (this bit is pretty much pseudo code as I have no idea how to do it:
$phonelist = Import-Csv .\phonelist.csv
foreach ($entry in $phonelist)
{
$name = $entry.Name
if ($report.Contains(Name))
{
# update the PhoneNumber property of that specific object in the array with
# another value pulled out of this second CSV
}
else
{
# Create a new object and add it to the report - don't worry I've already got
# a function for this
}
}
I'm guessing for this last bit I probably need my if statement to return an index, and then use that index to update the object. But I'm pretty lost at this stage.
For clarity this is a simplified example. After that I need to go through a second file containing the employee IDs, and in reality I have about 10 properties that need updating all from different data sources, and the data sources contain different lists of people, but with some overlaps. So there will be multiple iterations.
How do I do this?

I would read phonelist.csv into a hashtable, e.g. like this:
$phonelist = #{}
Import-Csv .\phonelist.csv | ForEach-Object { $phonelist[$_.name] = $_.number }
and use that hashtable for filling in the phone numbers in $report as you create it:
$report = foreach ($person in $mylist) {
New-Object -Type PSObject -Property #{
Name = $person.Name
EmployeeID = $null
PhoneNumber = $phonelist[$person.Name]
}
}
You can still check the phone list for entries that are not in the report like this:
Compare-Object $report.Name ([array]$phonelist.Keys) |
Where-Object { $_.SideIndicator -eq '=>' } |
Select-Object -Expand InputObject

I would iterate over the $phonelist two times. The first time, you could filter all phone entities where the name is in your $myList and create the desired object:
$phonelist = import-cse .\phonelist.csv
$report = $phonelist | Where Name -in ($mylist | select Name) | Foreach-Object {
[PSCustomObject]#{
Name = $_.Name
PhoneNumber = $_.PhoneNumber
EmployeeID = ''
}
}
The second time you filter all phone entities where the name is not in $myList and create the new object:
$report += $phonelist | Where Name -NotIn ($mylist | select Name) | Foreach-Object {
#Create a new object and add it to the report - don't worry I've already got a function for this
}

Related

Powershell nested JSON to csv conversion

I have a rather peculiar nested JSON where in some instances a key - value pair occurs as normal, but in others the type of the key appears in a further nesting.
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
In the example you can see metadata's fields are straightforward key-value pairs, but underneath recordContent, we have positionDate which is a straightforward key-value but "account":{"string":"G32"} and "strike":{"double":4.4} are not.
I'd like to ditch the type information and arrive at a CSV structure as follows:
systemId, legalEntity, positionDate, account,seg,strike
da1895, A0, 2019-04-08 00:00:00.0,G32, S, 4.4
4536d, 5G, 2019-04-08 00:00:00.0,G81, S, 5.0
Any ideas on how to convert such a structure to CSV using Powershell?
Here's what I tried:
$TemplateParametersFile = "c:\data\output.json"
$JsonParameters = Get-Content $TemplateParametersFile | ConvertFrom-Json
$metadatafields = $JsonParameters.metadata[0].PSObject.Properties.Name
$recordcontentfields = $JsonParameters.recordContent[0].PsObject.Properties.Name
$oData = New-Object PSObject
$metadatafields |
ForEach {
Add-Member -InputObject $oData -NotePropertyName ($_) -NotePropertyValue $JsonParameters.metadata.($_)
}
$recordcontentfields |
ForEach {
Add-Member -InputObject $oData -NotePropertyName ($_) -NotePropertyValue $JsonParameters.recordContent.($_)
}
This gave me:
$oData
systemId : {da1895, 45364d}
legalEntity : {A0, 5G}
positionDate : {2019-04-08 00:00:00.0, 2019-04-08 00:00:00.0}
account : {#{string=G32}, #{string=G81}}
seg : {#{string=S}, #{string=S}}
strike : {#{double=4.4}, #{double=5.0}}
I'm a bit stuck now and the above doesn't convert to csv.
Note that other than metadata and recordContent, I've not hardcoded any fieldnames and I'd like to maintain that flexibility in case the JSON structure changes.
Thanks
I suggest collecting the property-name-value pairs iteratively in an ordered hashtable ([ordered] #{}), which can then be cast to [pscustomobject] to convert it to a custom object.
No property names are hard-coded in the following solution, but the object-graph structure is assumed to follow the pattern in your sample JSON, which is limited to one level of nesting - if you need to process arbitrarily nested objects, this answer may be a starting point.
Reflection (discovery of the property names and values) is performed via the intrinsic .psobject property that PowerShell makes available on all objects.
# Parse sample JSON into an array of [pscustomobject] graphs.
$fromJson = ConvertFrom-Json #'
[
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
,
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
]
'#
# Initialize an aux. ordered hashtable to collect the property-name-value
# pairs in.
$oht = [ordered] #{}
$fromJson | ForEach-Object {
$oht.Clear()
# Loop over top-level properties.
foreach ($topLevelProp in $_.psobject.Properties) {
# Loop over second-level properties.
foreach ($prop in $topLevelProp.Value.psobject.Properties) {
if ($prop.Value -is [System.Management.Automation.PSCustomObject]) {
# A nested value: Use the value of the (presumed to be one-and-only)
# property of the object stored in the value.
$oht[$prop.Name] = $prop.Value.psobject.Properties.Value
}
else {
# A non-nested value: use as-is.
$oht[$prop.Name] = $prop.Value
}
}
}
# Construct and output a [pscustomobject] from the aux. ordered hashtble.
[pscustomobject] $oht
} |
ConvertTo-Csv # Replace this with Export-Csv to export to a file.
The above yields:
"systemId","legalEntity","positionDate","account","seg","strike"
"da1895","A0","2019-04-08 00:00:00.0","G32","S","4.4"
"45364d","5G","2019-04-08 00:00:00.0","G81","S","5"
A few years ago, I wrote a reusable Flatten-Object function for this.
The only difference is that it combines the (sub)property names with the parent property names as they might not be unique:
$JsonParameters |Flatten-Object |Format-Table
metadata.systemId metadata.legalEntity recordContent.positionDate recordContent.account.string recordContent.seg.string recordContent.strike.double
----------------- -------------------- -------------------------- ---------------------------- ------------------------ ---------------------------
da1895 A0 2019-04-08 00:00:00.0 G32 S 4.4
45364d 5G 2019-04-08 00:00:00.0 G81 S 5
Try this:
$data = ConvertFrom-Json #"
[
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}},
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
]
"#
$data | Select-Object -Property #{l="systemId"; e={$_.metadata.systemId}}, #{l="legalEntity"; e={$_.metadata.legalEntity}},
#{l="positionDate"; e={$_.recordContent.positionDate}}, #{l="account"; e={$_.recordContent.account.string}},
#{l="seg"; e={$_.recordContent.seg.string}}, #{l="strike"; e={$_.recordContent.strike.double}} | Export-Csv
This should work with any nested psobject.
$json = #'
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
'#
$obj = ConvertFrom-Json $json
$obj.recordContent | gm -MemberType NoteProperty | % {
$prop = $_.name
if ($obj.recordContent.$prop.GetType().name -eq 'pscustomobject') {
$obj.recordContent.$prop = $obj.recordContent.$prop.psobject.Members | where membertype -eq noteproperty | select -ExpandProperty value
}
$obj.metadata | add-member -MemberType NoteProperty -Name $prop -Value $obj.recordContent.$prop
}
$newobj = $obj.metadata
$newobj

Update different arrays via a common function

I'm currently making a Powershell script that will analyze multiple log files from a mail server to gather various statistics that will be stored in a number of different arrays.
I have the following code snippet as an example of updating one of the arrays.
#Update arrays
#Overall array
$objNewValue = New-Object -TypeName PSObject
$objNewValue = $PSOSSLOverall | Where-Object {($_.Version -contains $strVersion -and $_.Cipher -contains $strCipher -and $_.Bits -contains $strBits)}
If ($objNewValue -ne $null) {
try {
Write-Verbose "$strVersion $strCipher $strBits is already in the array, so we'll update TimeSeen value"
$objNewValue.TimesSeen++
$objNewValue.LastSeen = $dtTimestamp
} #try
catch {
Write-Host "Something whent wrong while attempting to update an existing object in the overall array" -BackgroundColor DarkRed
Write-Host "Current line: $strtemp[$i]"
Write-Host "Current values: $dtTimestamp <-> $strVersion <-> $strCipher <-> $strBits"
Write-Host "Current array:"
$PSOSSLOverall | Sort-Object -Property Version, Cipher -Descending | Format-Table -AutoSize
Write-Host "Exception object:"
$_
} #catch
} #If Check for existence in Overall array
Else {
try {
Write-Verbose "$strVersion $strCipher $strBits is not in the array, so it will be added "
$objNewValue = New-Object -TypeName PSObject
Add-Member -InputObject $objNewValue -MemberType 'NoteProperty' -Name 'Version' -Value $strVersion
Add-Member -InputObject $objNewValue -MemberType 'NoteProperty' -Name 'Cipher' -Value $strCipher
Add-Member -InputObject $objNewValue -MemberType 'NoteProperty' -Name 'Bits' -Value $strBits
Add-Member -InputObject $objNewValue -MemberType 'NoteProperty' -Name 'TimesSeen' -Value 1
Add-Member -InputObject $objNewValue -MemberType 'NoteProperty' -Name 'Percentage' -Value 0
Add-Member -InputObject $objNewValue -MemberType 'NoteProperty' -Name 'FirstSeen' -Value $dtTimestamp
Add-Member -InputObject $objNewValue -MemberType 'NoteProperty' -Name 'LastSeen' -Value $dtTimestamp
$PSOSSLOverall += $objNewValue
} #try
catch {
Write-Host "Something whent wrong while attempting to add a new object to the overall array"
Write-Host "Current line: $strtemp[$i]"
Write-Host "Current values: $dtTimestamp <-> $strVersion <-> $strCipher <-> $strBits"
Write-Host "Exception object:"
$_
} #catch
} #Else Check for existence in Overall array
However, when I have up to 10 or more arrays that I need to update, the result will be a lot of similar code as there's only relatively few lines that change each time - like the array being updated, the where clause, the variables used and number of columns in the arrays.
Would it be possible to create a function that can handle updating the different arrays?
Thanks in advance.
-Update-
To explain the code snippet above: All the variables are already set before this part of the script is run. $strtemp[$i] is actually where all the data comes from as that the current line in the log file from which I then extract the needed data and place it in various variables.
First I search the array in question, which in this case is $PSOSSLOverall, to see if the data from the current line is already in the array. If $objNewValue is not null, then the data is already there, and I then increment a counter and update a date stamp for that "row" of data. If $objNewValue is null, then the data is not already there, and then we added a now object (row) to the array with the data from various variables.
Each attempt is equipped with try/catch section for error handling.
The end result will be an array that looks like this (the percentage column is calculated elsewhere):
The other arrays have various number of columns, which I what I guess makes it difficult to make a common function to update them.
I have found myself in a similar situation once or twice. You'll have to refactor your code, converting inflexible static definitions into parameter variables. The idea is to separate the data from program logic so you can do something like pass a different set of attribute names and values into the same function for different circumstances.
Here were a couple places I found to improve flexibility:
Parameterize the Where-Object expressions used to find matching records so you don't write the same code for each combination of columns and values. See hasPropertyValues below. All it does is perform an -and concatenated -contains operation on each name-value pair you pass to it.
Parameterize the data you want changed for each eventuality. Your code does something when it finds matching records and when it finds no matching records. Pull those actions out of the script body and into an input parameter that can change when you're done working with the encryption dataset and have to move onto another. The UpdateRecords function takes hashtable parameters defining the shape of the data when new records are added and when matching records are found.
See below for the example. I think you can adapt some of the ideas in here to your code.
# this is a convenience function allowing us to test multiple name-value pairs against an object
function hasPropertyValues {
Param(
[object] $inputObject,
[hashtable] $properties
)
$result = $true
foreach($name in $properties.Keys){
$result = $result -and ($inputObject.$name -contains $properties[$name])
}
Write-Output $result
}
# this function evaluates each object in $inputDataset
# if an object matches the name-values defined in $matchProperties
# it updates the records according to $updateRecordProperties
# if no records are found which match $matchProperties
# a new object is created with the properties in both $matchProperties
# and $newRecordProperties
# All results are written to the pipeline, including unmodified objects
function UpdateRecords{
Param (
[object[]] $inputDataset,
[hashtable] $matchProperties,
[hashtable] $updateRecordProperties,
[hashtable] $newRecordProperties
)
$numberOfMatchingRecords = 0
foreach ($record in $inputDataset){
if ( hasPropertyValues -inputObject $record -properties $matchProperties) {
# a record with matching property values found.
# Update required attributes
$numberOfMatchingRecords++
foreach($name in $updateRecordProperties.Keys){
if ($updateRecordProperties[$name] -is 'ScriptBlock'){
# if the update is a scriptblock, we invoke the scriptblock
# passing the record as input. The result of the invocation
# will be set as the new attribute value
$newValue = & $updateRecordProperties[$name] $record
} else {
$newValue = $updateRecordProperties[$name]
}
$record | Add-Member -Force -MemberType NoteProperty -Name $name -Value $newValue
}
}
Write-Output $record
}
if ($numberOfMatchingRecords -eq 0) {
# no records found with the search parameters
$newRecord = New-Object -TypeName psobject -Property $newRecordProperties
foreach($key in $matchProperties.Keys){
$newRecord | Add-Member -MemberType NoteProperty -Name $key -Value $matchProperties[$key] -Force
}
Write-Output $newRecord
}
}
[object[]] $TestDataset= #(New-Object psobject -Property #{
version='TLSv1.2'
cipher='ECDHE-RSA-AES256-GCM-SHA384'
Bits=256
TimesSeen = 1833
Percentage = 87578
FirstSeen = [DateTime]::Now
LastSeen = [DateTime]::Now
})
function TestUpdateRecords{
$encryptionNewRecordDefaults = #{
TimesSeen = 1
Percentage = 0
FirstSeen = [DateTime]::Now
LastSeen = [DateTime]::Now
}
$encryptionUpdateAttributes = #{
LastSeen = [DateTime]::Now
TimesSeen = {$ARGS[0].TimesSeen + 1}
}
# test adding a new record
UpdateRecords -inputDataset $TestDataset `
-matchProperties #{ Version='TLSv1.0';cipher='unbreakable';bits=720} `
-updateRecordProperties $encryptionUpdateAttributes `
-newRecordProperties $encryptionNewRecordDefaults
# test modifying a matching record
UpdateRecords -inputDataset $things `
-matchProperties #{Version='TLSv1.2';cipher='ECDHE-RSA-AES256-GCM-SHA384';bits=256} `
-updateRecordProperties $encryptionUpdateAttributes `
-newRecordProperties $encryptionNewRecordDefaults
}
TestUpdateRecords
There are a lot of different ways to implement this kind of refactoring. You could, for instance, extract dataset-specific logic into scriptblocks and pass these to your main loop function.
Another possibility is to dig into the object-oriented features of PowerShell and try to build classes around each of your datasets. This could encapsulate the 'Update' and 'New' actions in a pleasant way. I'm not yet literate enough in powershell OO features to try.

How to add a custom property to a PowerShell array?

Say I have a PowerShell array $Sessions = #() which I am going to fill with PSCustomObjects. How can I add a custom property to the array itself? E.g. so I can have $Sessions.Count which is built-in and $Sessions.Active which I want to set to the active session count.
I know that I can add properties to PSCustomObjects (in a dirty way) with
$MyCustomObject = "" | Select-Object Machine, UserName, SessionTime
but though doing so on an array would not result in the property being added.
So how can I achieve my goal? Is there any way to create a custom array?
The answer to your question as stated would be to just use Add-Member on the array object.
Add-Member -InputObject $sessions -MemberType NoteProperty -Name "State" -Value "Fabulous"
Adding a property to each element after you created the object is similar.
$sessions | ForEach-Object{
$_ | Add-Member -MemberType NoteProperty -Name "State" -Value "Fabulous"
}
This of course comes with a warning (that I forgot about). From comments
Beware, though, that appending to that array ($sessions += ...) will replace the array, thus removing the additional property.
Ansgar Wiechers
Depending on your use case there are other options to get you want you want. You can save array elements into distinct variables:
# Check the current object state
$state = $object.Property .....
# Add to the appropriate array.
if($state -eq "Active"){
$activeSessions += $object
} else {
$inactiveSessions += $object
}
Or you could still store your state property and post process with Where-Object as required:
# Process each inactive session
$sessions | Where-Object{$_.State -eq "Active"} | ForEach-Object{}
To avoid the destroying / recreating array issue, which can be a performance hog, you could also use an array list instead.
$myArray = New-Object System.Collections.ArrayList
Add-Member -InputObject $myArray -MemberType ScriptMethod -Name "NeverTellMeTheOdds" -Value {
$this | Where-Object{$_ % 2 -ne 0}
}
$myArray.AddRange(1..10)
$myArray.NeverTellMeTheOdds()
Notice that the array had its member added then we added its elements.
As Matt commented, you can use the Add-Member on an enumerable type by supplying it as a positional argument to the -InputObject parameter.
To allow for resizing after adding the new property, use a generic List instead of #():
$list = [System.Collections.Generic.List[psobject]]::new()
$list.AddRange(#(
[pscustomobject]#{SessionId = 1; Active = $true}
[pscustomobject]#{SessionId = 2; Active = $false}
[pscustomobject]#{SessionId = 3; Active = $true}
) -as [psobject[]])
Add-Member -InputObject $list -MemberType ScriptProperty -Name ActiveSessionCount -Value {
return #($this |? Active -eq $true).Count
}
Now you can retrieve the active session count easily:
PS C:\> $list.ActiveSessionCount
2

Remove one or many members from Object in powershell

I have created a custom object called $info and moving it to an array $arr ,
How is it possible to remove one member along with its all properties ?
My script:
Get-Process | ForEach-Object{
$info = New-Object -TypeName PSObject
$info | Add-Member -Type NoteProperty -Name Process -Value $_.processname
$info | Add-Member -Type NoteProperty -Name ID -Value $_.id
$arr += $info
}
$arr | ft -AutoSize
The result looks like this :
Process ID
------- --
ApplicationFrameHost 38556
AppVShNotify 9792
armsvc 2336
atieclxx 6944
atiesrxx 1844
audiodg 59432
CcmExec 3988
chrome 46068
How can I remove one particular member for example "audiodg 59432" gets removed
audiodg 59432
Your terminology is a bit incorrect here. A member is on an individual object. When you use Add-Member above you're adding properties to each individual object, then you're returning an array of objects.
You're asking how to remove an individual object from the array.
In PowerShell you cannot remove an item from an array. You could instead filter the array based on some criteria and create a new one:
$newArr = $arr | Where-Object { $_.Name -ne 'audiodg' }
# or
$newArr = $arr | Where-Object { $_.ID -ne 59432 }

Finding matches in arrays of objects in Powershell

I'm using custom objects to hold the name and schema from a set of SQL Server objects. I put the objects into an array, then I get another set of objects and put those into another array. What I'd like to do now is find all exact matches between the two arrays.
I'm currently using this:
$filteredSQLObjects = #()
foreach ($SQLObject1 in $SQLObjects1)
{
foreach ($SQLObject2 in $SQLObjects2)
{
if ($SQLObject1.Name -eq $SQLObject2.Name -and
$SQLObject1.Schema -eq $SQLObject2.Schema)
{
$filteredSQLObjects += $SQLObject1
}
}
}
Is there a better/faster/cleaner way to do this? Originally when I was just working with arrays of strings I could just loop through one of the arrays and use -contains on the second, but with objects that doesn't seem possible.
Thanks!
I think its better if you define the equality condition in an IsEqualTo method on your custom object. So something like this:
$myObject = New-Object PSObject
$myObject | Add-Member -MemberType NoteProperty -Name Name -Value $name
$myObject | Add-Member -MemberType NoteProperty -Name Schema -Value $schema
$myObject | Add-Member -MemberType ScriptMethod -Name IsEqualTo -Value {
param (
[PSObject]$Object
)
return (($this.Name -eq $Object.Name) -and ($this.Schema -eq $Object.Schema))
}
Then you can either do a one-liner like Keith showed us, or just do your double foreach iteration. Whichever you think is more readable:
$filteredSQLObjects = $SQLObjects1 | Where-Object { $SQLObject1 = $_; $SQLObjects2 | Where-Object { $_.IsEqualTo($SQLOBject1) } }
foreach ($SQLObject1 in $SQLObjects1)
{
foreach ($SQLObject2 in $SQLObjects2)
{
if ($SQLObject1.IsEqualTo($SQLObject2))
{
$filteredSQLObjects += $SQLObject1
}
}
}
EDIT
OK, for a start, you can't add an Equals member because it already exists on System.Object (doh!). So I guess IsEqualTo will have to do instead.
What you can do is define your own function called Intersect-Object (the equivalent of .NET's Enumerable.Intersect method) which accepts pipeline input and returns the set intersection of two sequences (the ones that appear in both sequences). Be aware that I haven't fully-implemented this function (assumes each item in the collection specified by Sequence has an IsEqualTo method, doesn't check for duplicates before adding to $filteredSequence etc), but I hope you get the idea.
function Intersect-Object
{
param (
[Parameter(ValueFromPipeline = $true)]
[PSObject]$Object,
[Parameter(Mandatory = $true)]
[PSObject[]]$Sequence
)
begin
{
$filteredSequence = #()
}
process
{
$Sequence | Where-Object { $_.IsEqualTo($Object) } | ForEach-Object { $filteredSequence += $_ }
}
end
{
return $filteredSequence
}
}
Then your double foreach loop turns into this:
$filteredSQLObjects = $SQLObjects1 | Intersect-Object -Sequence $SQLObjects2
You could condense this to a one-liner which would be appropriate if you were writing this at the console:
$filtered = $SQLObjects1 | ? {$o1=$_; $SQLObjects2 | ? {$_.Name -eq $o1.Schema `
-and $_.Name -eq $o1.Schema}}
But in a script, I would expand it out like you have it. It is more readable that way.

Resources