I have a rather peculiar nested JSON where in some instances a key - value pair occurs as normal, but in others the type of the key appears in a further nesting.
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
In the example you can see metadata's fields are straightforward key-value pairs, but underneath recordContent, we have positionDate which is a straightforward key-value but "account":{"string":"G32"} and "strike":{"double":4.4} are not.
I'd like to ditch the type information and arrive at a CSV structure as follows:
systemId, legalEntity, positionDate, account,seg,strike
da1895, A0, 2019-04-08 00:00:00.0,G32, S, 4.4
4536d, 5G, 2019-04-08 00:00:00.0,G81, S, 5.0
Any ideas on how to convert such a structure to CSV using Powershell?
Here's what I tried:
$TemplateParametersFile = "c:\data\output.json"
$JsonParameters = Get-Content $TemplateParametersFile | ConvertFrom-Json
$metadatafields = $JsonParameters.metadata[0].PSObject.Properties.Name
$recordcontentfields = $JsonParameters.recordContent[0].PsObject.Properties.Name
$oData = New-Object PSObject
$metadatafields |
ForEach {
Add-Member -InputObject $oData -NotePropertyName ($_) -NotePropertyValue $JsonParameters.metadata.($_)
}
$recordcontentfields |
ForEach {
Add-Member -InputObject $oData -NotePropertyName ($_) -NotePropertyValue $JsonParameters.recordContent.($_)
}
This gave me:
$oData
systemId : {da1895, 45364d}
legalEntity : {A0, 5G}
positionDate : {2019-04-08 00:00:00.0, 2019-04-08 00:00:00.0}
account : {#{string=G32}, #{string=G81}}
seg : {#{string=S}, #{string=S}}
strike : {#{double=4.4}, #{double=5.0}}
I'm a bit stuck now and the above doesn't convert to csv.
Note that other than metadata and recordContent, I've not hardcoded any fieldnames and I'd like to maintain that flexibility in case the JSON structure changes.
Thanks
I suggest collecting the property-name-value pairs iteratively in an ordered hashtable ([ordered] #{}), which can then be cast to [pscustomobject] to convert it to a custom object.
No property names are hard-coded in the following solution, but the object-graph structure is assumed to follow the pattern in your sample JSON, which is limited to one level of nesting - if you need to process arbitrarily nested objects, this answer may be a starting point.
Reflection (discovery of the property names and values) is performed via the intrinsic .psobject property that PowerShell makes available on all objects.
# Parse sample JSON into an array of [pscustomobject] graphs.
$fromJson = ConvertFrom-Json #'
[
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
,
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
]
'#
# Initialize an aux. ordered hashtable to collect the property-name-value
# pairs in.
$oht = [ordered] #{}
$fromJson | ForEach-Object {
$oht.Clear()
# Loop over top-level properties.
foreach ($topLevelProp in $_.psobject.Properties) {
# Loop over second-level properties.
foreach ($prop in $topLevelProp.Value.psobject.Properties) {
if ($prop.Value -is [System.Management.Automation.PSCustomObject]) {
# A nested value: Use the value of the (presumed to be one-and-only)
# property of the object stored in the value.
$oht[$prop.Name] = $prop.Value.psobject.Properties.Value
}
else {
# A non-nested value: use as-is.
$oht[$prop.Name] = $prop.Value
}
}
}
# Construct and output a [pscustomobject] from the aux. ordered hashtble.
[pscustomobject] $oht
} |
ConvertTo-Csv # Replace this with Export-Csv to export to a file.
The above yields:
"systemId","legalEntity","positionDate","account","seg","strike"
"da1895","A0","2019-04-08 00:00:00.0","G32","S","4.4"
"45364d","5G","2019-04-08 00:00:00.0","G81","S","5"
A few years ago, I wrote a reusable Flatten-Object function for this.
The only difference is that it combines the (sub)property names with the parent property names as they might not be unique:
$JsonParameters |Flatten-Object |Format-Table
metadata.systemId metadata.legalEntity recordContent.positionDate recordContent.account.string recordContent.seg.string recordContent.strike.double
----------------- -------------------- -------------------------- ---------------------------- ------------------------ ---------------------------
da1895 A0 2019-04-08 00:00:00.0 G32 S 4.4
45364d 5G 2019-04-08 00:00:00.0 G81 S 5
Try this:
$data = ConvertFrom-Json #"
[
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}},
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
]
"#
$data | Select-Object -Property #{l="systemId"; e={$_.metadata.systemId}}, #{l="legalEntity"; e={$_.metadata.legalEntity}},
#{l="positionDate"; e={$_.recordContent.positionDate}}, #{l="account"; e={$_.recordContent.account.string}},
#{l="seg"; e={$_.recordContent.seg.string}}, #{l="strike"; e={$_.recordContent.strike.double}} | Export-Csv
This should work with any nested psobject.
$json = #'
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
'#
$obj = ConvertFrom-Json $json
$obj.recordContent | gm -MemberType NoteProperty | % {
$prop = $_.name
if ($obj.recordContent.$prop.GetType().name -eq 'pscustomobject') {
$obj.recordContent.$prop = $obj.recordContent.$prop.psobject.Members | where membertype -eq noteproperty | select -ExpandProperty value
}
$obj.metadata | add-member -MemberType NoteProperty -Name $prop -Value $obj.recordContent.$prop
}
$newobj = $obj.metadata
$newobj
Two comma separated item added in array list and I would like to group them to count the total.
$list_distinct = [System.Collections.ArrayList]#()
$list_distinct.Add("Site A,Item A")
$list_distinct.Add("Site A,Item A")
$list_distinct.Add("Site A,Item B")
$list_distinct.Add("Site B,Item C")
$list_distinct.Add("Site B,Item D")
$list_distinct.Add("Site B,Item D")
Tried this:
$test = $list_distinct | Group-Object Values
The result shows Count (the whole total), Name(empty) and Group (the whole added items).
Any way to fix this? Or is there any better method?
Desired output example:
Site | Item | Count
Site A | Item A | 2
Site A | Item B | 1
Site B | Item C | 1
Site B | Item D | 2
Neither the ArrayList object nor its elements have a property Values. Non-existent properties are expanded to an empty result, so all of your values are grouped under the same (empty) name.
Change this
$list_distinct | Group-Object Values
into this
$list_distinct | Group-Object
and the problem will disappear.
For your desired output you will also need to split the values and create new (custom) objects:
$list_distinct | Group-Object | ForEach-Object {
$site, $item = $_.Name -split ','
New-Object -Type PSObject -Property #{
'Site' = $site
'Item' = $item
'Count' = $_.Count
}
} | Select-Object Site, Item, Count
The trailing Select-Object is to enforce field order since PowerShell hashtables aren't ordered by default.
In PowerShell v3 and newer you can simplify that to
$list_distinct | Group-Object | ForEach-Object {
$site, $item = $_.Name -split ','
[PSCustomObject]#{
'Site' = $site
'Item' = $item
'Count' = $_.Count
}
}
The trailing Select-Object isn't needed here, because the [PSCustomObject] type accelerator implicitly uses an ordered hashtable.
When we're trying to export data to other functions via the pipeline, we observe some strange behavior in PowerShell.
Example code:
$Array = #()
$Obj1 = [PSCustomObject]#{
Member1 = 'First'
Member2 = 'Second'
}
$Obj2 = [PSCustomObject]#{
Member1 = 'First'
Member2 = 'Second'
Member3 = 'Third'
}
$Array = $Obj1, $Obj2
$Array | Out-GridView -Title 'Not showing Member3'
$Array = $Obj2, $Obj1
$Array | Out-GridView -Title 'All members correctly displayed'
In the example above you can see that when the first object only contains 2 properties, the Out-GridView CmdLet (and others) only show 2 properties, even though the second object has 3 properties. However, when the first object in the array has 3 properties it does display them all correctly.
Is there a way around this? Because it's not possible to predict up front how many properties on an object there will be and if the object with the most properties will be the first one in the array.
I had the same experience once and created the following reusable 'Union' function:
# 2021-08-25 Removed Union function
Usage:
$Obj1, $Obj2 | Union | Out-GridView -Title 'Showing all members'
It is also supposed to work with complex objects. Some standard cmdlets output multiple object types at once and if you view them (e.g. Out-GridView) or dump them in a file (e.g. Export-Csv) you might miss a lot of properties. Take as another example:
Get-WmiObject -Namespace root/hp/instrumentedBIOS -Class hp_biosSetting | Union | Export-Csv ".\HPBIOS.csv"
Added 2014-09-19:
Maybe this is already between the lines in the comments $Array | Select * | … will not resolve the issue but specifically selecting the properties $Array | Select Member1, Member2, Member3 | … does.
Besides, although in most cases the Union function will work, there are some exceptions to that as it will only align the first object with the rest.
Consider the following object:
$List = #(
New-Object PSObject -Property #{Id = 2}
New-Object PSObject -Property #{Id = 1}
New-Object PSObject -Property #{Id = 3; Name = "Test"}
)
If you Union this object everything appears to be fine and if you e.g. ExportTo-CSV and work with the export .csv file from then on you will never have any issue.
$List | Union
Id Name
-- ----
2
1
3 Test
Still there is a catch as only the first object is aligned. If you e.g. sort the result on Id (Sort Id) or take just the last 2 (Select -Last 2) entries, the Name is not listed because the second object doesn’t contain the Name property:
$List | Union | Sort Id
Id
--
1
2
3
Therefor I have rewritten the Union-Object (Alias Union) function`):
Union-Object
# 2021-08-25 Removed Union-Object function
Syntax:
$Array | Union | Out-GridView -Title 'All members correctly displayed'
Update 2021-08-25
Based on az1d helpful feedback on an error caused by equal property names with different casing, I have created a new UnifyProperties function.
(I will no longer use the name UnionObject for his)
function UnifyProperties {
$Names = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::OrdinalIgnoreCase)
$InputCollected = #($Input)
$InputCollected.ForEach({
foreach ($Name in $_.psobject.Properties.Name) { $Null = $Names.Add($Name) }
})
$inputCollected | Select-Object #($Names)
}
Usage:
[pscustomobject] #{ one = 1; two = 2; three = 3 },
[pscustomobject] #{ ONE = 10; THREE = 30; FOUR = 4 } |
UnifyProperties
one two three FOUR
--- --- ----- ----
1 2 3
10 30 4
See also: #13906 Add -UnifyProperties parameter to Select-Object
I have an array of custom objects:
$report = #()
foreach ($person in $mylist)
{
$objPerson = New-Object System.Object
$objPerson | Add-Member -MemberType NoteProperty -Name Name -Value $person.Name
$objPerson | Add-Member -MemberType NoteProperty -Name EmployeeID
$objPerson | Add-Member -MemberType NoteProperty -Name PhoneNumber
$report += $objPerson
}
Note that I haven't set values for the last two properties. The reason I've done this is because I'm trying to produce a matrix where I'll easily be able to see where these are blanks (although I could just set these to = "" if I have to).
Then, I want to iterate through a second dataset and update these values within this array, before exporting the final report. E.g. (this bit is pretty much pseudo code as I have no idea how to do it:
$phonelist = Import-Csv .\phonelist.csv
foreach ($entry in $phonelist)
{
$name = $entry.Name
if ($report.Contains(Name))
{
# update the PhoneNumber property of that specific object in the array with
# another value pulled out of this second CSV
}
else
{
# Create a new object and add it to the report - don't worry I've already got
# a function for this
}
}
I'm guessing for this last bit I probably need my if statement to return an index, and then use that index to update the object. But I'm pretty lost at this stage.
For clarity this is a simplified example. After that I need to go through a second file containing the employee IDs, and in reality I have about 10 properties that need updating all from different data sources, and the data sources contain different lists of people, but with some overlaps. So there will be multiple iterations.
How do I do this?
I would read phonelist.csv into a hashtable, e.g. like this:
$phonelist = #{}
Import-Csv .\phonelist.csv | ForEach-Object { $phonelist[$_.name] = $_.number }
and use that hashtable for filling in the phone numbers in $report as you create it:
$report = foreach ($person in $mylist) {
New-Object -Type PSObject -Property #{
Name = $person.Name
EmployeeID = $null
PhoneNumber = $phonelist[$person.Name]
}
}
You can still check the phone list for entries that are not in the report like this:
Compare-Object $report.Name ([array]$phonelist.Keys) |
Where-Object { $_.SideIndicator -eq '=>' } |
Select-Object -Expand InputObject
I would iterate over the $phonelist two times. The first time, you could filter all phone entities where the name is in your $myList and create the desired object:
$phonelist = import-cse .\phonelist.csv
$report = $phonelist | Where Name -in ($mylist | select Name) | Foreach-Object {
[PSCustomObject]#{
Name = $_.Name
PhoneNumber = $_.PhoneNumber
EmployeeID = ''
}
}
The second time you filter all phone entities where the name is not in $myList and create the new object:
$report += $phonelist | Where Name -NotIn ($mylist | select Name) | Foreach-Object {
#Create a new object and add it to the report - don't worry I've already got a function for this
}
I'm attempting to add data through a loop to a new third column in an array but I'm out of luck.
I'm locating the data that I need
$DL = Get-DistributionGroup -Identity "*" | Select Name,Manag*
Name
ManagedBy
I attempt to loop through it, successfully, but I have no idea on how to add a new field here and then input it in the correct position...
$DL.Name | ForEach-Object {$DL.Members += Get-DistributionGroupMember -Identity $_ | Select Name}
Name
ManagedBy
Members
I'm not familiar with Exchange cmdlets, but I think you could use a calculated property for this:
$DL = Get-DistributionGroup -Identity '*' | Select-Object Name, Manag*, #{
Name = 'Members'
Expression = {Get-DistributionGroupMember -Identity $_ | Select-Object Name}
}