I have a rather peculiar nested JSON where in some instances a key - value pair occurs as normal, but in others the type of the key appears in a further nesting.
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
In the example you can see metadata's fields are straightforward key-value pairs, but underneath recordContent, we have positionDate which is a straightforward key-value but "account":{"string":"G32"} and "strike":{"double":4.4} are not.
I'd like to ditch the type information and arrive at a CSV structure as follows:
systemId, legalEntity, positionDate, account,seg,strike
da1895, A0, 2019-04-08 00:00:00.0,G32, S, 4.4
4536d, 5G, 2019-04-08 00:00:00.0,G81, S, 5.0
Any ideas on how to convert such a structure to CSV using Powershell?
Here's what I tried:
$TemplateParametersFile = "c:\data\output.json"
$JsonParameters = Get-Content $TemplateParametersFile | ConvertFrom-Json
$metadatafields = $JsonParameters.metadata[0].PSObject.Properties.Name
$recordcontentfields = $JsonParameters.recordContent[0].PsObject.Properties.Name
$oData = New-Object PSObject
$metadatafields |
ForEach {
Add-Member -InputObject $oData -NotePropertyName ($_) -NotePropertyValue $JsonParameters.metadata.($_)
}
$recordcontentfields |
ForEach {
Add-Member -InputObject $oData -NotePropertyName ($_) -NotePropertyValue $JsonParameters.recordContent.($_)
}
This gave me:
$oData
systemId : {da1895, 45364d}
legalEntity : {A0, 5G}
positionDate : {2019-04-08 00:00:00.0, 2019-04-08 00:00:00.0}
account : {#{string=G32}, #{string=G81}}
seg : {#{string=S}, #{string=S}}
strike : {#{double=4.4}, #{double=5.0}}
I'm a bit stuck now and the above doesn't convert to csv.
Note that other than metadata and recordContent, I've not hardcoded any fieldnames and I'd like to maintain that flexibility in case the JSON structure changes.
Thanks
I suggest collecting the property-name-value pairs iteratively in an ordered hashtable ([ordered] #{}), which can then be cast to [pscustomobject] to convert it to a custom object.
No property names are hard-coded in the following solution, but the object-graph structure is assumed to follow the pattern in your sample JSON, which is limited to one level of nesting - if you need to process arbitrarily nested objects, this answer may be a starting point.
Reflection (discovery of the property names and values) is performed via the intrinsic .psobject property that PowerShell makes available on all objects.
# Parse sample JSON into an array of [pscustomobject] graphs.
$fromJson = ConvertFrom-Json #'
[
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
,
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
]
'#
# Initialize an aux. ordered hashtable to collect the property-name-value
# pairs in.
$oht = [ordered] #{}
$fromJson | ForEach-Object {
$oht.Clear()
# Loop over top-level properties.
foreach ($topLevelProp in $_.psobject.Properties) {
# Loop over second-level properties.
foreach ($prop in $topLevelProp.Value.psobject.Properties) {
if ($prop.Value -is [System.Management.Automation.PSCustomObject]) {
# A nested value: Use the value of the (presumed to be one-and-only)
# property of the object stored in the value.
$oht[$prop.Name] = $prop.Value.psobject.Properties.Value
}
else {
# A non-nested value: use as-is.
$oht[$prop.Name] = $prop.Value
}
}
}
# Construct and output a [pscustomobject] from the aux. ordered hashtble.
[pscustomobject] $oht
} |
ConvertTo-Csv # Replace this with Export-Csv to export to a file.
The above yields:
"systemId","legalEntity","positionDate","account","seg","strike"
"da1895","A0","2019-04-08 00:00:00.0","G32","S","4.4"
"45364d","5G","2019-04-08 00:00:00.0","G81","S","5"
A few years ago, I wrote a reusable Flatten-Object function for this.
The only difference is that it combines the (sub)property names with the parent property names as they might not be unique:
$JsonParameters |Flatten-Object |Format-Table
metadata.systemId metadata.legalEntity recordContent.positionDate recordContent.account.string recordContent.seg.string recordContent.strike.double
----------------- -------------------- -------------------------- ---------------------------- ------------------------ ---------------------------
da1895 A0 2019-04-08 00:00:00.0 G32 S 4.4
45364d 5G 2019-04-08 00:00:00.0 G81 S 5
Try this:
$data = ConvertFrom-Json #"
[
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}},
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
]
"#
$data | Select-Object -Property #{l="systemId"; e={$_.metadata.systemId}}, #{l="legalEntity"; e={$_.metadata.legalEntity}},
#{l="positionDate"; e={$_.recordContent.positionDate}}, #{l="account"; e={$_.recordContent.account.string}},
#{l="seg"; e={$_.recordContent.seg.string}}, #{l="strike"; e={$_.recordContent.strike.double}} | Export-Csv
This should work with any nested psobject.
$json = #'
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
'#
$obj = ConvertFrom-Json $json
$obj.recordContent | gm -MemberType NoteProperty | % {
$prop = $_.name
if ($obj.recordContent.$prop.GetType().name -eq 'pscustomobject') {
$obj.recordContent.$prop = $obj.recordContent.$prop.psobject.Members | where membertype -eq noteproperty | select -ExpandProperty value
}
$obj.metadata | add-member -MemberType NoteProperty -Name $prop -Value $obj.recordContent.$prop
}
$newobj = $obj.metadata
$newobj
Related
I feel like my Delphi background is destroying my ability to figure this out. I'm trying to create an empty array (no data, just structure) in Powershell where each item has two properties. The end result would look something like this:
$WIP[0].signature = 'data'
$WIP[0].PID = 'data'
# other fake stuff in between
Write-host "Item 43 has signature: " $WIP[43].signature
For some reason, I'm roadblocking on every attempt to create what should be simple to do. Thoughts?
Update to answer questions
I know some people do similar to the following, but this isn't as flexible as I'd like:
$array = #()
$object = New-Object -TypeName PSObject
$object | Add-Member -Name 'Name' -MemberType Noteproperty -Value 'Joe'
$object | Add-Member -Name 'Age' -MemberType Noteproperty -Value 32
$object | Add-Member -Name 'Info' -MemberType Noteproperty -Value 'something about him'
$array += $object
This requires the values to be present for all three members when creating each $object. I was thinking the init would look more along the lines of (pseudocode):
$MyRecord = {
Signature as string
PID as integer
}
$RecArray = array of $MyRecord
That's notably a bad mashup of Delphi and Powershell. But would create a fully structured array, addressable as noted up top.
A PSv5+ solution that uses a PS class and a generic list ([System.Collections.Generic.List[]]) to store the instances (loosely speaking, an array that can grow efficiently).
# Your custom type.
class MyRecord {
[string] $Signature
[int] $PID
}
# If you want a list that can grow efficiently,
# use [System.Collections.Generic.List[]]
$RecList = [System.Collections.Generic.List[MyRecord]]::new()
# Add a new instance...
$RecList.Add([MyRecord]::new())
# ... and initialize it.
$RecList[0].Signature = 'sig1'
$RecList[0].Pid = 666
# ... or initialize it first, and then add it.
# Note the use of a cast from a hashtable with the property values.
$newRec = [MyRecord] #{ Signature = 'sig2'; PID = 667}
$RecList.Add($newRec)
# Output the list
$RecList
The above yields:
Signature PID
--------- ---
sig1 666
sig2 667
As for removing objects from the list:
To remove by index, use .RemoveAt(); an out-of-range index throws an error:
$RecList.RemoveAt(1)
To remove by object already stored in the list, use .Remove().
Note that the [bool] return value indicates whether the value was
actually removed (if the object wasn't in the list, the operation is
a no-op and $False is returned)
$actuallyRemoved = $RecList.Remove($newRec)
For details, see the docs.
You want to create a custom object.
You create an object that has all the properties you need. Then you create a collection, and you stuff instances of your new object into the collection. Here's an example:
$WIP = #()
$o = New-Object –TypeName PSObject
Add-Member -InputObject $o –MemberType NoteProperty –Name signature –Value 'foo'
Add-Member -InputObject $o –MemberType NoteProperty –Name data –Value 'bar'
$WIP += $o
$WIP[0].signature
$WIP[0].data
You'd need to execute the New-Object and Add-Member statements for each object you're creating.
So here's working example of how You can get something like this working:
$list=#()
1..100|foreach-object{
$obj=""|select signature,pid
$obj.signature="$_ signature"
$obj.pid="$_ PID"
$list+=$obj
}
With the object created this way - You can do $list[43].signature and it does work.
What exactly do you mean by "Dynamic"?
$array = #(
# Some type of loop to create multiple items foreach/for/while/dowhile
foreach ($item in $collection) {
New-Object psobject -Property #{
Signature = 'data'
PID = 'data'
}
}
)
Or you can manually add objects like so
$array = #()
# Later in code
$array += New-object psobject #{
Signature = 'data'
PID = 'data'
}
Then you can access each item like so:
$array[1].Signature
$array[1].PID
There is no real difference between this an what you have already been shown but I think this gives you what you are asking for (even though it is not the powershelly way to do things).
$object = "New-Object PSCustomObject -Property #{'Name' = ''; 'Age' = [int]}"
$array = 1..100 | %{Invoke-Expression $object}
$array[0].Name = 'Joe'
$array[0].Age = 12
$array[0]
You can use a hashtable with indices as keys, and your hashtable as values. It's pretty easy to work with.
$WIP = #{
0 = #{
signature = 'signature 0'
PID = 'PID 0'
}
1 = #{
signature = 'signature 1'
PID = 'PID 1'
}
}
You can add any index you want.
$WIP[12] = #{
signature = "signature 12"
PID = "PID 12"
}
$WIP[12].PID
# PID 12
You can initialize both, any, or none.
$WIP[76] = #{
signature = "signature 76"
}
$WIP[76].signature
# signature 76
$WIP[76].PID
# $null
Count gives you number of "active" elements.
$WIP.Count
# 4
Say I have a PowerShell array $Sessions = #() which I am going to fill with PSCustomObjects. How can I add a custom property to the array itself? E.g. so I can have $Sessions.Count which is built-in and $Sessions.Active which I want to set to the active session count.
I know that I can add properties to PSCustomObjects (in a dirty way) with
$MyCustomObject = "" | Select-Object Machine, UserName, SessionTime
but though doing so on an array would not result in the property being added.
So how can I achieve my goal? Is there any way to create a custom array?
The answer to your question as stated would be to just use Add-Member on the array object.
Add-Member -InputObject $sessions -MemberType NoteProperty -Name "State" -Value "Fabulous"
Adding a property to each element after you created the object is similar.
$sessions | ForEach-Object{
$_ | Add-Member -MemberType NoteProperty -Name "State" -Value "Fabulous"
}
This of course comes with a warning (that I forgot about). From comments
Beware, though, that appending to that array ($sessions += ...) will replace the array, thus removing the additional property.
Ansgar Wiechers
Depending on your use case there are other options to get you want you want. You can save array elements into distinct variables:
# Check the current object state
$state = $object.Property .....
# Add to the appropriate array.
if($state -eq "Active"){
$activeSessions += $object
} else {
$inactiveSessions += $object
}
Or you could still store your state property and post process with Where-Object as required:
# Process each inactive session
$sessions | Where-Object{$_.State -eq "Active"} | ForEach-Object{}
To avoid the destroying / recreating array issue, which can be a performance hog, you could also use an array list instead.
$myArray = New-Object System.Collections.ArrayList
Add-Member -InputObject $myArray -MemberType ScriptMethod -Name "NeverTellMeTheOdds" -Value {
$this | Where-Object{$_ % 2 -ne 0}
}
$myArray.AddRange(1..10)
$myArray.NeverTellMeTheOdds()
Notice that the array had its member added then we added its elements.
As Matt commented, you can use the Add-Member on an enumerable type by supplying it as a positional argument to the -InputObject parameter.
To allow for resizing after adding the new property, use a generic List instead of #():
$list = [System.Collections.Generic.List[psobject]]::new()
$list.AddRange(#(
[pscustomobject]#{SessionId = 1; Active = $true}
[pscustomobject]#{SessionId = 2; Active = $false}
[pscustomobject]#{SessionId = 3; Active = $true}
) -as [psobject[]])
Add-Member -InputObject $list -MemberType ScriptProperty -Name ActiveSessionCount -Value {
return #($this |? Active -eq $true).Count
}
Now you can retrieve the active session count easily:
PS C:\> $list.ActiveSessionCount
2
Is it possible to display the results of a PowerShell Compare-Object in two columns showing the differences of reference vs difference objects?
For example using my current cmdline:
Compare-Object $Base $Test
Gives:
InputObject SideIndicator
987654 =>
555555 <=
123456 <=
In reality the list is rather long. For easier data reading is it possible to format the data like so:
Base Test
555555 987654
123456
So each column shows which elements exist in that object vs the other.
For bonus points it would be fantastic to have a count in the column header like so:
Base(2) Test(1)
555555 987654
123456
Possible? Sure. Feasible? Not so much. PowerShell wasn't really built for creating this kind of tabular output. What you can do is collect the differences in a hashtable as nested arrays by input file:
$ht = #{}
Compare-Object $Base $Test | ForEach-Object {
$value = $_.InputObject
switch ($_.SideIndicator) {
'=>' { $ht['Test'] += #($value) }
'<=' { $ht['Base'] += #($value) }
}
}
then transpose the hashtable:
$cnt = $ht.Values |
ForEach-Object { $_.Count } |
Sort-Object |
Select-Object -Last 1
$keys = $ht.Keys | Sort-Object
0..($cnt-1) | ForEach-Object {
$props = [ordered]#{}
foreach ($key in $keys) {
$props[$key] = $ht[$key][$_]
}
New-Object -Type PSObject -Property $props
} | Format-Table -AutoSize
To include the item count in the header name change $props[$key] to $props["$key($($ht[$key].Count))"].
I have an array of custom objects:
$report = #()
foreach ($person in $mylist)
{
$objPerson = New-Object System.Object
$objPerson | Add-Member -MemberType NoteProperty -Name Name -Value $person.Name
$objPerson | Add-Member -MemberType NoteProperty -Name EmployeeID
$objPerson | Add-Member -MemberType NoteProperty -Name PhoneNumber
$report += $objPerson
}
Note that I haven't set values for the last two properties. The reason I've done this is because I'm trying to produce a matrix where I'll easily be able to see where these are blanks (although I could just set these to = "" if I have to).
Then, I want to iterate through a second dataset and update these values within this array, before exporting the final report. E.g. (this bit is pretty much pseudo code as I have no idea how to do it:
$phonelist = Import-Csv .\phonelist.csv
foreach ($entry in $phonelist)
{
$name = $entry.Name
if ($report.Contains(Name))
{
# update the PhoneNumber property of that specific object in the array with
# another value pulled out of this second CSV
}
else
{
# Create a new object and add it to the report - don't worry I've already got
# a function for this
}
}
I'm guessing for this last bit I probably need my if statement to return an index, and then use that index to update the object. But I'm pretty lost at this stage.
For clarity this is a simplified example. After that I need to go through a second file containing the employee IDs, and in reality I have about 10 properties that need updating all from different data sources, and the data sources contain different lists of people, but with some overlaps. So there will be multiple iterations.
How do I do this?
I would read phonelist.csv into a hashtable, e.g. like this:
$phonelist = #{}
Import-Csv .\phonelist.csv | ForEach-Object { $phonelist[$_.name] = $_.number }
and use that hashtable for filling in the phone numbers in $report as you create it:
$report = foreach ($person in $mylist) {
New-Object -Type PSObject -Property #{
Name = $person.Name
EmployeeID = $null
PhoneNumber = $phonelist[$person.Name]
}
}
You can still check the phone list for entries that are not in the report like this:
Compare-Object $report.Name ([array]$phonelist.Keys) |
Where-Object { $_.SideIndicator -eq '=>' } |
Select-Object -Expand InputObject
I would iterate over the $phonelist two times. The first time, you could filter all phone entities where the name is in your $myList and create the desired object:
$phonelist = import-cse .\phonelist.csv
$report = $phonelist | Where Name -in ($mylist | select Name) | Foreach-Object {
[PSCustomObject]#{
Name = $_.Name
PhoneNumber = $_.PhoneNumber
EmployeeID = ''
}
}
The second time you filter all phone entities where the name is not in $myList and create the new object:
$report += $phonelist | Where Name -NotIn ($mylist | select Name) | Foreach-Object {
#Create a new object and add it to the report - don't worry I've already got a function for this
}
I have created a custom object called $info and moving it to an array $arr ,
How is it possible to remove one member along with its all properties ?
My script:
Get-Process | ForEach-Object{
$info = New-Object -TypeName PSObject
$info | Add-Member -Type NoteProperty -Name Process -Value $_.processname
$info | Add-Member -Type NoteProperty -Name ID -Value $_.id
$arr += $info
}
$arr | ft -AutoSize
The result looks like this :
Process ID
------- --
ApplicationFrameHost 38556
AppVShNotify 9792
armsvc 2336
atieclxx 6944
atiesrxx 1844
audiodg 59432
CcmExec 3988
chrome 46068
How can I remove one particular member for example "audiodg 59432" gets removed
audiodg 59432
Your terminology is a bit incorrect here. A member is on an individual object. When you use Add-Member above you're adding properties to each individual object, then you're returning an array of objects.
You're asking how to remove an individual object from the array.
In PowerShell you cannot remove an item from an array. You could instead filter the array based on some criteria and create a new one:
$newArr = $arr | Where-Object { $_.Name -ne 'audiodg' }
# or
$newArr = $arr | Where-Object { $_.ID -ne 59432 }