Transpose JSON array index column values into rows & export to CSV - arrays

I have a JSON file with below format:
[
{
"id": "12345",
"name": "test_group",
"members":
[
{
"id": "11111",
"name": "test_member_1"
},
{
"id": "22222",
"name": "test_member_2"
},
{
"id": "33333",
"name": "test_member_3"
}
]
}
]
My requirement is to convert these into below format/output to CSV:
groupId,groupName,memberId,memberName
12345,test_group,11111,test_member_1
12345,test_group,22222,test_member_2
12345,test_group,33333,test_member_3
I am able to get the array index values as concatenated strings in one column using below code, but need the output in above format only:
$jsonfile = 'C:\newjsonfile.json'
$outputcsv = 'C:\newcsvfile.csv'
$fields=#(
#{Name='groupId';Expression={$_.id}}
,#{Name='groupName';Expression={$_.name}}
,#{Name='memberId';Expression={$_.members.id, -join '|~|'}}
,#{Name='memberName';Expression={$_.members.name -join '|~|'}}
)
(Get-Content $jsonfile -Raw | ConvertFrom-Json) |
Select-Object -Property $fields |
Export-Csv $outputcsv -NoTypeInformation -Force
CSV Output from above query:
groupId, groupName, memberId, memberName
----- --------- -------- ----------
12345, test_group, {11111 |~| 22222 |~| 33333}, {test_member_1 |~| test_member_2 |~| test_member_3}
Please help/suggest in resolving this query in PowerShell.
Current version is:
PS C:\WINDOWS\system32> $PSVersionTable.PSVersion
Major Minor Build Revision
----- ----- ----- --------
5 1 17134 407

You want to combine information from a parent object with information from nested child objects. To do that you need to store the information from the parent in variables and then inject it when processing the child objects:
(Get-Content $jsonfile -Raw | ConvertFrom-Json) | ForEach-Object {
$gid = $_.id
$name = $_.name
$_.members |
Select-Object #{n='groupId';e={$gid}},
#{n='groupName';e={$name}},
#{n='memberId';e={$_.id}},
#{n='memberName';e={$_.name}}
} | Export-Csv $outputcsv -NoType -Force

Related

Creating a 2D Array with escaped characters

I have a CSV, let's call it test.csv
"country","city","population","remarks"
"germany","munih","1472000","i am a
line
brake"
"italy","rome","2873000","<>|'"
"spain","madrid","6642000","!"§$%&/("
I need a full HTML escaped 2D JS-array without headers and names as like below:
"germany","munih","1472000","i am a \nline\nbrake"
respectively:
var MyVar = [["germany","munih","1472000","i am a \nline\nbrake"],["italy","rome","2873000","<>|'"],["spain","madrid","6642000","!§$%&/(""]];
ConvertTo-Json can escape, but did not find any easy way to produce an 2D-Array without Object Names:
Import-Csv ".\test.csv" | ConvertTo-Json -EscapeHandling EscapeHtml
[
{
"country": "germany",
"city": "munih",
"population": "1472000",
"remarks": "i am a \r\nline\r\nbrake"
},
{
"country": "italy",
"city": "rome",
"population": "2873000",
"remarks": "\u003c\u003e|\u0027"
},
{
"country": "spain",
"city": "madrid",
"population": "6642000",
"remarks": "!§$%\u0026/(\u0022"
}
]
ConverTo-CSV does a good job, but can't escape:
Import-Csv ".\test.csv" | ConvertTo-Csv -UseQuotes Always
"country","city","population","remarks"
"germany","munih","1472000","i am a
line
brake"
"italy","rome","2873000","<>|'"
"spain","madrid","6642000","!§$%&/("""
Following routine works. But I believe, there must be easier way.
$t1 = Import-Csv ".\test.csv"
$arr_len = $t1.Count
$obj_len = $t1[0].Psobject.Properties.value.count
$str_builder = [System.Text.StringBuilder]::new()
for ($i=0; $i -lt $arr_len; $i++){
#first outer loop
if ($i -eq 0){[void]$str_builder.Append('var MyVar = [')}
for ($j=0; $j -lt $obj_len; $j++){
#first inner loop
if ($j -eq 0){[void]$str_builder.Append('[')}
#$t1[$i].Psobject.Properties.Name[$j]
$each = $t1[$i].Psobject.Properties.Value[$j]
$each_esc = ([System.Net.WebUtility]::HtmlEncode($each)) -replace("`r`n|`r|'n","\n")
[void]$str_builder.Append('"')
[void]$str_builder.Append($each_esc)
[void]$str_builder.Append('"')
#last inner loop
if ($j -eq ($obj_len-1)){
if ($i -ne ($arr_len-1)){
[void]$str_builder.Append('],')
} else {
[void]$str_builder.Append(']')
}
} else{
[void]$str_builder.Append(',')
}
}
#last outer loop
if ($i -eq ($arr_len-1)){[void]$str_builder.Append('];')}
}
$final = $str_builder.ToString()
[void]$str_builder.Clear()
Any hints, ideas? Thx :)
You could simplify the code like:
Add-Type -AssemblyName System.Web
$data = Import-Csv -Path 'D:\Test\test.csv'
$jsData = foreach ($item in $data) {
# create an array of quoted encoded strings
$values = $item.PsObject.Properties.Value | ForEach-Object {
'"{0}"' -f [System.Web.HttpUtility]::HtmlEncode(($_ -replace '\r?\n', '\n'))
}
# output the encoded strings grouped inside square brackets
'[{0}]' -f ($values -join ',')
}
# combine the resulting data into a 2D array
$result = 'var MyVar = [{0}];' -f ($jsData -join ',')
$result
Output:
var MyVar = [["germany","munih","1472000","i am a \nline\nbrake"],["italy","rome","2873000","<>|'"],["spain","madrid","6642000","!§$%&/(""]];
Right from the given input it isn't possible to produce the required output.
Youe have to create a nested list first, then the json document.
Try the following:
$list = [System.Collections.Generic.List[psobject]]::new()
foreach($item in $input) {
$nestedList = [System.Collections.Generic.List[string]]::new()
$itemProps = $item.psobject.properties
foreach($itemProp in $itemProps) {
$nestedList.Add($itemProp.Value)
}
$list.Add($nestedList)
}
$list | ConvertTo-Json -EscapeHandling EscapeHtml
<#
$PSVersionTable.PSVersion
Major Minor Patch PreReleaseLabel BuildLabel
----- ----- ----- --------------- ----------
7 1 0
#>
$fieldsCount = 4
$rawText = #'
"country","city","population","remarks"
"germany","munih","1472000","i am a
line
brake"
"italy","rome","2873000","<>|'"
"spain","madrid","6642000","!"§$%&/("
'#
$rawArray = $rawText.Trim("`"") -split "`r`n" -join "," -split '","'
$table = $(for ($i = $fieldsCount; $i -lt $rawArray.Count; $i += $fieldsCount){
[PSCustomObject]#{
country = $rawArray[$i]
city = $rawArray[$i+1]
population = $rawArray[$i+2]
remarks = $rawArray[$i+3]
}
})
$table | ConvertTo-Json
<#
country city population remarks
------- ---- ---------- -------
germany munih 1472000 i am a ,line,brake
italy rome 2873000 <>|'
spain madrid 6642000 !"§$%&/(
[
{
"country": "germany",
"city": "munih",
"population": "1472000",
"remarks": "i am a ,line,brake"
},
{
"country": "italy",
"city": "rome",
"population": "2873000",
"remarks": "<>|'"
},
{
"country": "spain",
"city": "madrid",
"population": "6642000",
"remarks": "!\"§$%&/("
}
]
#>

PowerShell Array, adding new records where one property is an array

I am unable to use ArrayList or avoid using += for array manipulation. Wishing that powerShell had a universal add or append available.
I have the below JSON array for $aksAppRules.RulesText
[{
"Name": "A2B",
"Description": null,
"SourceAddresses": [
"10.124.176.0/21",
"10.124.184.0/21"
],
"TargetFqdns": [
"*.github.com",
"*.grafana.com",
"*.trafficmanager.net",
"*.loganalytics.io",
"*.applicationinsights.io",
"*.azurecr.io",
"*.debian.org"
],
"FqdnTags": [],
"Protocols": [
{
"ProtocolType": "Https",
"Port": 443
}
],
"SourceIpGroups": []
},
{
"Name": "Y2office365",
"Description": null,
"SourceAddresses": [
"10.124.176.0/21",
"10.124.184.0/21"
],
"TargetFqdns": [
"smtp.office365.com"
],
"FqdnTags": [],
"Protocols": [
{
"ProtocolType": "Http",
"Port": 25
},
{
"ProtocolType": "Http",
"Port": 587
}
],
"SourceIpGroups": []
}
]
I managed to make this work with the below powershell snippet
$new_list = #()
$collectionRules = $aksAppRules.RulesText | ConvertFrom-Json
foreach ($rule in $collectionRules) {
$protoArray = #()
ForEach ($protocol in $rule.Protocols) {
$protoArray += $protocol.ProtocolType + "," + $protocol.Port
}
#$new_list += , #($rule.Name, $rule.SourceAddresses, $rule.TargetFqdns, $protoArray )
# the 'comma' right after += in below line tells powershell to add new record.
$new_list += , #{Name=$rule.Name;SourceAddresses=$rule.SourceAddresses; TargetFqdns=$rule.TargetFqdns;Protocol=$protoArray}
}
$new_list | ConvertTo-Json | ConvertFrom-Json | select Name, SourceAddresses, TargetFqdns, Protocol| Convert-OutputForCSV -OutputPropertyType Comma | Export-Csv .\test.csv
The CSV looks like
I am unable to do this using Arraylists and without using += as I heard it is inefficient with large arrays.
I have to copy things to a new array because I have to change the key:value format of the original "Protocols" to a 2 d array.
Any pointers will be appreciated.
Yes, you should avoid using the increase assignment operator (+=) to create a collection as it exponential expensive. Instead you should use the pipeline
collectionRules = $aksAppRules.RulesText | ConvertFrom-Json
foreach ($rule in $collectionRules) {
[pscustomobject]#{
Name = $rule.Name
SourceAddresses = $rule.SourceAddresses
TargetFqdns = $rule.TargetFqdns
Protocol = #(
ForEach ($protocol in $rule.Protocols) {
$protocol.ProtocolType + "," + $protocol.Port
}
)
}
} | Convert-OutputForCSV -OutputPropertyType Comma | Export-Csv .\test.csv
Note 1: I have no clue why you are doing | ConvertTo-Json | ConvertFrom-Json, so far I can see there is no need for this if you use a [pscustomobject] rather than a [Hashtabe] type.
Note 2: I no clue what the function Convert-OutputForCSV is doing and suspect that isn't required either (but left it in).

storing foreach loop output to an array

Have a JSON array with same key values , want to loop through those and get one key of the same value of the array and store the output to an array
{
"contents":[
{
"name":"windows-Instance",
"Buildid":"1234",
"Buildtime":"1563350400238"
},
{
"name":"linux-Instance",
"Buildid":"1454",
"Buildtime":"1563264000198"
},
{
"name":"linux-Instance",
"Buildid":"1278685",
"Buildtime":"1563177600092"
}
]
}
Here is code i tried and doesn't give any output.
$result = #()
foreach ($Builtime in $contents) {
}
return $result
You can do it like this:
First convert the json string to an object:
$contents = '{
"contents":[
{
"name":"windows-Instance",
"Buildid":"1234",
"Buildtime":"1563350400238"
},
{
"name":"linux-Instance",
"Buildid":"1454",
"Buildtime":"1563264000198"
},
{
"name":"linux-Instance",
"Buildid":"1278685",
"Buildtime":"1563177600092"
}
]
}' | ConvertFrom-Json
Than retrieve the required properties:
$result = $contents | Select-Object -ExpandProperty "contents" | Select-Object -ExpandProperty "Buildtime"
You could loop over the JSON and create a custom-object that would allow you to further "manipulate" the data if needed.
Otherwise the example of #mhu is a perfect onliner
$json_content = (Get-Content ".\sample.json") | ConvertFrom-Json
$result = foreach ($content in $json_content."contents") {
[PSCustomObject]#{
"Buildtime" = $content."Buildtime"
}
}

convert 1 dimension strings array into json file

Perhaps it's easy question and will be banned but I spend hours today and not make it work as I expected.
so I have such object:
data_1 data_2 ... abra_12 ...
I want convert it into json like this:
[
{"data_1" : "data_1"},
{"data_2" : "data_2"},
...
]
I tried this:
$result = Get-ChildItem $Path2Search -recurse | Select-String -Pattern '(?<=locI(.*)\(\").+?(?=\")' -casesensitive | foreach {$_.matches} | select value | ConvertTo-Json | Out-File $save2file
But I'm getting this:
{
"Value": "data_1"
},
{
"Value": "data_2"
},
While I want it like this:
{
"data_1": "data_1"
},
{
"data_2": "data_2"
},
Any advice how? :)
You can first shape the result in a list of key-values then convert it to json.
For example if your input is a comma separated string, you can use:
$input = "data_1, data_2, data_3"
$input.Split(',').Trim() |
ForEach-Object {#{$_=$_}} |
ConvertTo-Json |
Out-File "C:\test.txt"
Or if the input is a string array:
$input = #("data_1", "data_2", "data_3")
$input | ForEach-Object {#{$_=$_}} |
ConvertTo-Json |
Out-File "C:\test.txt"
And the result would be:
[
{
"data_1": "data_1"
},
{
"data_2": "data_2"
},
{
"data_3": "data_3"
}
]
While I prefer to use above solution for creating json result, but you also can rely on string manipulation as well:
$input = "data_1, data_2, data_3"
#"
[
`t$(($input -split '\s*,\s*' |%{"{ `"$_`" : `"$_`" }"}) -join ",`n`t")
]
"#

Looping through $json variable and grabbing each id to add to an array

Given that I am processing the following $json variable, how would I grab each server id (i.e. 215d1109-216d-48c3-af8e-998bb9bc3ca0 and 440cf918-3ee0-4143-b289-f63e1d2000e6 in this case) and put it into an array?
Right now $obj.servers.id returns nothing (as expected), but $obj.servers.id[0] returns an error message.
clear
[System.Reflection.Assembly]::LoadWithPartialName("System.Web.Extensions") | out-null
$json = '
{
"servers": [
{
"admin_password": "qpYU66rKxmnK",
"id": "215d1109-216d-48c3-af8e-998bb9bc3ca0",
"links": [
{
"href": "http://openstack.example.com/v3/servers/<id>",
"rel": "self"
},
{
"href": "http://openstack.example.com/servers/<id>",
"rel": "bookmark"
}
]
},
{
"admin_password": "wfksH3GTTseP",
"id": "440cf918-3ee0-4143-b289-f63e1d2000e6",
"links": [
{
"href": "http://openstack.example.com/v3/servers/<id>",
"rel": "self"
},
{
"href": "http://openstack.example.com/servers/<id>",
"rel": "bookmark"
}
]
}
]
}
'
$ser = New-Object System.Web.Script.Serialization.JavaScriptSerializer
$obj = $ser.DeserializeObject($json)
$obj.servers.id;
Please note that I am using Powershell 2.0.
You're misunderstanding the structure of your data. $obj.servers is an array with two fields, each of which contains a dictionary, not an object containing an array id. You need to make the index access like this:
$obj.servers[0].id
$obj.servers.id[0] will certainly throw an error, because the array object $obj.servers doesn't have a property id, so in PowerShell v2 $obj.servers.id returns $null, and index access on a null array fails (as you'd expect).
Demonstration:
PS C:\> [void][Reflection.Assembly]::LoadWithPartialName('System.Web.Extensions')
$json = #'
...
'#
PS C:\> $ser = New-Object Web.Script.Serialization.JavaScriptSerializer
PS C:\> $obj = $ser.DeserializeObject($json)
PS C:\> $obj.servers.GetType().FullName
System.Object[]
PS C:\> $obj.servers.id -eq $null
True
PS C:\> $obj.servers[0].GetType().Name
Dictionary`2
PS C:\> $obj.servers[0] | Format-Table -AutoSize
Key Value
--- -----
admin_password qpYU66rKxmnK
id 215d1109-216d-48c3-af8e-998bb9bc3ca0
links {System.Collections.Generic.Dictionary`2[System.Str...
PS C:\> $obj.servers[0].id
215d1109-216d-48c3-af8e-998bb9bc3ca0
To extract all IDs simply pipe the array into a ForEach-Object loop where you echo the id property:
PS C:\> $ids = $obj.servers | % { $_.id }
PS C:\> $ids
215d1109-216d-48c3-af8e-998bb9bc3ca0
440cf918-3ee0-4143-b289-f63e1d2000e6

Resources