I'm totally brand new to JSON and jq so this might seem like a simple question.
I'd like to change an array of numbers into an array of objects with a key for each value (number).
Let's say I have a JSON file like this:
{
"foo": [1519739200, 1519739600, 1519740000]
}
Then my desired output would be:
{
"foo": [
{
"id": 1519739200
},
{
"id": 1519739600
},
{
"id": 1519740000
},
]
}
So far everything I've seen was connected with adding new keys with values to existing object or merging two arrays into few objects. I know that I can add more keys into already existing object but how can I add keys to an array? I assume I have to change array elements into objects first but how do I do it?
Thank you for answer.
Check this,
https://jqplay.org/s/xF9DyVbhXD
{ foo : [ { id : .foo[] } ] }
There are a few ways to go about it.
First you want to create a new object for each item in foo:
$ jq -c '{ id: .foo[] }'
{"id":1519739200}
{"id":1519739600}
{"id":1519740000}
You can then rebuild the "shape" you had - first with [ ... ]
$ jq -c '[ { id: .foo[] } ]'
[{"id":1519739200},{"id":1519739600},{"id":1519740000}]
Then the { foo: }
$ jq -c '{ foo: [ { id: .foo[] } ] }'
{"foo":[{"id":1519739200},{"id":1519739600},{"id":1519740000}]}
Another option is to use |= to modify/update .foo directly.
$ jq -c '.foo |= [{id: .[]}]'
{"foo":[{"id":1519739200},{"id":1519739600},{"id":1519740000}]}
I am unable to use ArrayList or avoid using += for array manipulation. Wishing that powerShell had a universal add or append available.
I have the below JSON array for $aksAppRules.RulesText
[{
"Name": "A2B",
"Description": null,
"SourceAddresses": [
"10.124.176.0/21",
"10.124.184.0/21"
],
"TargetFqdns": [
"*.github.com",
"*.grafana.com",
"*.trafficmanager.net",
"*.loganalytics.io",
"*.applicationinsights.io",
"*.azurecr.io",
"*.debian.org"
],
"FqdnTags": [],
"Protocols": [
{
"ProtocolType": "Https",
"Port": 443
}
],
"SourceIpGroups": []
},
{
"Name": "Y2office365",
"Description": null,
"SourceAddresses": [
"10.124.176.0/21",
"10.124.184.0/21"
],
"TargetFqdns": [
"smtp.office365.com"
],
"FqdnTags": [],
"Protocols": [
{
"ProtocolType": "Http",
"Port": 25
},
{
"ProtocolType": "Http",
"Port": 587
}
],
"SourceIpGroups": []
}
]
I managed to make this work with the below powershell snippet
$new_list = #()
$collectionRules = $aksAppRules.RulesText | ConvertFrom-Json
foreach ($rule in $collectionRules) {
$protoArray = #()
ForEach ($protocol in $rule.Protocols) {
$protoArray += $protocol.ProtocolType + "," + $protocol.Port
}
#$new_list += , #($rule.Name, $rule.SourceAddresses, $rule.TargetFqdns, $protoArray )
# the 'comma' right after += in below line tells powershell to add new record.
$new_list += , #{Name=$rule.Name;SourceAddresses=$rule.SourceAddresses; TargetFqdns=$rule.TargetFqdns;Protocol=$protoArray}
}
$new_list | ConvertTo-Json | ConvertFrom-Json | select Name, SourceAddresses, TargetFqdns, Protocol| Convert-OutputForCSV -OutputPropertyType Comma | Export-Csv .\test.csv
The CSV looks like
I am unable to do this using Arraylists and without using += as I heard it is inefficient with large arrays.
I have to copy things to a new array because I have to change the key:value format of the original "Protocols" to a 2 d array.
Any pointers will be appreciated.
Yes, you should avoid using the increase assignment operator (+=) to create a collection as it exponential expensive. Instead you should use the pipeline
collectionRules = $aksAppRules.RulesText | ConvertFrom-Json
foreach ($rule in $collectionRules) {
[pscustomobject]#{
Name = $rule.Name
SourceAddresses = $rule.SourceAddresses
TargetFqdns = $rule.TargetFqdns
Protocol = #(
ForEach ($protocol in $rule.Protocols) {
$protocol.ProtocolType + "," + $protocol.Port
}
)
}
} | Convert-OutputForCSV -OutputPropertyType Comma | Export-Csv .\test.csv
Note 1: I have no clue why you are doing | ConvertTo-Json | ConvertFrom-Json, so far I can see there is no need for this if you use a [pscustomobject] rather than a [Hashtabe] type.
Note 2: I no clue what the function Convert-OutputForCSV is doing and suspect that isn't required either (but left it in).
I'm trying to format some data that consists of json objects that includes some identifying information along with an array of one or more json objects, I would like the result to be one line of data per array element where each line should include some fields from the array element and some fields from the identifying information.
My sample data is below:
{
"eventCreation": {
"timeStamp": "2020-06-06T15:07:20Z",
"epoch": 1591456040
},
"eventData": {
"applName": "SampleApp",
"channelName": "SYSTEM.DEF.SVRCONN",
"connectionName": "127.0.0.1",
"channelType": "Svrconn",
"remoteProduct": "MQJM",
"remoteVersion": "09010005",
"activityTrace": [
{
"operationId": "Get",
"operationTime": "11:07:18",
"qmgrOpDuration": 102,
"reasonCode": {
"name": "No Msg Available",
"value": 2033
},
"objectName": "SYSTEM.DEFAULT.LOCAL.QUEUE"
},
{
"operationId": "Cb",
"operationTime": "11:07:18",
"qmgrOpDuration": 10,
"reasonCode": {
"name": "None",
"value": 0
},
"objectName": "SYSTEM.DEFAULT.LOCAL.QUEUE"
},
{
"operationId": "Cb",
"operationTime": "11:07:18",
"qmgrOpDuration": 12,
"reasonCode": {
"name": "None",
"value": 0
},
"objectName": "SYSTEM.DEFAULT.LOCAL.QUEUE"
}
]
}
}
I would like to get an output like this:
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","11:07:18","Get",102,"SYSTEM.DEFAULT.LOCAL.QUEUE",2033
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","11:07:18","Cb",10,"SYSTEM.DEFAULT.LOCAL.QUEUE",0
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","11:07:18","Cb",12,"SYSTEM.DEFAULT.LOCAL.QUEUE",0
I can pick any one element of the array and get it to print 3 lines, but if I add a 2nd element it will print 9 lines, 3rd element prints 27, etc.
For example:
jq -r '{channelName: .eventData.channelName, channelType: .eventData.channelType, connectionName: .eventData.connectionName, applName: .eventData.applName, remoteProduct: .eventData.remoteProduct, remoteVersion: .eventData.remoteVersion, operationId: .eventData.activityTrace[].operationId}|[.[]]|#csv' TEST.json
Will produce this:
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Get"
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb"
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb"
If I add a second like this:
jq -r '{channelName: .eventData.channelName, channelType: .eventData.channelType, connectionName: .eventData.connectionName, applName: .eventData.applName, remoteProduct: .eventData.remoteProduct, remoteVersion: .eventData.remoteVersion, operationId: .eventData.activityTrace[].operationId, qmgrOpDuration: .eventData.activityTrace[].qmgrOpDuration}|[.[]]|#csv' TEST.json
Will produce this:
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Get",102
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Get",10
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Get",12
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",102
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",10
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",12
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",102
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",10
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",12
Using your approach, the following is a solution:
.eventData
| ({channelName, channelType, connectionName, applName, remoteProduct, remoteVersion}
+ ( .activityTrace[]
| { operationTime, operationId, qmgrOpDuration, objectName, v: .reasonCode.value}))
| [.[]]
| #csv
The key is to iterate just once.
Notice also that this solution achieves its brevity in part by using the fact that {foo: .foo} can be abbreviated to {foo}.
A slightly more efficient approach
.eventData
| [.channelName, .channelType, .connectionName, .applName, .remoteProduct, .remoteVersion]
+ ( .activityTrace[]
| [.operationTime, .operationId, .qmgrOpDuration, .objectName, .reasonCode.value] )
| #csv
.operationTime first
.eventData
| [.channelName, .channelType, .connectionName, .applName, .remoteProduct, .remoteVersion] as $x
| ( .activityTrace[]
| [.operationTime]
+ $x
+ [.operationId, .qmgrOpDuration, .objectName, .reasonCode.value] )
| #csv
I have a JSON with the following content:
{
"data": [
{
"name": "Test",
"program": {
"publicAccess": "--------",
"externalAccess": false,
"userGroupAccesses": [
{
"access": "r-------"
},
{
"access": "rw------"
}
],
"id": "MmBqeMLIC2r"
},
"publicAccess": "rw------"
}
]
}
And I want to delete all keys (recursively) which match publicAccess or userGroupAccesses, so that my JSON looks like this:
{
"data": [
{
"name": "Test",
"program": {
"externalAccess": false,
"id": "MmBqeMLIC2r"
}
}
]
}
I've copied jq's builtin walk function from source.
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
# My Code
walk(if (type == "object" and .publicAccess)
then del(.publicAccess)
elif (type == "array" and .userGroupAccesses)
then del(.userGroupAccesses)
else
.
end )
Gives me jq: error (at <stdin>:2622): Cannot index array with string "userGroupAccesses". Also if I use .userGroupAccesses[] - How do I get the result?
Snippet on jqplay: https://jqplay.org/s/1m7wAeHMTu
Your problem is when type == "array" is true . will be an array so .userGroupAccesses won't work. What you want to do is focus on the case when . is an object. In your call to walk you only need to check for type == "object" and then remove the members you don't want. e.g.
walk(if type == "object" then del(.publicAccess, .userGroupAccesses) else . end)
Try it online at jqplay.org
You can also solve this without walk by using Recursive Descent .. e.g.
del(.. | .publicAccess?, .userGroupAccesses?)
Try it online at jqplay.org
I have a json file that I use for work that I need to parse that is in the following format:
(^)#(^)#(^)#(^)bminter#ubuntu:~$ cat jqtest
{
"files":[
{
"BLOCK1":{
"SUBBLOCK1":{
"akey1":"avalue1",
"bkey1":"bvalue1",
"ckey1":"cvalue1"
},
"dkey1":"dvalue1",
"key":"evalue1"
}
},
{
"BLOCK-2":{
"SUBBLOCK2":{
"akey2":"avalue2",
"bkey2":"bvalue2"
},
"ckey2":"cvalue2",
"key":"dvalue2"
}
},
{
"BLOCK-A":{
"SUBBLOCK2":{
"akey2":"avalue2",
"bkey2":"bvalue2"
},
"ckey2":"cvalue2",
"key":"dvalue2"
}
}],
"NOBLOCK":"value",
"key":"NOBLOCKvalue"
}
So it's an array nested within a json file. jq .[] jqtest gives me everything in the file. Even the data outside the array. Except, outside the array, I'm only given the values not the keys:
(^)#(^)#(^)#(^)bminter#ubuntu:~$ jq .[] jqtest
[
{
"BLOCK1": {
"SUBBLOCK1": {
"akey1": "avalue1",
"bkey1": "bvalue1",
"ckey1": "cvalue1"
},
"dkey1": "dvalue1",
"key": "evalue1"
}
},
{
"BLOCK-2": {
"SUBBLOCK2": {
"akey2": "avalue2",
"bkey2": "bvalue2"
},
"ckey2": "cvalue2",
"key": "dvalue2"
}
},
{
"BLOCK-A": {
"SUBBLOCK2": {
"akey2": "avalue2",
"bkey2": "bvalue2"
},
"ckey2": "cvalue2",
"key": "dvalue2"
}
}
]
"value"
"NOBLOCKvalue"
(^)#(^)#(^)#(^)bminter#ubuntu:~$
Beyond that I can't access any block inside the array:
(^)#(^)#(^)#(^)bminter#ubuntu:~$ jq '.[].BLOCK1' jqtest
jq: error (at jqtest:36): Cannot index array with string "BLOCK1"
(^)#(^)#(^)#(^)bminter#ubuntu:~$ jq '.[].BLOCK-2' jqtest
jq: error (at jqtest:36): Cannot index array with string "BLOCK"
(^)#(^)#(^)#(^)bminter#ubuntu:~$ jq '.[].BLOCK-A' jqtest
jq: error: A/0 is not defined at <top-level>, line 1:
.[].BLOCK-A
jq: 1 compile error
(^)#(^)#(^)#(^)bminter#ubuntu:~$
How do I access the array?
The array of objects with non-uniform keys is making things a little tricky here. Once you've gotten past .files you need to start using Array Iteration [] to access those elements and then use object operations like keys to go deeper.
Here is a function which may help in this situation. It scans .files for an object with a key matching the specified key and then returns the corresponding value:
def getfile($k): .files[] | select(keys[] | .==$k) | .[$k];
If jqtest contains the sample data the command
$ jq -M '
def getfile($k): .files[] | select(keys[] | .==$k) | .[$k];
getfile("BLOCK1").SUBBLOCK1.akey1
' jqtest
Returns
"avalue1"
Another approach is to use a function to convert .files[] into a more useful form. e.g.
$ jq -M '
def files: reduce .files[] as $f ({}; ($f|keys[0]) as $k | .[$k] = $f[$k]) ;
files
' jqtest
this returns a more uniform structure without arrays
{
"BLOCK1": {
"SUBBLOCK1": {
"akey1": "avalue1",
"bkey1": "bvalue1",
"ckey1": "cvalue1"
},
"dkey1": "dvalue1",
"key": "evalue1"
},
"BLOCK-2": ...
so with it you can write
files.BLOCK1.SUBBLOCK1
to obtain
{
"akey1": "avalue1",
"bkey1": "bvalue1",
"ckey1": "cvalue1"
}
Note that jq will re-evaluate the files function with each use so the following form may be more practical:
files as $files
| $files.BLOCK1.SUBBLOCK1
If you find this representation useful you may want to skip the function and instead just start your filter with
.files = reduce .files[] as $f ({}; ($f|keys[0]) as $k | .[$k] = $f[$k])
e.g.
$ jq -M '
.files = reduce .files[] as $f ({}; ($f|keys[0]) as $k | .[$k] = $f[$k])
# more stuff goes here
' jqtest
which converts your input to
{
"files": {
"BLOCK1": {
"SUBBLOCK1": {
"akey1": "avalue1",
"bkey1": "bvalue1",
"ckey1": "cvalue1"
},
"dkey1": "dvalue1",
"key": "evalue1"
},
"BLOCK-2": {
"SUBBLOCK2": {
"akey2": "avalue2",
"bkey2": "bvalue2"
},
"ckey2": "cvalue2",
"key": "dvalue2"
},
"BLOCK-A": {
"SUBBLOCK2": {
"akey2": "avalue2",
"bkey2": "bvalue2"
},
"ckey2": "cvalue2",
"key": "dvalue2"
}
},
"NOBLOCK": "value",
"key": "NOBLOCKvalue"
}
making whatever else you need to do after that easier