Delete objects and arrays with jq which match a key - arrays

I have a JSON with the following content:
{
"data": [
{
"name": "Test",
"program": {
"publicAccess": "--------",
"externalAccess": false,
"userGroupAccesses": [
{
"access": "r-------"
},
{
"access": "rw------"
}
],
"id": "MmBqeMLIC2r"
},
"publicAccess": "rw------"
}
]
}
And I want to delete all keys (recursively) which match publicAccess or userGroupAccesses, so that my JSON looks like this:
{
"data": [
{
"name": "Test",
"program": {
"externalAccess": false,
"id": "MmBqeMLIC2r"
}
}
]
}
I've copied jq's builtin walk function from source.
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
# My Code
walk(if (type == "object" and .publicAccess)
then del(.publicAccess)
elif (type == "array" and .userGroupAccesses)
then del(.userGroupAccesses)
else
.
end )
Gives me jq: error (at <stdin>:2622): Cannot index array with string "userGroupAccesses". Also if I use .userGroupAccesses[] - How do I get the result?
Snippet on jqplay: https://jqplay.org/s/1m7wAeHMTu

Your problem is when type == "array" is true . will be an array so .userGroupAccesses won't work. What you want to do is focus on the case when . is an object. In your call to walk you only need to check for type == "object" and then remove the members you don't want. e.g.
walk(if type == "object" then del(.publicAccess, .userGroupAccesses) else . end)
Try it online at jqplay.org
You can also solve this without walk by using Recursive Descent .. e.g.
del(.. | .publicAccess?, .userGroupAccesses?)
Try it online at jqplay.org

Related

Powershell - Add a key in a specific manner to a JSON object

i have an array converted to a json ($user_data_JSON = $user_data | ConvertTo-Json) in powershell which looks like this
$user_data_JSON
{
[
{
"targetId": "5007Y00000K5nkjQAB",
"login": "login1",
"password": "scRHDztkKbO"
},
{
"targetId": "5007Y00000MNbDvQAL",
"login": "login2",
"password": "scRHDztkKbO"
}
]
}
But i need to modify it like this with a KEY value on top:
$user_data_JSON
{
"logins":[
{
"targetId": "5007Y00000K5nkjQAB",
"login": "login1",
"password": "scRHDztkKbO"
},
{
"targetId": "5007Y00000MNbDvQAL",
"login": "login2",
"password": "scRHDztkKbO"
}
]
}
How can I manage to achieve this?
I've already tried to create a new array object, add the key and convert it to a json file like this:
$jsonBase = #{}
$list = New-Object System.Collections.ArrayList
$list.Add("Foo")
$list.Add("Bar")
$jsonBase.Add("Data",$list)
$jsonBase | ConvertTo-Json
To get something like this:
{
"Data": [
"Foo",
"Bar"
]
}
but when I convert my array to a json again, it looks kind of trunkated:
$jsonBase | ConvertTo-Json -Depth 10
{
"logins": [
"[\r\n {\r\n \"targetId\": \"5007Y00000K5nkjQAB\",\r\n \"login\": \"login1\",\r\n \"password\": \"scRHDztkKbO\"\r\n },\r\n {\r\n
\"targetId\": \"5007Y00000MNbDvQAL\",\r\n \"login\": \"login2\",\r\n \"password\": \"scRHDztkKbO\"\r\n },"
]
}
How can I get a propoer JSON object?
Thanks
Don't use the JSON (text) representation of your array ($user_data_JSON), use its original, object form ($user_data) to construct a wrapper object with the desired top-level property:
[pscustomobject] #{ logins = $user_data } | ConvertTo-Json -Depth 10
As for what you tried:
If you use a preexisting JSON string as the value of property to be converted to JSON with ConvertTo-Json, it is treated like any other string value, resulting in the representation you saw, with " characters escaped as \" and (CRLF) newlines as \r\n
A simple example:
[pscustomobject] #{
foo = '{ "bar":
"baz" }'
} | ConvertTo-Json
Output (note how the foo property value became a single-line string with " and newlines escaped):
{
"foo": "{ \"bar\": \n \"baz\" }"
}

Adding Element to array of element in json using jq

Ok, i spent way to much time on this.
Using jq, i want to test if an element exist in an array, if yes being able to mofify this element, if not adding this element to that array
I have read
Update one value in array of dicts, using jq
How to check if element exists in array with jq
Which gave hints But i am stuck
Let say i have a json file like :
{
"LOGIN":"user",
"COPY": [
{
"CHAINLIST":"chain1",
"ELEMENT": [ "element1-1","element1-2" ]
},
{
"CHAINLIST":"chain2",
"ELEMENT": [ "element2-1","element2-2" ]
}
]
}
I Would like to add elements to the COPY array based in the CHAINLIST id , let say those 3 elements :
//ELEM A
{
"CHAINLIST":"chain3",
"ELEMENT": [ "element3-1" ]
}
//ELEM B
{
"CHAINLIST":"chain2",
"ELEMENT": [ "element2-1","element2-3" ]
}
//ELEM C
{
"CHAINLIST":"chain1",
"ELEMENT": [ "element1-1","element1-2" ]
}
ELEM A has the CHAINLIST id chain3 who doesn't exit in the COPY Array _> let's add it
ELEM B Is already present in the COPY aray, I want to update the ELEMENT array
content with the new value
ELEM C Already exist and is up to date, i don't need to do anything.
In my example the final json object would be
{
"LOGIN":"user",
"COPY": [
{
"CHAINLIST":"chain1",
"ELEMENT": [ "element1-1","element1-2" ]
},
{
"CHAINLIST":"chain2",
"ELEMENT": [ "element2-1","element2-2","element2-3" ]
},
{
"CHAINLIST":"chain3",
"ELEMENT": [ "element3-1"]
}
]
}
I am using jq The procedure to add the element Elem to the object Obj would be :
Elem.CHAINLIST exist in Obj.COPY [] ?
yes : add Elem.ELEMENT to Obj.ELEMENT[] with unique to qvoid duplicates
no : add Elem to Obj.COPY[]
The best i get is
(if .COPY[]? | select (.CHAINLIST=="chain3") == "" then . else .COPY[.COPY | length ] |= . + { "CHAINLIST":"chain3","ELEMENT":[ "element3-1"]} end )
But select (.CHAINLIST=="chain3") == "" return nothing I i can figure out how to test if empty.
Thank you for your time !
The following solution uses the built-in function INDEX/2 to create convenient and efficient representations
of the collections of CHAINLIST-ELEMENT objects, together with a bespoke function, augment/1, for combining elements.
It is assumed that the objects defining the updates are presented as an array in the file edits.json.
< input.json jq --argfile edits edits.json '
# Use `unique` to combine the .ELEMENT fields
def augment($b):
reduce ($b|keys_unsorted[]) as $key (.;
.[$key].ELEMENT = (.[$key].ELEMENT + $b[$key].ELEMENT | unique));
(INDEX($edits[];.CHAINLIST) | map_values(del(.CHAINLIST))) as $edits
| (INDEX(.COPY[]; .CHAINLIST) | map_values(del(.CHAINLIST))) as $COPY
| .COPY |= ($COPY
| augment($edits)
| to_entries
| map( { CHAINLIST: .key, ELEMENT: .value.ELEMENT} ))
Output: as specified in the question.
Note: if the objects defining the updates are presented as a sequence of objects in a file, you could use the above solution with --slurpfile instead of --argfile.

Using jq to produce one line of output per array element in a json object that also includes other fields from the json object

I'm trying to format some data that consists of json objects that includes some identifying information along with an array of one or more json objects, I would like the result to be one line of data per array element where each line should include some fields from the array element and some fields from the identifying information.
My sample data is below:
{
"eventCreation": {
"timeStamp": "2020-06-06T15:07:20Z",
"epoch": 1591456040
},
"eventData": {
"applName": "SampleApp",
"channelName": "SYSTEM.DEF.SVRCONN",
"connectionName": "127.0.0.1",
"channelType": "Svrconn",
"remoteProduct": "MQJM",
"remoteVersion": "09010005",
"activityTrace": [
{
"operationId": "Get",
"operationTime": "11:07:18",
"qmgrOpDuration": 102,
"reasonCode": {
"name": "No Msg Available",
"value": 2033
},
"objectName": "SYSTEM.DEFAULT.LOCAL.QUEUE"
},
{
"operationId": "Cb",
"operationTime": "11:07:18",
"qmgrOpDuration": 10,
"reasonCode": {
"name": "None",
"value": 0
},
"objectName": "SYSTEM.DEFAULT.LOCAL.QUEUE"
},
{
"operationId": "Cb",
"operationTime": "11:07:18",
"qmgrOpDuration": 12,
"reasonCode": {
"name": "None",
"value": 0
},
"objectName": "SYSTEM.DEFAULT.LOCAL.QUEUE"
}
]
}
}
I would like to get an output like this:
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","11:07:18","Get",102,"SYSTEM.DEFAULT.LOCAL.QUEUE",2033
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","11:07:18","Cb",10,"SYSTEM.DEFAULT.LOCAL.QUEUE",0
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","11:07:18","Cb",12,"SYSTEM.DEFAULT.LOCAL.QUEUE",0
I can pick any one element of the array and get it to print 3 lines, but if I add a 2nd element it will print 9 lines, 3rd element prints 27, etc.
For example:
jq -r '{channelName: .eventData.channelName, channelType: .eventData.channelType, connectionName: .eventData.connectionName, applName: .eventData.applName, remoteProduct: .eventData.remoteProduct, remoteVersion: .eventData.remoteVersion, operationId: .eventData.activityTrace[].operationId}|[.[]]|#csv' TEST.json
Will produce this:
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Get"
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb"
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb"
If I add a second like this:
jq -r '{channelName: .eventData.channelName, channelType: .eventData.channelType, connectionName: .eventData.connectionName, applName: .eventData.applName, remoteProduct: .eventData.remoteProduct, remoteVersion: .eventData.remoteVersion, operationId: .eventData.activityTrace[].operationId, qmgrOpDuration: .eventData.activityTrace[].qmgrOpDuration}|[.[]]|#csv' TEST.json
Will produce this:
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Get",102
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Get",10
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Get",12
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",102
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",10
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",12
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",102
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",10
"SYSTEM.DEF.SVRCONN","Svrconn","127.0.0.1","SampleApp","MQJM","09010005","Cb",12
Using your approach, the following is a solution:
.eventData
| ({channelName, channelType, connectionName, applName, remoteProduct, remoteVersion}
+ ( .activityTrace[]
| { operationTime, operationId, qmgrOpDuration, objectName, v: .reasonCode.value}))
| [.[]]
| #csv
The key is to iterate just once.
Notice also that this solution achieves its brevity in part by using the fact that {foo: .foo} can be abbreviated to {foo}.
A slightly more efficient approach
.eventData
| [.channelName, .channelType, .connectionName, .applName, .remoteProduct, .remoteVersion]
+ ( .activityTrace[]
| [.operationTime, .operationId, .qmgrOpDuration, .objectName, .reasonCode.value] )
| #csv
.operationTime first
.eventData
| [.channelName, .channelType, .connectionName, .applName, .remoteProduct, .remoteVersion] as $x
| ( .activityTrace[]
| [.operationTime]
+ $x
+ [.operationId, .qmgrOpDuration, .objectName, .reasonCode.value] )
| #csv

Searching in a JSON array with Perl

I need to search for a specific term in a JSON array and get the index of the array element which contains the term as its value of a key, below is a sample JSON file that I'm working on, I need to get the array element whose "artifact_id" is "jar", so in this case I need to get the index of the array which is 1, the "artifact_id" is guaranteed to be unique:
{
"maven": [
{
"version": "1.2",
"artifact_id": "zip"
},
{
"version": "1.2",
"artifact_id": "jar"
}
]
}
I'm using the JSON lib and can decode the JSON to a Perl object, anyone can suggest the next step for me? Thanks a lot.
my $json_text = do {
open( my $json_fh, "<:encoding(UTF-8)", $filename[0] )
or die("Can't open \$filename\": $!\n");
local $/;
<$json_fh>;
};
my $json = JSON->new;
my $json_data = $json->decode($json_text);
Simply iterate through your array, and store away the index once the matching node is found. The example below should speak for itself, but given the contents of your post I guess you should read up on loops, comparitive operators, as well as basic programming.
After you have decoded your JSON-data, there is nothing special about iterating over the array, after all; it is just an array (reference given the tree like structure of JSON).
use JSON qw(decode_json);
my $needle = "jar";
my $needle_location;
my $data = decode_json( join '', <DATA> );
while( my( $idx, $elem ) = each( #{$data->{'maven'}} ) )
{
if( $elem->{'artifact_id'} eq $needle )
{
$needle_location = $idx;
last;
}
}
die "unable to find entry" unless defined $needle_location;
print "needle_location: $needle_location\n";
__DATA__
{
"maven": [
{
"version": "1.2",
"artifact_id": "zip"
},
{
"version": "1.2",
"artifact_id": "jar"
}
]
}
needle_location: 1
My answer is based on that of Filip Roséen - refp, but hopefully a more advanced and abbreviated approach:
use Modern::Perl;
use List::Util qw(first);
use JSON qw(decode_json);
my $needle = "jar";
my $data = decode_json do { local $/; <DATA> };
my $needle_location = first( sub {
my $artifact_id = $data->{ maven }->[ $_ ]->{ artifact_id };
defined($artifact_id) && $artifact_id eq $needle
}, 0 .. $#{ $data->{ maven } } ) //
die "unable to find entry";
say "needle_location: $needle_location";
__DATA__
{
"maven": [
{
"version": "1.2",
"artifact_id": "zip"
},
{
"version": "1.2",
"artifact_id": "jar"
}
]
}

Can't get jq to recognize an array within a json file

I have a json file that I use for work that I need to parse that is in the following format:
(^)#(^)#(^)#(^)bminter#ubuntu:~$ cat jqtest
{
"files":[
{
"BLOCK1":{
"SUBBLOCK1":{
"akey1":"avalue1",
"bkey1":"bvalue1",
"ckey1":"cvalue1"
},
"dkey1":"dvalue1",
"key":"evalue1"
}
},
{
"BLOCK-2":{
"SUBBLOCK2":{
"akey2":"avalue2",
"bkey2":"bvalue2"
},
"ckey2":"cvalue2",
"key":"dvalue2"
}
},
{
"BLOCK-A":{
"SUBBLOCK2":{
"akey2":"avalue2",
"bkey2":"bvalue2"
},
"ckey2":"cvalue2",
"key":"dvalue2"
}
}],
"NOBLOCK":"value",
"key":"NOBLOCKvalue"
}
So it's an array nested within a json file. jq .[] jqtest gives me everything in the file. Even the data outside the array. Except, outside the array, I'm only given the values not the keys:
(^)#(^)#(^)#(^)bminter#ubuntu:~$ jq .[] jqtest
[
{
"BLOCK1": {
"SUBBLOCK1": {
"akey1": "avalue1",
"bkey1": "bvalue1",
"ckey1": "cvalue1"
},
"dkey1": "dvalue1",
"key": "evalue1"
}
},
{
"BLOCK-2": {
"SUBBLOCK2": {
"akey2": "avalue2",
"bkey2": "bvalue2"
},
"ckey2": "cvalue2",
"key": "dvalue2"
}
},
{
"BLOCK-A": {
"SUBBLOCK2": {
"akey2": "avalue2",
"bkey2": "bvalue2"
},
"ckey2": "cvalue2",
"key": "dvalue2"
}
}
]
"value"
"NOBLOCKvalue"
(^)#(^)#(^)#(^)bminter#ubuntu:~$
Beyond that I can't access any block inside the array:
(^)#(^)#(^)#(^)bminter#ubuntu:~$ jq '.[].BLOCK1' jqtest
jq: error (at jqtest:36): Cannot index array with string "BLOCK1"
(^)#(^)#(^)#(^)bminter#ubuntu:~$ jq '.[].BLOCK-2' jqtest
jq: error (at jqtest:36): Cannot index array with string "BLOCK"
(^)#(^)#(^)#(^)bminter#ubuntu:~$ jq '.[].BLOCK-A' jqtest
jq: error: A/0 is not defined at <top-level>, line 1:
.[].BLOCK-A
jq: 1 compile error
(^)#(^)#(^)#(^)bminter#ubuntu:~$
How do I access the array?
The array of objects with non-uniform keys is making things a little tricky here. Once you've gotten past .files you need to start using Array Iteration [] to access those elements and then use object operations like keys to go deeper.
Here is a function which may help in this situation. It scans .files for an object with a key matching the specified key and then returns the corresponding value:
def getfile($k): .files[] | select(keys[] | .==$k) | .[$k];
If jqtest contains the sample data the command
$ jq -M '
def getfile($k): .files[] | select(keys[] | .==$k) | .[$k];
getfile("BLOCK1").SUBBLOCK1.akey1
' jqtest
Returns
"avalue1"
Another approach is to use a function to convert .files[] into a more useful form. e.g.
$ jq -M '
def files: reduce .files[] as $f ({}; ($f|keys[0]) as $k | .[$k] = $f[$k]) ;
files
' jqtest
this returns a more uniform structure without arrays
{
"BLOCK1": {
"SUBBLOCK1": {
"akey1": "avalue1",
"bkey1": "bvalue1",
"ckey1": "cvalue1"
},
"dkey1": "dvalue1",
"key": "evalue1"
},
"BLOCK-2": ...
so with it you can write
files.BLOCK1.SUBBLOCK1
to obtain
{
"akey1": "avalue1",
"bkey1": "bvalue1",
"ckey1": "cvalue1"
}
Note that jq will re-evaluate the files function with each use so the following form may be more practical:
files as $files
| $files.BLOCK1.SUBBLOCK1
If you find this representation useful you may want to skip the function and instead just start your filter with
.files = reduce .files[] as $f ({}; ($f|keys[0]) as $k | .[$k] = $f[$k])
e.g.
$ jq -M '
.files = reduce .files[] as $f ({}; ($f|keys[0]) as $k | .[$k] = $f[$k])
# more stuff goes here
' jqtest
which converts your input to
{
"files": {
"BLOCK1": {
"SUBBLOCK1": {
"akey1": "avalue1",
"bkey1": "bvalue1",
"ckey1": "cvalue1"
},
"dkey1": "dvalue1",
"key": "evalue1"
},
"BLOCK-2": {
"SUBBLOCK2": {
"akey2": "avalue2",
"bkey2": "bvalue2"
},
"ckey2": "cvalue2",
"key": "dvalue2"
},
"BLOCK-A": {
"SUBBLOCK2": {
"akey2": "avalue2",
"bkey2": "bvalue2"
},
"ckey2": "cvalue2",
"key": "dvalue2"
}
},
"NOBLOCK": "value",
"key": "NOBLOCKvalue"
}
making whatever else you need to do after that easier

Resources