Loop on JSON array in bash script and pull data through JQ - arrays

i want to pull data from jSON file through jq in bash script. I have 50 plus SG objects in JSON .
here is an example of one SG . I want to print a VPC id and group id in one line and so on for another objects.
Solution i tried :
jq -c . $old |
while IFS= read -r obj; do
vpcidold=$( printf '%s' "$obj" | jq '.SecurityGroups[].VpcId')
securityidold=$( printf '%s' "$obj" | jq '.SecurityGroups[].GroupId')
echo "${vpcidold}||${securityidold}"
done > oldtest.json
it is working file but giving data in a line by line and want to do more optimised this with for loop.
How can I create a loop on JSON array to get desire output
"SG": [
{
"Description": "des",
"GroupName": "Gpname",
"IpPermissions": [
{
"FromPort": 80,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"ToPort": 80,
"UserIdGroupPairs": []
}
],
"OwnerId": "123",
"GroupId": "sg",
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"UserIdGroupPairs": []
}
],
"Tags": [
{
"Key": "projectcode",
"Value": "none"
},
{
"Key": "sgid",
"Value": "sg-123"
}
],
"VpcId": "vpc-123"
}
]
},

If the JSON file is just an array of the objects, you don't need to loop over them in bash. jq will loop over them implicitely:
jq -r '.[][][] | (.VpcId + "||" + .GroupId)' file.json
Tested on the following input:
[
{ "SG": [
{
"Description": "des",
"GroupName": "Gpname",
"GroupId": "sg",
"VpcId": "vpc-123"
}
] },
{ "SG": [
{
"Description": "des",
"GroupName": "xyz",
"GroupId": "sg-12345",
"VpcId": "vpc-12345"
}
] }
]
Output:
vpc-123||sg
vpc-12345||sg-12345

Related

How to use jq to convert 2 objects into CSV?

I'm trying to convert objects that look like this:
{
"metricId": "metric1",
"data": [
{
"dimensions": [
"DEVICE-a1b2c3",
"queue1"
],
"dimensionMap": {
"Queue": "queue1",
"enitity": "DEVICE-a1b2c3"
},
"timestamps": [
1626286800000
],
"values": [
1
]
},
{
"dimensions": [
"DEVICE-a1b2c3",
"queue2"
],
"dimensionMap": {
"Queue": "queue2",
"entity": "DEVICE-a1b2c3"
},
"timestamps": [
1626286800000
],
"values": [
2
]
}
]
}
{
"metricId": "metric2",
"data": [
{
"dimensions": [
"DEVICE-a1b2c3",
"queue1"
],
"dimensionMap": {
"Queue": "queue1",
"entity": "DEVICE-a1b2c3"
},
"timestamps": [
1626286800000
],
"values": [
11
]
},
{
"dimensions": [
"DEVICE-a1b2c3",
"queue2"
],
"dimensionMap": {
"Queue": "queue2",
"entity": "DEVICE-a1b2c3"
},
"timestamps": [
1626286800000
],
"values": [
22
]
}
]
}
To CSV that looks like this:
"metric1","queue1",1626286800000,1
"metric1","queue1",1626286800000,2
"metric2","queue1",1626286800000,11
"metric2","queue1",1626286800000,22
I was somewhat successful but I'm getting duplicates in my results.
Command: jq -r '. | {id:.metricId, queue: .data[].dimensionMap.Queue, time: .data[].timestamps[0], value: .data[].values[0]} | [.id, .queue, .time, .value] | #csv'
Output:
"metric1","queue1",1626286800000,1
"metric1","queue1",1626286800000,2
"metric1","queue1",1626286800000,1
"metric1","queue1",1626286800000,2
"metric1","queue2",1626286800000,1
"metric1","queue2",1626286800000,2
"metric1","queue2",1626286800000,1
"metric1","queue2",1626286800000,2
"metric2","queue1",1626286800000,11
"metric2","queue1",1626286800000,22
"metric2","queue1",1626286800000,11
"metric2","queue1",1626286800000,22
"metric2","queue2",1626286800000,11
"metric2","queue2",1626286800000,22
"metric2","queue2",1626286800000,11
"metric2","queue2",1626286800000,2
I've looked over the documentation and several blog posts/videos but I haven't been able to find a solution so far. Thank you for your help.
One way to tackle the problem is to use jq "$-variables":
.metricId as $metricId
| .data[]
| .dimensionMap.Queue as $q
| [.timestamps, .values] | transpose[]
| [$metricId, $q, .[]]
| #csv

using jq how to query and replace value within an array

How do I query and replace the value for SMT_PORT_3306_TCP_ADDR.
I tried
echo $task_definition | jq -r '.taskDefinition.containerDefinitions[0].environment[] | select(.name=="SMT_PORT_3306_TCP_ADDR")| .value = "myvalue" '
the output I get
{
"name": "SMT_PORT_3306_TCP_ADDR",
"value": "myvalue"
}
I do not get the full json
Input Json :
{
"taskDefinition": {
"taskDefinitionArn": "some value",
"containerDefinitions": [
{
"name": "common-api-img",
"environment": [
{
"name": "SERVER_API_TIMEOUT_SUBSCRIPTIONS_CANCEL_REQUEST",
"value": "false"
},
{
"name": "SMT_PORT_3306_TCP_ADDR",
"value": "valueToReplace"
}
],
"mountPoints": [],
"volumesFrom": []
}
],
"revision": 65,
"volumes": [],
"status": "ACTIVE"
}
}
Expected output without the top level taskDefinition value:
{
"taskDefinitionArn":"some value",
"containerDefinitions":[
{
"name":"common-api-img",
"environment":[
{
"name":"SERVER_API_TIMEOUT_SUBSCRIPTIONS_CANCEL_REQUEST",
"value":"false"
},
{
"name":"SMT_PORT_3306_TCP_ADDR",
"value":"myvalue"
}
],
"mountPoints":[
],
"volumesFrom":[
]
}
],
"revision":65,
"volumes":[
],
"status":"ACTIVE"
}
Use |= with if.
jq '.taskDefinition.containerDefinitions[0].environment[]
|= if .name == "SMT_PORT_3306_TCP_ADDR"
then .value = "myvalue"
else .
end'

Flatten a hierarchical JSON array using JQ

Can anyone help me get the correct jq command to flatten the below example? I've seen a few other posts and I'm hacking away at it but can't seem to get it. I'd greatly appreciate any help.
Input:
[
{
"name": "level1",
"children": [
{
"name": "level2",
"children": [
{
"name": "level3-1",
"children": []
},
{
"name": "level3-2",
"children": []
}
]
}
]
}
]
Output:
[
{
"displayName": "level1",
"parent": ""
},
{
"displayName": "level2",
"parent": "level1"
},
{
"displayName": "level3-1",
"parent": "level2"
},
{
"displayName": "level3-2",
"parent": "level2"
}
]
Here's a straightforward solution that does not involve a helper function and actually solves a more general problem. It is based on the idea of beginning by adding a "parent" key to each child, and then using .. to collect all the name/parent pairs.
So first consider:
[ walk(if type=="object" and has("children")
then .name as $n | .children |= map(.parent = $n)
else . end)
| ..
| select(type=="object" and has("name"))
| {displayName: .name, parent}
]
This meets the requirements except that for the top-level (parentless) object, it produces a .parent value of null. That would generally be more JSON-esque than "", but if the empty string is really required, one has simply to replace the last non-trivial line above by:
| {displayName: .name, parent: (.parent // "")}
With a simple recursive function:
def f: .name as $parent | .children[] | {$parent, displayName: .name}, f;
[ {name: "", children: .} | f ]
Online demo

jq: error (at <stdin>:1): Cannot index array with string "name" SOLVED

I'm trying to make a JSON collection of country, region, city, org, ip of intrusion attempts.
My JSON test info:
[
{
"total": 0,
"country": [
{
"name": "CN",
"nr": 0,
"region": [
{
"name": "Beijing",
"nr": 0,
"City": [
{
"name": "Haidian",
"nr": 0,
"Organisation": [
{
"name": "AS45090 Shenzhen Tencent Computer Systems Company Limited",
"nr": 0,
"IPS": [
{
"192.144.207.22": 0
}
]
}
]
}
]
}
]
},
{
"name": "NL",
"nr": 0,
"region": [
{
"name": "Noord Holland",
"nr": 0,
"City": [
{
"name": "Amsterdam",
"nr": 0,
"Organisation": [
{
"name": "FEAS",
"nr": 0,
"IPS": [
{
"192.162.1.1": 0
}
]
}
]
}
]
}
]
}
]
}
]
I load the existing json (test) string into $geoInfo. Now i'm trying to change the nr value in the object where"name": "CN"
i have tested two sollutions:
geoInfo="$( jq --arg country ${tmpGeo[0]} --arg count $count -r '.country | map( if .name == $country then . + { .nr=$count } else . end )'<<<"${geoInfo}" )"
And
geoInfo="$( jq --arg country ${tmpGeo[0]} --arg count $count -r '.country | select(.[].name == "CN") | .nr) = $count'<<<"${geoInfo}" )"
With both solutions I get:
jq: error (at <stdin>:1): Cannot index array with string "name"
I use jq version 1.6.
What is going wrong?
Would you please try the following:
geoInfo=$(jq "(.[].country[] | select(.name == \"CN\") | .nr) = 1" <<<"$geoInfo")
Just forget all of this! I'm terribly sorry. The errors i got was for a statement in the else section and I was changing the query in the then section. Probably worked to long on the code yesterday. my first solution had a small mistake after changing the .nr= to "nr": it worked:
geoInfo="$( jq --arg country ${tmpGeo[0]} --arg count $count -r '.country | map( if .name == $country then . + { "nr": $count } else . end )'<<<"${geoInfo}" )"

Using jq to convert list of map into an array

I am struggling to get formatted output from an aws dynamodb scan command.
An item in the dynamodb table looks like below:
{
"labels": {
"Category": [
"Data",
"EMR"
],
"Environment": "NonProd",
"Severity": "Critical"
},
"subscriber_id": "blah#blah.com",
"subscriber_type": "email"
}
When I run the query:
aws dynamodb scan --table-name dummy_table --region region_name --
profile default --query "Items[?subscriber_id.S ==
'blah#blah.com'].labels.M[]"
I get the output as below:
[
{
"Environment": {
"S": "NonProd"
},
"Severity": {
"S": "Critical"
},
"Category": {
"L": [
{
"S": "Data"
},
{
"S": "EMR"
}
]
}
}
]
Desired output is:
{
"Category": [
"Data",
"EMR"
],
"Environment": "NonProd",
"Severity": "Critical"
}
To achieve the desired output, I tried to manipulate using jq.
Updated query:
aws dynamodb scan --table-name dummy_table --region
region_name --
profile default --query "Items[?subscriber_id.S ==
'blah#blah.com'].labels.M[]" |
jq -r '.[]
| to_entries[]
| [{key:.key, value:.value[]}]
| from_entries' | jq -s add
Output is:
{
"Environment": "NonProd",
"Severity": "Critical",
"Category": [
{
"S": "Data"
},
{
"S": "EMR"
}
]
}
As you see it's close, but it's not processing the Category list. Any help with getting the desired output is appreciated.
Thanks
You can use the following :
.[] | { Environment: .Environment.S, Severity: .Severity.S, Category: (.Category.L | map(.S)) }
Output :
{
"Environment": "NonProd",
"Severity": "Critical",
"Category": [
"Data",
"EMR"
]
}
Try it here !

Resources