I am struggling to get formatted output from an aws dynamodb scan command.
An item in the dynamodb table looks like below:
{
"labels": {
"Category": [
"Data",
"EMR"
],
"Environment": "NonProd",
"Severity": "Critical"
},
"subscriber_id": "blah#blah.com",
"subscriber_type": "email"
}
When I run the query:
aws dynamodb scan --table-name dummy_table --region region_name --
profile default --query "Items[?subscriber_id.S ==
'blah#blah.com'].labels.M[]"
I get the output as below:
[
{
"Environment": {
"S": "NonProd"
},
"Severity": {
"S": "Critical"
},
"Category": {
"L": [
{
"S": "Data"
},
{
"S": "EMR"
}
]
}
}
]
Desired output is:
{
"Category": [
"Data",
"EMR"
],
"Environment": "NonProd",
"Severity": "Critical"
}
To achieve the desired output, I tried to manipulate using jq.
Updated query:
aws dynamodb scan --table-name dummy_table --region
region_name --
profile default --query "Items[?subscriber_id.S ==
'blah#blah.com'].labels.M[]" |
jq -r '.[]
| to_entries[]
| [{key:.key, value:.value[]}]
| from_entries' | jq -s add
Output is:
{
"Environment": "NonProd",
"Severity": "Critical",
"Category": [
{
"S": "Data"
},
{
"S": "EMR"
}
]
}
As you see it's close, but it's not processing the Category list. Any help with getting the desired output is appreciated.
Thanks
You can use the following :
.[] | { Environment: .Environment.S, Severity: .Severity.S, Category: (.Category.L | map(.S)) }
Output :
{
"Environment": "NonProd",
"Severity": "Critical",
"Category": [
"Data",
"EMR"
]
}
Try it here !
Related
How do I query and replace the value for SMT_PORT_3306_TCP_ADDR.
I tried
echo $task_definition | jq -r '.taskDefinition.containerDefinitions[0].environment[] | select(.name=="SMT_PORT_3306_TCP_ADDR")| .value = "myvalue" '
the output I get
{
"name": "SMT_PORT_3306_TCP_ADDR",
"value": "myvalue"
}
I do not get the full json
Input Json :
{
"taskDefinition": {
"taskDefinitionArn": "some value",
"containerDefinitions": [
{
"name": "common-api-img",
"environment": [
{
"name": "SERVER_API_TIMEOUT_SUBSCRIPTIONS_CANCEL_REQUEST",
"value": "false"
},
{
"name": "SMT_PORT_3306_TCP_ADDR",
"value": "valueToReplace"
}
],
"mountPoints": [],
"volumesFrom": []
}
],
"revision": 65,
"volumes": [],
"status": "ACTIVE"
}
}
Expected output without the top level taskDefinition value:
{
"taskDefinitionArn":"some value",
"containerDefinitions":[
{
"name":"common-api-img",
"environment":[
{
"name":"SERVER_API_TIMEOUT_SUBSCRIPTIONS_CANCEL_REQUEST",
"value":"false"
},
{
"name":"SMT_PORT_3306_TCP_ADDR",
"value":"myvalue"
}
],
"mountPoints":[
],
"volumesFrom":[
]
}
],
"revision":65,
"volumes":[
],
"status":"ACTIVE"
}
Use |= with if.
jq '.taskDefinition.containerDefinitions[0].environment[]
|= if .name == "SMT_PORT_3306_TCP_ADDR"
then .value = "myvalue"
else .
end'
i want to pull data from jSON file through jq in bash script. I have 50 plus SG objects in JSON .
here is an example of one SG . I want to print a VPC id and group id in one line and so on for another objects.
Solution i tried :
jq -c . $old |
while IFS= read -r obj; do
vpcidold=$( printf '%s' "$obj" | jq '.SecurityGroups[].VpcId')
securityidold=$( printf '%s' "$obj" | jq '.SecurityGroups[].GroupId')
echo "${vpcidold}||${securityidold}"
done > oldtest.json
it is working file but giving data in a line by line and want to do more optimised this with for loop.
How can I create a loop on JSON array to get desire output
"SG": [
{
"Description": "des",
"GroupName": "Gpname",
"IpPermissions": [
{
"FromPort": 80,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"ToPort": 80,
"UserIdGroupPairs": []
}
],
"OwnerId": "123",
"GroupId": "sg",
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"UserIdGroupPairs": []
}
],
"Tags": [
{
"Key": "projectcode",
"Value": "none"
},
{
"Key": "sgid",
"Value": "sg-123"
}
],
"VpcId": "vpc-123"
}
]
},
If the JSON file is just an array of the objects, you don't need to loop over them in bash. jq will loop over them implicitely:
jq -r '.[][][] | (.VpcId + "||" + .GroupId)' file.json
Tested on the following input:
[
{ "SG": [
{
"Description": "des",
"GroupName": "Gpname",
"GroupId": "sg",
"VpcId": "vpc-123"
}
] },
{ "SG": [
{
"Description": "des",
"GroupName": "xyz",
"GroupId": "sg-12345",
"VpcId": "vpc-12345"
}
] }
]
Output:
vpc-123||sg
vpc-12345||sg-12345
Can anyone help me get the correct jq command to flatten the below example? I've seen a few other posts and I'm hacking away at it but can't seem to get it. I'd greatly appreciate any help.
Input:
[
{
"name": "level1",
"children": [
{
"name": "level2",
"children": [
{
"name": "level3-1",
"children": []
},
{
"name": "level3-2",
"children": []
}
]
}
]
}
]
Output:
[
{
"displayName": "level1",
"parent": ""
},
{
"displayName": "level2",
"parent": "level1"
},
{
"displayName": "level3-1",
"parent": "level2"
},
{
"displayName": "level3-2",
"parent": "level2"
}
]
Here's a straightforward solution that does not involve a helper function and actually solves a more general problem. It is based on the idea of beginning by adding a "parent" key to each child, and then using .. to collect all the name/parent pairs.
So first consider:
[ walk(if type=="object" and has("children")
then .name as $n | .children |= map(.parent = $n)
else . end)
| ..
| select(type=="object" and has("name"))
| {displayName: .name, parent}
]
This meets the requirements except that for the top-level (parentless) object, it produces a .parent value of null. That would generally be more JSON-esque than "", but if the empty string is really required, one has simply to replace the last non-trivial line above by:
| {displayName: .name, parent: (.parent // "")}
With a simple recursive function:
def f: .name as $parent | .children[] | {$parent, displayName: .name}, f;
[ {name: "", children: .} | f ]
Online demo
I have the following data structure as a result of aws logs get-query-results:
{
"status": "Complete",
"statistics": {
"recordsMatched": 2.0,
"recordsScanned": 13281.0,
"bytesScanned": 7526096.0
},
"results": [
[
{
"field": "time",
"value": "2019-01-31T21:53:01.136Z"
},
{
"field": "requestId",
"value": "a9c233f7-0b1b-3326-9b0f-eba428e4572c"
},
{
"field": "logLevel",
"value": "INFO"
},
{
"field": "callerId",
"value": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
],
[
{
"field": "time",
"value": "2019-01-25T13:13:01.062Z"
},
{
"field": "requestId",
"value": "a4332628-1b9b-a9c2-0feb-0cd4a3f7cb63"
},
{
"field": "logLevel",
"value": "INFO"
},
{
"field": "callerId",
"value": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
],
]
}
The AWS CLI support JMESPath language for filtering output. I need to apply a query string, to filter among the returned "results" the objects that contain the "callerId" as a "field", retrieve the "value" property and obtain the following output:
[
{
callerId: "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
},
{
callerId: "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
]
The first step I do is flatter the results array with the query string: results[]
This will get read of the other root properties (status, statistics) and return only one big array with all of the {field: ..., value: ...} alike objects. But after this I can't manage to properly filter for those objects that match field=="callerId". I tried, among others, the following expressions without success:
'results[][?field=="callerId"]'
'results[][*][?field=="callerId"]'
'results[].{ callerId: #[?field=="callerId"].value }'
I'm not an expert in JMESPath and I was doing the tutorials of the jmespath.org site but couldn't manage to make it work.
Thanks!
Using jq is a good thing because it's more complete language, but if you want to do it with JMES Path here the solution:
results[*][?field=='callerId'].{callerId: value}[]
to get:
[
{
"callerId": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
},
{
"callerId": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
]
I'm not able to reproduce fully since I don't have the same logs in my log stream but I was able to do this using jq and putting the sample JSON object in a file
cat sample_output.json | jq '.results[][] | select(.field=="callerId") | .value'
OUTPUT:
"a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
"a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
you could pipe the output from the aws cli to jq.
I was able to get pretty close with the native JMESPath query and using the built in editor in this site
http://jmespath.org/examples.html#filtering-and-selecting-nested-data
results[*][?field==`callerId`][]
OUTPUT:
[
{
"field": "callerId",
"value": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
},
{
"field": "callerId",
"value": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
]
but I'm not sure how to get callerId to be the key and the value to be the value from another key.
I want to search exact string in array.
My data in ES is like below:
{ category": [
"abc test"
],
"es_flag": false,
"bullet_points": [],
"content": "",
"description": false }
I have multiple category like "abc test", "new abc test" etc...
I am trying below query but I am getting multiple category result, I was searching for "abc test" but "new abc test" category is also coming in the result.
{
"from": 0,
"size": 30,
"query": {
"bool" : {
"must": [
{ "match_phrase": { "category": "abc test" } }
]
}
},
"sort": [ { "createdAt": { "order": "desc" } } ]
}
Help will be appreciated.
I'm assuming you are using default analyzer. In that case match_phrase against "field": "abc test" will match all documents which will have the fields with adjacent tokens of abc test, including:
new abc test
abc test new
foo abc test bar
And it will not match:
abc new test - query tokens are not adjacent
test abc - query tokens are adjacent, but in the wrong order
What would actually help you is using the keyword analyzer for your field (you either need to build new index from scratch or update your mappings). If you're building from scrach:
curl -XPUT http://localhost:9200/my_index -d '
{
"mappings": {
"categories": {
"properties": {
"category": {
"type": "text",
"analyzer": "keyword"
}
}
}
}
}'
And afterwards you need to use just simple query, e.g. like this (either match or term will do):
curl -XGET http://localhost:9200/my_index/_search -d '
{
"query": {
"match" : {
"message" : "abc test"
}
}
}'
My version of elasticsearch is 6.0.1. I am using this approach:
GET <your index>/_search
{
"query": {
"bool": {
"must": [{
"query_string": {
"query": "category:abc OR category:test"
}
}]
}
},
"sort":[{"createdAt": {
"order": "desc"
}}]
}