I am trying to group several items by their name based on the JSON-Data. Furthermore I want to calculate the sum of the size. Therefore I am using jq to transforming the data. Attached you see the raw data.
{
"fields": {
"issuetype": {
"name": "Server"
},
"size": 2
}
}
{
"fields": {
"issuetype": {
"name": "Server"
},
"size": 2
}
}
{
"fields": {
"issuetype": {
"name": "Hardware"
},
"size": 0.58
}
}
The transformed data should be structered like this:
[
{
"item": "Hardware",
"size": 0.58
},
{
"item": "Server",
"size": 4
}
]
I am using the following code
jq -s < input.json "group_by( .fields.issuetype.name ) | .[] | item: .[0].fields.issuetype.name), size:([.[].fields.customfield_1234] | add)" > transformedData.json
Almost there. To construct objects, you need object contructors.
group_by(.fields.issuetype.name)
| map({item: .[0].fields.issuetype.name, size: map(.fields.size) | add})
Online demo
Related
I have data in this format:
{
"searchResult": [
{
"key": "common1",
"value": "A string"
},
{
"key": "common2",
"value": "2149944"
},
{
"key": "varying1",
"value": "604516"
},
{
"key": "varying73",
"value": "58.92"
}
]
}
Within searchResult are some constantly present fields (timestamp, identifiers etc). The other keys are constantly changing and can be named anything. I need them transformed to the format below, with the predefined constant keys pulled out to the top level and the variable keys staying in the searchResult array.
{
"common1": "A string",
"common2": "2149944",
"searchResult": [
{
"key": "varying1",
"value": "604516"
},
{
"key": "varying73",
"value": "58.92"
}
]
}
Seeing as jq is already being used in the process, how can I do this transformation in jq please?
I have tried extracting the values using .name, but haven't managed to bring them to this top level.
Many thanks
Ben
You could use IN/1 as follows:
(.searchResult | (from_entries | {common1, common2})) + { searchResult }
| .searchResult |= map(select(.key | IN("common1", "common2") | not))
Can anyone help me get the correct jq command to flatten the below example? I've seen a few other posts and I'm hacking away at it but can't seem to get it. I'd greatly appreciate any help.
Input:
[
{
"name": "level1",
"children": [
{
"name": "level2",
"children": [
{
"name": "level3-1",
"children": []
},
{
"name": "level3-2",
"children": []
}
]
}
]
}
]
Output:
[
{
"displayName": "level1",
"parent": ""
},
{
"displayName": "level2",
"parent": "level1"
},
{
"displayName": "level3-1",
"parent": "level2"
},
{
"displayName": "level3-2",
"parent": "level2"
}
]
Here's a straightforward solution that does not involve a helper function and actually solves a more general problem. It is based on the idea of beginning by adding a "parent" key to each child, and then using .. to collect all the name/parent pairs.
So first consider:
[ walk(if type=="object" and has("children")
then .name as $n | .children |= map(.parent = $n)
else . end)
| ..
| select(type=="object" and has("name"))
| {displayName: .name, parent}
]
This meets the requirements except that for the top-level (parentless) object, it produces a .parent value of null. That would generally be more JSON-esque than "", but if the empty string is really required, one has simply to replace the last non-trivial line above by:
| {displayName: .name, parent: (.parent // "")}
With a simple recursive function:
def f: .name as $parent | .children[] | {$parent, displayName: .name}, f;
[ {name: "", children: .} | f ]
Online demo
I have the below format of JSON file which is having issues[] array and I tried to use it for Kibana. But unfortunately Kibana doesn't support nested objects and array and there is a plugin to utilize so that I need to downgrade which I can't do right now because in that case I will lose all my data.
Sample data:
{
"expand": "schema,names",
"startAt": 0,
"maxResults": 50,
"total": 4,
"issues": [{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "1999875",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/1999875",
"key": "KINDLEAMZ-67578",
"fields": {
"summary": "contingency is displaying for confirmed card.",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
},
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "2019428",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/2019428",
"key": "KINDLEAMZ-68661",
"fields": {
"summary": "card",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
},
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "2010958",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/2010958",
"key": "KINDLEAMZ-68167",
"fields": {
"summary": "Test Card",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
}
]
}
So I just planned to restructure this payload like all the issues[] into as an object and write down in separate file. So that I can avoid that issue.
Expected output:
For that above sample data I have 4 records in issues[].length so I just want to create 4 different files with below format:
File1.json:
{
"key": "KINDLEAMZ-67578",
"summary": "contingency is displaying for confirmed card.",
"name": "P1",
"created": "2019-09-23T11:25:21.000+0000"
}
The same way I want to looping the other arrays and get the values as like above and write down in File2.json, File3.json, and File4.json.
Since the data is dynamic and so I just want this file creation happen based of length of issues[] array.
Is there anyway to achieve this by using shell script? Or any CLI library.
Please advise me.
Specify -c/--compact-output flag to make jq put each entity on a single, separate line, then use awk to write each line to a separate file.
jq -c '.issues[] | {
key,
summary: .fields.summary,
name: .fields.priority.name,
created: .fields.created
}' file | awk '{
f = ("file" NR ".json")
print > f
close(f)
}'
Using GNU awk and extension gawk-json:
awk '
#load "json"
{
lines=lines $0
if(json_fromJSON(lines,data)==1){
for(i in data["issues"]) {
out["key"] = data["issues"][i]["key"]
out["summary"] = data["issues"][i]["fields"]["summary"]
out["created"] = data["issues"][i]["fields"]["created"]
out["name"] = data["issues"][i]["fields"]["priority"]["name"]
file="file" i ".json"
print json_toJSON(out) > file
close(file)
delete out
}
}
}' file.json
Output:
$ cat file1.json | jq '.' # useless use of cat but used to emphasize
{
"created": "2019-09-23T11:25:21.000+0000",
"key": "KINDLEAMZ-67578",
"summary": "contingency is displaying for confirmed card.",
"name": "P1"
}
I am trying to parse this JSON using Powershell's ConvertFrom-Json feature, but it seems to truncate the data:
{
"MessagesMonitoring": {
"version": 1,
"description": "Message Description"
},
"data": {
"swindon": {
"totalMessages": 0,
"identifier": [
{
"name": "ET",
"staleCount": 4
},
{
"name": "ET_2",
"staleCount": 4
}
]
},
"Reading": {
"totalMessages": 0,
"identifier": [
{
"name": "J3",
"staleCount": 2
}
]
},
"Yanki": {
"totalMessages": 0,
"identifier": [
{
"name": "UT",
"staleCount": 4
},
{
"name": "UT_2",
"staleCount": 4
}
]
}
}
}
Request:
$request = 'http://localhost:8000/hi.json'
Invoke-WebRequest $request |
ConvertFrom-Json |
Select swindon
Response:
StatusCode : 200 StatusDescription : OK
Content : {
"MessagesMonitoring": {
"version": 1,
"description": "Message Description"
},
"data": {
"swindon": {
"totalMessages": 0,
"identifier": [
{
"na...
Not sure what I may be doing incorrectly. Any advise/guidance on how to parse the JSON into this format would be great.
swindon|identifier|ET|4
swindon|totalMessages|0
swindon|identifier|ET2|4
Reading|identifier|J3|2
Reading|totalMessages|0
Yanki|identifier|UT|4
Yanki|identifier|U_T|4
Yanki|totalMessages|0
You are missing a step. The Content element of the response contains the JSON, so that's what you need to feed into ConvertFrom-Json:
$request = 'http://localhost:8000/hi.json'
$resp = $(Invoke-WebRequest $request).Content | ConvertFrom-Json
Then, within the JSON you have a dictionary, within which the "data" key contains the information I think you're interested in, access it using this syntax:
$resp.data
That should get you started
this is my data structure:
[
{
"name": "name1",
"organizations": [
{
"name": "name2",
"spaces": [
{
"name": "name3",
"otherkey":"otherval"
},
{
"name": "name4",
"otherkey":"otherval"
}
]
}
]
},
{
"name": "name21",
"organizations": [
{
"name": "name22",
"spaces": [
{
"name": "name23",
"otherkey":"otherval"
},
{
"name": "name24",
"otherkey":"otherval"
}
]
}
]
}
]
i just want to keep name=name1, remove the nested array object with name=name4 and want to keep the rest of the object intact. I tried with map(select) but this will just give me the full object. Is it possible to work with del on specific subarrays and keep the rest as it is?
result should be the following. in addition i want to avoid enumeration all attributes to keep on outer objects:
[
{
"name": "name1",
"organizations": [
{
"name": "name2",
"spaces": [
{
"name": "name3",
"otherkey":"otherval"
}
]
}
]
}
]
any idea? thanks!
A very targeted solution would be:
path(.[0].organizations[0].spaces) as $target
| (getpath($target) | map(select(.name != "name4"))) as $new
| setpath($target; $new)
If permissible, though, you might consider:
walk(if type == "object" and .spaces|type == "array"
then .spaces |= map(select(.name != "name4"))
else . end)
or:
del(.. | .spaces? // empty | .[] | select(.name == "name4") )
(If your jq does not have walk/1 then its jq definition can easily be found by googling.)
You can use the below and it will remove the "name": "name4" array only.
jq 'del(.[] | .organizations? | .[] | .spaces?|.[] | select(.name? == "name4"))' yourJsonFile.json
Here is a solution using select, reduce, tostream and delpaths
map(
select(.name == "name1")
| reduce (tostream|select(length==2)) as [$p,$v] (
.
; if [$p[-1],$v] == ["name","name4"] then delpaths([$p[:-1]]) else . end
)
)
I took a similar approach as #peak but inverted it, so instead of selecting what you want and setting that in the output we're selecting what we don't want and deleting it.
[path(.organizations[0].spaces[]| select(.name == "name4")] as $trash | delpaths($trash)