Say I have a sample.json file that looks like this. Notice the objects are not separated by a comma. I am aware that this is not a JSON file without the commas.
[
{
"type": "message",
"user": "U024HFHU5",
"text": "hey there",
"ts": "1385407681.000003"
}
{
"type": "message",
"user": "U024HGJ4E",
"text": "right back at you",
"ts": "1385407706.000006"
}
]
How do I make it look like this (example from I got the example from https://thoughtbot.com/blog/jq-is-sed-for-json.)
[
{
"type": "message",
"user": "U024HFHU5",
"text": "hey there",
"ts": "1385407681.000003"
},
{
"type": "message",
"user": "U024HGJ4E",
"text": "right back at you",
"ts": "1385407706.000006"
}
]
I tried doing },{ as some have suggested but it does not work.
I think it's probably something like this in jq on the command line but am not sure. Here, my hope is that sample2.json outputs the desired format.
jq -r '. | join(", ")' sample.json > sample2.json
If the square brackets were not in the input, jq could process it like normal as a stream of objects. You could then throw them into an array or whatever.
You'll just need to invoke jq twice in this case. First read it in raw (-Rs) and trim out the leading and trailing brackets, output it raw (-r), then process in jq like normal.
$ jq -Rsr 'gsub("^\\s*\\[|\\]\\s*$"; "")' input.json | jq -n '[inputs]'
Using human JSON utility to recover your malformed JSON file:
hjson -j file
[
{
"type": "message",
"user": "U024HFHU5",
"text": "hey there",
"ts": "1385407681.000003"
},
{
"type": "message",
"user": "U024HGJ4E",
"text": "right back at you",
"ts": "1385407706.000006"
}
]
The option -j outputs the file with correct JSON syntax, by adding the missing comma between objects.
Related
I'm using the AWS CLI command aws iam list-users in Powershell 7.2.5 and trying to convert the output from JSON to a Powershell Object.
The JSON output of the above command looks like this (actual values omitted):
{
"Users": [
{
"Path": "/",
"UserName": "username1",
"UserId": "userid1",
"Arn": "arnid1",
"CreateDate": "createddate"
},
{
"Path": "/",
"UserName": "username2",
"UserId": "userid2",
"Arn": "arnid2",
"CreateDate": "createddate"
},
{
"Path": "/",
"UserName": "username3",
"UserId": "userid3",
"Arn": "arnid3",
"CreateDate": "createddate"
}
]
}
When I try and run the following code to create an array of Powershell Objects the output comes out as one Object.
$users = ConvertFrom-Json -InputObject $usersJson
Users
-----
{#{Path=/; UserName=username1; UserId=userid1; Arn=arnid1; CreateDate=createddate}, #{Path=/; UserName=username2; UserId=userid2; Arn=arnid2; CreateDate=createddate}, #{Path=/; UserName=username3; UserId=userid3; Arn=arnid3; CreateDate=createddate}}
Have read over countless posts and am now at a loss of what to do.
Any help would be greatly appreciated.
Santiago's comment was correct:
$users.Users shows all the entries as individual objects.
This question already has an answer here:
Output the results of select operation in an array - jq
(1 answer)
Closed 2 years ago.
I'm using the select function with jq to parse for items in a list that contain a certain value. I want the resulting output to be a json list, but jq gives me the objects individually:
Example:
$ a='{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}'
$ echo $a | jq '.[] | select(.location=="Stockholm")'
{
"name": "Donald",
"location": "Stockholm"
}
{
"name": "Walt",
"location": "Stockholm"
}
Instead I want the output to be a json list like this:
[
{
"name": "Donald",
"location": "Stockholm"
},
{
"name": "Walt",
"location": "Stockholm"
}
]
How can I do this with jq?
In general, you can stick square brackets around any expression to gather all its outputs into an array.
[.[] | select(.location=="Stockholm")]
Sometimes it makes sense not to break up the input array in the first place, but use map to transform it:
map(select(.location=="Stockholm"))
I have the below format of JSON file which is having issues[] array and I tried to use it for Kibana. But unfortunately Kibana doesn't support nested objects and array and there is a plugin to utilize so that I need to downgrade which I can't do right now because in that case I will lose all my data.
Sample data:
{
"expand": "schema,names",
"startAt": 0,
"maxResults": 50,
"total": 4,
"issues": [{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "1999875",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/1999875",
"key": "KINDLEAMZ-67578",
"fields": {
"summary": "contingency is displaying for confirmed card.",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
},
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "2019428",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/2019428",
"key": "KINDLEAMZ-68661",
"fields": {
"summary": "card",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
},
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "2010958",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/2010958",
"key": "KINDLEAMZ-68167",
"fields": {
"summary": "Test Card",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
}
]
}
So I just planned to restructure this payload like all the issues[] into as an object and write down in separate file. So that I can avoid that issue.
Expected output:
For that above sample data I have 4 records in issues[].length so I just want to create 4 different files with below format:
File1.json:
{
"key": "KINDLEAMZ-67578",
"summary": "contingency is displaying for confirmed card.",
"name": "P1",
"created": "2019-09-23T11:25:21.000+0000"
}
The same way I want to looping the other arrays and get the values as like above and write down in File2.json, File3.json, and File4.json.
Since the data is dynamic and so I just want this file creation happen based of length of issues[] array.
Is there anyway to achieve this by using shell script? Or any CLI library.
Please advise me.
Specify -c/--compact-output flag to make jq put each entity on a single, separate line, then use awk to write each line to a separate file.
jq -c '.issues[] | {
key,
summary: .fields.summary,
name: .fields.priority.name,
created: .fields.created
}' file | awk '{
f = ("file" NR ".json")
print > f
close(f)
}'
Using GNU awk and extension gawk-json:
awk '
#load "json"
{
lines=lines $0
if(json_fromJSON(lines,data)==1){
for(i in data["issues"]) {
out["key"] = data["issues"][i]["key"]
out["summary"] = data["issues"][i]["fields"]["summary"]
out["created"] = data["issues"][i]["fields"]["created"]
out["name"] = data["issues"][i]["fields"]["priority"]["name"]
file="file" i ".json"
print json_toJSON(out) > file
close(file)
delete out
}
}
}' file.json
Output:
$ cat file1.json | jq '.' # useless use of cat but used to emphasize
{
"created": "2019-09-23T11:25:21.000+0000",
"key": "KINDLEAMZ-67578",
"summary": "contingency is displaying for confirmed card.",
"name": "P1"
}
I have the following data structure as a result of aws logs get-query-results:
{
"status": "Complete",
"statistics": {
"recordsMatched": 2.0,
"recordsScanned": 13281.0,
"bytesScanned": 7526096.0
},
"results": [
[
{
"field": "time",
"value": "2019-01-31T21:53:01.136Z"
},
{
"field": "requestId",
"value": "a9c233f7-0b1b-3326-9b0f-eba428e4572c"
},
{
"field": "logLevel",
"value": "INFO"
},
{
"field": "callerId",
"value": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
],
[
{
"field": "time",
"value": "2019-01-25T13:13:01.062Z"
},
{
"field": "requestId",
"value": "a4332628-1b9b-a9c2-0feb-0cd4a3f7cb63"
},
{
"field": "logLevel",
"value": "INFO"
},
{
"field": "callerId",
"value": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
],
]
}
The AWS CLI support JMESPath language for filtering output. I need to apply a query string, to filter among the returned "results" the objects that contain the "callerId" as a "field", retrieve the "value" property and obtain the following output:
[
{
callerId: "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
},
{
callerId: "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
]
The first step I do is flatter the results array with the query string: results[]
This will get read of the other root properties (status, statistics) and return only one big array with all of the {field: ..., value: ...} alike objects. But after this I can't manage to properly filter for those objects that match field=="callerId". I tried, among others, the following expressions without success:
'results[][?field=="callerId"]'
'results[][*][?field=="callerId"]'
'results[].{ callerId: #[?field=="callerId"].value }'
I'm not an expert in JMESPath and I was doing the tutorials of the jmespath.org site but couldn't manage to make it work.
Thanks!
Using jq is a good thing because it's more complete language, but if you want to do it with JMES Path here the solution:
results[*][?field=='callerId'].{callerId: value}[]
to get:
[
{
"callerId": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
},
{
"callerId": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
]
I'm not able to reproduce fully since I don't have the same logs in my log stream but I was able to do this using jq and putting the sample JSON object in a file
cat sample_output.json | jq '.results[][] | select(.field=="callerId") | .value'
OUTPUT:
"a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
"a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
you could pipe the output from the aws cli to jq.
I was able to get pretty close with the native JMESPath query and using the built in editor in this site
http://jmespath.org/examples.html#filtering-and-selecting-nested-data
results[*][?field==`callerId`][]
OUTPUT:
[
{
"field": "callerId",
"value": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
},
{
"field": "callerId",
"value": "a9b0f9c2-eb42-3986-33f7-8e450b1b72cf"
}
]
but I'm not sure how to get callerId to be the key and the value to be the value from another key.
I have some restmethods with powershell (VMWare vRA API) where I get a template for a development request and then I need to fill that (JSON format).
The "data" part of the template has different properties like:
Name : Test
selectedNetworks :
selectedServices :
My question:
How do I get my array $networks (network1,network2,network3) to fill the json element "selectedNetworks" that it becomes like this:
"key": "selectedNetworks",
"value": {
"type": "multiple",
"elementTypeId": "STRING",
"items": [
{
"type": "string",
"value": "network1"
},
{
"type": "string",
"value": "network2"
},
{
"type": "string",
"value": "network3"
}
]
}
I know how to add in "simple" values like the name is
$WebRequestBodyData.Name="$Name"
But how do I get the complex format above into $WebRequestBodyData.selectedNetworks?
Any help is greatly appreciated!
Thanks and best regards,
Ville
I could solve it like that:
$networkjson = '{"type": "multiple", "elementTypeId": "STRING", "items": [] }' | ConvertFrom-Json
foreach($nwitem in $networks){
$networkjson.items += $nwitem
}
$BodyData.selectedNetworks += $networkjson