I have a series of JSON files that I've been working with via jq. Each file consists of an array of dictionaries, e.g.
file1.json: [{ "id": 1 }]
file2.json: [{ "id": 2 }]
The only command I have found which successfully merges all input files into one output array is:
jq --slurp '.[0] + .[1]' file1.json file2.json
This command outputs [{ "id": 1 }, { "id": 2 }], as expected.
I'm writing a shell script which is expected to merge a variable set of files into a single JSON array as an output. My script will execute something like:
find . -type f -iname '*.json' | xargs jq 'FILTER'
This should invoke jq like jq 'FILTER' file1.json file2.json ....
Is there a feature that I'm missing that will take all input files and first merge them into one contiguous list of objects without having to rewrite the filter to something like .[0] + .[1] + .[2] ...?
Given:
1.json
[{ "id": 1 }]
2.json
[{ "id": 2 }]
3.json
[{ "id": 3 }]
Then this command:
jq --slurp 'map(.[])' 1.json 2.json 3.json
Returns:
[
{
"id": 1
},
{
"id": 2
},
{
"id": 3
}
]
Or simply:
jq --slurp 'flatten' 1.json 2.json 3.json
It's generally best to avoid the -s option, especially if your version of jq supports inputs, as do all versions >= 1.5.
In any case, if your version of jq supports inputs, you could write:
jq -n '[inputs[]]' 1.json 2.json 3.json # etc
or whichever variant meets your needs.
Otherwise, you could simply write:
jq -s add 1.json 2.json 3.json # etc
Note on flatten
flatten itself is ruthless:
$ jq flatten <<< '[[[1], [[[[2]]]]]]'
[1,2]
flatten(1) is less so.
Related
Given the example JSON below:
{
"account_number": [
"123456"
],
"account_name": [
"name"
],
"account_id": [
654321
],
"username": [
"demo"
]
}
I'd like to get:
{
"account_number": "123456",
"account_name": "name",
"account_id": 654321,
"username": "demo"
}
Currently, I'm brute forcing it with | sed 's/\[//g' | sed 's/\]//g' | jq '.' ... but of course, that's ugly and causes issues if any of the values contain [ or ].
I've been unsuccessful with jq's flatten and other loops and mapping techniques like | jq -s '{Item:.[]} | .Item |add' to try and flatten the single-item arrays. Ideally, it would work where it would flatten arrays [...] to flat elements/objects {...}. Either way something better than replacing all occurrences of square brackets.
Short and sweet:
map_values(first)
Use with_entries, changing each value to the first element of itself:
jq 'with_entries(.value |= .[0])' file.json
I have some JSON data which is pretty typical CSV-style data, however it's represented in JSON. I am struggling to figure out the correct jq expression to convert the following JSON back to some JSON which can generate the appropriate CSV with #csv.
There's a fixed number of 'columns', i.e. the "AAA" values, but the number of values in each 'column' is dynamic yet fixed across columns. That is, the length of the arrays "AAA", "BBB", "CCC", etc are all the same, but the length is dynamic and can change between data sets.
Input (note invalid numbers present, to illustrate example):
{
"AAA": [
111.1,
111.2,
111.3,
111..,
111.n
],
"BBB": [
222.1,
222.2,
222.3,
222..,
222.n
],
"CCC": [
333.1,
333.2,
333.3,
333..,
333.n
],
"DDD": [
444.1,
444.2,
444.3,
444..,
444.n
],
"EEE": [
555.1,
555.2,
555.3,
555..,
555.n
]
}
Desired output (note invalid numbers present, to illustrate example):
{
[
"AAA",
"BBB",
"CCC",
"DDD",
"EEE"
],
[
111.1,
222.1,
333.1,
444.1,
555.1
],
[
111.2,
222.2,
333.2,
444.2,
555.2
],
[
111.3,
222.3,
333.3,
444.3,
555.3
],
[
111..,
222..,
333..,
444..,
555..
],
[
111.n,
222.n,
333.n,
444.n,
555.n
]
}
Here is the desired CSV, for illustration purposes (as converting with #csv is pretty straightforward):
AAA,BBB,CCC,DDD,EEE
111.1,222.1,333.1,444.1,555.1
111.2,222.2,333.2,444.2,555.2
111.3,222.3,333.3,444.3,555.3
111..,222..,333..,444..,555..
111.n,222.n,333.n,444.n,555.n
If the required expression is far easier without the first array in the result object containing the "AAA" 'header' values then I can easily live without them.
Thank you.
You can use the transpose function in jq to do the transposing of arrays, formed from keys/values.
jq '[ to_entries[] | [.key, .value[]] ] | transpose'
The bulk of the magic is performed by the transpose built-in, but before that you just need to collect the values into an array of arrays. The CSV result can be generated with the #csv function.
jq --raw-output '[ to_entries[] | [.key, .value[]] ] | transpose[] | #csv'
You could also use map() and be avoid the redundant [..]
jq 'to_entries | map([.key, .value[]]) | transpose'
jq --raw-output 'to_entries | map([.key, .value[]]) | transpose[] | #csv'
I have a dictionary which looks like:
cat dictionary.json
[
{
"key": "key01",
"value": "value01"
},
{
"key": "key02",
"value": "value02"
},
{
"key": "key03",
"value": "value03",
"extraProperty": {
"foo": "bar"
}
},
{
"key": "key04",
"value": "value04"
}
]
Then, I have an array which is:
echo $array
key01 key02 key03
Expected output:
value01 value02 value03
I have some trouble to make jq using an array which is not json format.
I tried various solutions that I found, but none of them worked.
This post jq - How to select objects based on a 'whitelist' of property values seems to solve a similar problem but it doesn't work with my input:
echo $array | jq --argfile whitelist dictionary.json 'select(any(.key== $whitelist[]; .value))'
parse error: Invalid numeric literal at line 1, column 6
I also tried to use
jq -n --arg array $array --argfile whitelist dico.json 'select(any(.key== $whitelist[]; .valuee))'
jq: error: key02/0 is not defined at <top-level>, line 1:
key02
jq: 1 compile error
Thanks!
Here
jq -r --arg array "$array" \
'from_entries | .[($array | split(" "))[]]' \
dictionary.json
Output
value01
value02
value03
See man jq for further information.
Using INDEX/2, which constructs a dictionary:
echo 'key01 key02 key03' |
jq -Rr --argfile dict dictionary.json '
INDEX($dict[]; .key) as $d
| split(" ") | map( $d[.]|.value )
| join(" ")'
yields:
value01 value02 value03
If your jq does not have INDEX, then now would be an excellent time to upgrade to jq 1.6; alternatively, you can simply snarf its def by googling: jq "def INDEX"
I have a json array that looks like this:
{
"StackSummaries": [
{
"CreationTime": "2016-06-01T22:22:49.890Z",
"StackName": "foo-control-eu-west-1",
"StackStatus": "UPDATE_COMPLETE",
"LastUpdatedTime": "2016-06-01T22:47:58.433Z"
},
{
"CreationTime": "2016-04-13T11:22:04.250Z",
"StackName": "foo-bar-testing",
"StackStatus": "UPDATE_COMPLETE",
"LastUpdatedTime": "2016-04-26T16:17:07.570Z"
},
{
"CreationTime": "2016-04-10T01:09:49.428Z",
"StackName": "foo-ldap-eu-west-1",
"StackStatus": "UPDATE_COMPLETE",
"LastUpdatedTime": "2016-04-17T13:44:04.758Z"
}
]
}
I am looking to create text output that looks like this:
foo-control-eu-west-1
foo-bar-testing
foo-ldap-eu-west-1
Is jq able to do this? Specifically, what would the jq command line be that would select each StackName in the array and output each key one per line?
jq --raw-output '.StackSummaries[].StackName'
$ jq -r '[.StackSummaries[] | .StackName] | unique[]' input.json
foo-bar-testing
foo-control-eu-west-1
foo-ldap-eu-west-1
The -r option strips the quotation marks from the output. You might not want the call to 'unique'.
For reference, if you wanted all the key names:
$ jq '[.StackSummaries[] | keys[]] | unique' input.json
[
"CreationTime",
"LastUpdatedTime",
"StackName",
"StackStatus"
]
Here is another solution
jq -M -r '..|.StackName?|values' input.json
I'm trying to use jq to combine two arrays and running into a bit of trouble.
I'm attempting to parse out the data from netdata (netdata.firehol.org) and the two pieces of data within the json response that I"m interested in are both part of an array. The first array is labels for the datapoints in the second array.
Sample Input
[
"time",
"guest_nice",
"guest",
"steal",
"softirq",
"irq",
"user",
"system",
"nice",
"iowait"
]
[
1460728600,
0,
0,
0,
0.45731,
0,
0.25108,
11.74702,
48.22465,
0
]
Input
If you want to grab fresh data yourself to test against, you can use the following:
curl -s -X GET --header 'Accept: application/json'
'http://netdata.firehol.org/api/v1/data?chart=system.cpu&after=-10&before=0&points=1&group=average&format=json&options=seconds%2Cjsonwrap' | jq '.result.labels, .result.data[]'
I've tried to use map() as well as trying to assign vars to both arrays and then print out the objects together, but have been unsuccessful (below).
Code
| jq '.result.labels as $labels | .result.data[] as $data | .result.data[] | Label: $labels[.], data: $data[.]}'
I appreciate anyone's insight in advance as I'm a little stuck, and would prefer to be able to do this all in jq rather than using for loops in bash (if possible).
Expected Ouput
{
"time": "1460728600",
"guest_nice": "0",
...
}
You haven't specified exactly how you want the arrays to be combined, but one approach is to use transpose, which in this case is effectively a kind of zip. For example:
$ jq -n -c '[["a","b"], [1,2]] | transpose'
yields: [["a",1],["b",2]]
If you wanted an array of objects, then with the same input,
transpose | map( { (.[0]) : .[1] } )
would yield: [{"a":1},{"b":2}]
If your jq does not have transpose, here is its definition:
# transpose a possibly jagged matrix, quickly;
# rows are padded with nulls so the result is always rectangular.
def transpose:
[range(0; (map(length) | max)) as $j
| [range(0; length) as $i | .[$i][$j] ] ] ;
Alternatively, if you would prefer a very brief zip:
def zip: [range(0; .[0]|length) as $i | [.[0][$i], .[1][$i]]];
Here is a solution that handles the general case where the first array contains the key names and the following arrays contain values using transpose and from_entries
{h:.[0], v:.[1:][]} # {h:[keys], v:[values]}
| [.h, .v] # [ [keys], [values] ] ...
| [ transpose[] | {key:.[0], value:.[1]} ] # [ {"key":key, "value":value}, ... ]
| from_entries # { key:value, key:value, ... }
For example, if this filter is in filter.jq and data.json contains
["time","guest_nice","guest","steal","softirq","irq","user","system","nice","iowait"]
[1460728600,0,0,0,0.45731,0,0.25108,11.74702,48.22465,0]
[1460728601,0,0,0,0.45732,0,0.25109,12.74703,49,0]
then the command
jq -M -s -c -f filter.jq data.json
produces
{"time":1460728600,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45731,"irq":0,"user":0.25108,"system":11.74702,"nice":48.22465,"iowait":0}
{"time":1460728601,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45732,"irq":0,"user":0.25109,"system":12.74703,"nice":49,"iowait":0}