I have, for complicated reasons involving a trip from an Apple plist through xml2json, a number of JSON files with data in this form:
{
"key": [ "key1", "key2", "key3" ],
"string": [ "value1", "value2", "value3" ]
}
And I would like to convert that to a normal JSON object:
{
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
After some head-banging, I came up with this jq program to do the trick:
jq '. as $d|[range(.key|length)|{"key":$d.key[.],"value":$d.string[.]}]|from_entries'
That works, but it seems a little convoluted. I was wondering if there were a cleaner solution?
This is indeed similar to this question, but the difference is that this is an object with named key and value elements instead just an array containing the two arrays directly.
The script you provide is already pretty good! This variation of your script saves the index into the variable instead of the input object, which feels more natural to read for me. It then creates an array of one key objects and adds them together.
jq '[range(.key | length) as $i | {(.key[$i]): .string[$i]}] | add'
When I first looked at this issue, I though that a zip builtin would improve the situation. Then I remembered: there is already a zip builtin! It's just called transpose. Using it, you can create a script such as this:
jq '[.key, .string] | transpose | map({key: .[0], value: .[1]}) | from_entries'
It seems easier to follow to me as well, although it is quite long too; I assume, however, that the focus is readability and not character count. Of course, you can also mix up both solutions:
jq '[.key, .string] | transpose | map({(.[0]): .[1]}) | add'
Here is a solution which uses reduce with a state object holding an iteration index and a result object. It iterates over .key setting corresponding values in the result from .string
.string as $v
| reduce .key[] as $k (
{idx:0, result:{}}; .result[$k] = $v[.idx] | .idx += 1
)
| .result
Related
{
"Key": "value"
"Results": [
{
"KeyIwant":"value"
...
}
]
}
I want to get a list of objects that have only the keys and their values that i specifiy.
So far Ive found something from the internet, but it creates objects and not a list and there are no commas.
jq '.Results | .[] | with_entries(select([.key] | inside(["key","key2", "key3"])))' input.json
For efficiency, you could use IN:
[.Results[]|with_entries(select(.key|IN("KeyIwant","etc"))) ]
If you want the whitelist to be presented as a JSON array, say $w, then write IN($w[])
Given a bash array, how to convert it to a JSON array in order to output to a file with jq?
Additionnally: is there a way to keep the server_nohup array unchanged instead of re-writing the whole json file each time?
newArray=(100 200 300)
jq -n --arg newArray $newArray '{
client_nohup: [
$newArray
],
server_nohup: [
]
}' > $projectDir/.watch.json
Current output:
{
"client_nohup": [
"100"
],
"server_nohup": []
}
Desired output:
{
"client_nohup": [
100,
200,
300
],
"server_nohup": []
}
(1) If all the values in newArray are valid as JSON values without spaces, then you could get away with piping the values as a stream, e.g.
newArray=(100 200 300)
echo "${newArray[#]}" |
jq -s '{client_nohup: ., server_nohup: []}'
(2)
Now let's suppose you merely wish to update the "nohup" object in a file, say nohup.json:
{ "client_nohup": [], "server_nohup": [ "keep me" ] }
Since you are using bash, you can then write:
echo "${newArray[#]}" |
jq -s --argjson nohup "$(cat nohup.json)" '
. as $newArray | $nohup | .client_nohup = $newArray
'
Output
(1)
{
"client_nohup": [
100,
200,
300
],
"server_nohup": []
}
(2)
{
"client_nohup": [
100,
200,
300
],
"server_nohup": [
"keep me"
]
}
Other cases
Where there's a will, there's a jq way :-)
See for example the accepted answer at How to format a bash array as a JSON array (though this is not a completely generic solution).
For a generic solution, see 𝑸: How can a variable number of arguments be passed to jq? How can a bash array of values be passed in to jq as a single argument? at the jq FAQ https://github.com/stedolan/jq/wiki/FAQ
Generic Solutions
To be clear, if the array values are known to be valid JSON, there are several good options; if the array values are arbitrary bash strings, then the only efficient, generic way to handle them with jq is by using the -R jq option (e.g. in conjunction with -s), but then the (bash) strings will all be read in as JSON strings, so any intended type information will be lost. (The point here hinges on the technicality that bash strings cannot CONTAIN NUL characters.)
Often, to alleviate the latter concern, one can convert numeric strings to JSON numbers, e.g. using the jq idiom: (tonumber? // .).
In general, the only truly safe way to do this is with multiple invocations of jq, adding each element to the output of the previous command.
arr='[]' # Empty JSON array
for x in "${newArray[#]}"; do
arr=$(jq -n --arg x "$x" --argjson arr "$arr" '$arr + [$x]')
done
This ensures that each element x of your bash array is properly encoded prior to adding it to the JSON array.
This is complicated, though, by the fact that bash doesn't not distinguish between numbers and strings. This encodes your array as ["100", "200", "300"], not [100, 200, 300]. In the end, you need to have some awareness of what your array contains, and preprocess it accordingly.
the file is
{
"ContentKey--4-0-47--Vovb1BQ": ["infra", "qa", "qa-ContentKey-4-0-47-Vovb1BQ", "internal-qa-Conten-WebServi-19E4PUWHRGD44-460820639.us-east-1.elb.amazonaws.com", "plan--default"],
"ContentKey--4-0-47--zjOkiQ": ["svc", "dev", "dev-ContentKey-4-0-47-zjOkiQ", "dev-Conte-WebServi-KXJXZBDY113W-2116785917.us-east-1.elb.amazonaws.com", "plan--default"],
"IdGenService--2001-4-22--CJUFaMQ": ["svc", "dev", "dev-IdGenService-2001-4-22-CJUFaMQ", "dev-IdGen-WebServi-R7RVXSYAV92W-304073075.us-east-1.elb.amazonaws.com"],
"IdGenService--2001-4-22--Uhf9CTQ": ["svc", "qa", "qa-IdGenService-2001-4-22-Uhf9CTQ", "internal-qa-IdGenS-WebServi-RT5BI5EEVZP3-665537643.us-east-1.elb.amazonaws.com"]
}
I want to find the list of keys whose array value have the entry svc
i could get the following to work
cat list.json | jq '. | map(select (. | contains(["svc"])))'
But the output is the value array and not the key itself
[
[
"svc",
"dev",
"dev-ContentKey-4-0-47-zjOkiQ",
"dev-Conte-WebServi-KXJXZBDY113W-2116785917.us-east-1.elb.amazonaws.com",
"plan--default"
],
[
"svc",
"dev",
"dev-IdGenService-2001-4-22-CJUFaMQ",
"dev-IdGen-WebServi-R7RVXSYAV92W-304073075.us-east-1.elb.amazonaws.com"
],
[
"svc",
"qa",
"qa-IdGenService-2001-4-22-Uhf9CTQ",
"internal-qa-IdGenS-WebServi-RT5BI5EEVZP3-665537643.us-east-1.elb.amazonaws.com"
]
]
With your input, the following filter yields the output as shown:
to_entries[] | select( .value | index("svc") ) | .key
Output:
"ContentKey--4-0-47--zjOkiQ"
"IdGenService--2001-4-22--CJUFaMQ"
"IdGenService--2001-4-22--Uhf9CTQ"
In cases like this, using index/1 is both simpler and (potentially much) faster than using any/2.
The top-level object in your json is an object, not an array. So .[] would only yield its values and discard the keys. Use with_entries/1 to filter that object. This converts an object to an array of key/value pairs and back with which you can apply filters to.
$ jq --arg key 'svc' 'with_entries(select(any(.value[]; . == $key)))' list.json
Also, you should avoid using contains/1 here. It's applied recursively so it will also match strings that contain the substring svc. i.e., "Foosvcbar" will be matched.
I'm trying to use jq to combine two arrays and running into a bit of trouble.
I'm attempting to parse out the data from netdata (netdata.firehol.org) and the two pieces of data within the json response that I"m interested in are both part of an array. The first array is labels for the datapoints in the second array.
Sample Input
[
"time",
"guest_nice",
"guest",
"steal",
"softirq",
"irq",
"user",
"system",
"nice",
"iowait"
]
[
1460728600,
0,
0,
0,
0.45731,
0,
0.25108,
11.74702,
48.22465,
0
]
Input
If you want to grab fresh data yourself to test against, you can use the following:
curl -s -X GET --header 'Accept: application/json'
'http://netdata.firehol.org/api/v1/data?chart=system.cpu&after=-10&before=0&points=1&group=average&format=json&options=seconds%2Cjsonwrap' | jq '.result.labels, .result.data[]'
I've tried to use map() as well as trying to assign vars to both arrays and then print out the objects together, but have been unsuccessful (below).
Code
| jq '.result.labels as $labels | .result.data[] as $data | .result.data[] | Label: $labels[.], data: $data[.]}'
I appreciate anyone's insight in advance as I'm a little stuck, and would prefer to be able to do this all in jq rather than using for loops in bash (if possible).
Expected Ouput
{
"time": "1460728600",
"guest_nice": "0",
...
}
You haven't specified exactly how you want the arrays to be combined, but one approach is to use transpose, which in this case is effectively a kind of zip. For example:
$ jq -n -c '[["a","b"], [1,2]] | transpose'
yields: [["a",1],["b",2]]
If you wanted an array of objects, then with the same input,
transpose | map( { (.[0]) : .[1] } )
would yield: [{"a":1},{"b":2}]
If your jq does not have transpose, here is its definition:
# transpose a possibly jagged matrix, quickly;
# rows are padded with nulls so the result is always rectangular.
def transpose:
[range(0; (map(length) | max)) as $j
| [range(0; length) as $i | .[$i][$j] ] ] ;
Alternatively, if you would prefer a very brief zip:
def zip: [range(0; .[0]|length) as $i | [.[0][$i], .[1][$i]]];
Here is a solution that handles the general case where the first array contains the key names and the following arrays contain values using transpose and from_entries
{h:.[0], v:.[1:][]} # {h:[keys], v:[values]}
| [.h, .v] # [ [keys], [values] ] ...
| [ transpose[] | {key:.[0], value:.[1]} ] # [ {"key":key, "value":value}, ... ]
| from_entries # { key:value, key:value, ... }
For example, if this filter is in filter.jq and data.json contains
["time","guest_nice","guest","steal","softirq","irq","user","system","nice","iowait"]
[1460728600,0,0,0,0.45731,0,0.25108,11.74702,48.22465,0]
[1460728601,0,0,0,0.45732,0,0.25109,12.74703,49,0]
then the command
jq -M -s -c -f filter.jq data.json
produces
{"time":1460728600,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45731,"irq":0,"user":0.25108,"system":11.74702,"nice":48.22465,"iowait":0}
{"time":1460728601,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45732,"irq":0,"user":0.25109,"system":12.74703,"nice":49,"iowait":0}
I've been banging my head against the wall for several hours on this and just can't seem to find a way to do this. I have an array of keys and an array of values, how can I generate an object? Input:
[["key1", "key2"], ["val1", "val2"]]
Output:
{"key1": "val1", "key2": "val2"}
Resolved this on github:
.[0] as $keys |
.[1] as $values |
reduce range(0; $keys|length) as $i ( {}; . + { ($keys[$i]): $values[$i] })
The current version of jq has a transpose filter that can be used to pair up the keys and values. You could use it to build out the result object rather easily.
transpose | reduce .[] as $pair ({}; .[$pair[0]] = $pair[1])
Just to be clear:
(0) Abdullah Jibaly's solution is simple, direct, efficient and generic, and should work in all versions of jq;
(1) transpose/0 is a builtin in jq 1.5 and has been available in pre-releases since Oct 2014;
(2) using transpose/0 (or zip/0 as defined above), an even shorter but still simple, fast, and generic solution to the problem is:
transpose | map( {(.[0]): .[1]} ) | add
Example:
$ jq 'transpose | map( {(.[0]): .[1]} ) | add'
Input:
[["k1","k2","k3"], [1,2,3] ]
Output:
{
"k1": 1,
"k2": 2,
"k3": 3
}
Scratch this, it doesn't actually work for any array greater than size 2.
[map(.[0]) , map(.[1])] | map({(.[0]):.[1]}) | add
Welp, I thought this would be easy, having a little prolog experience... oh man. I ended up banging my head against a wall too. Don't think I'll ever use jq ever again.
Here is a solution which uses reduce with a state object holding an iteration index and a result object. It iterates over the keys in .[0] setting corresponding values in the result from .[1]
.[1] as $v
| reduce .[0][] as $k (
{idx:0, result:{}}; .result[$k] = $v[.idx] | .idx += 1
)
| .result