the file is
{
"ContentKey--4-0-47--Vovb1BQ": ["infra", "qa", "qa-ContentKey-4-0-47-Vovb1BQ", "internal-qa-Conten-WebServi-19E4PUWHRGD44-460820639.us-east-1.elb.amazonaws.com", "plan--default"],
"ContentKey--4-0-47--zjOkiQ": ["svc", "dev", "dev-ContentKey-4-0-47-zjOkiQ", "dev-Conte-WebServi-KXJXZBDY113W-2116785917.us-east-1.elb.amazonaws.com", "plan--default"],
"IdGenService--2001-4-22--CJUFaMQ": ["svc", "dev", "dev-IdGenService-2001-4-22-CJUFaMQ", "dev-IdGen-WebServi-R7RVXSYAV92W-304073075.us-east-1.elb.amazonaws.com"],
"IdGenService--2001-4-22--Uhf9CTQ": ["svc", "qa", "qa-IdGenService-2001-4-22-Uhf9CTQ", "internal-qa-IdGenS-WebServi-RT5BI5EEVZP3-665537643.us-east-1.elb.amazonaws.com"]
}
I want to find the list of keys whose array value have the entry svc
i could get the following to work
cat list.json | jq '. | map(select (. | contains(["svc"])))'
But the output is the value array and not the key itself
[
[
"svc",
"dev",
"dev-ContentKey-4-0-47-zjOkiQ",
"dev-Conte-WebServi-KXJXZBDY113W-2116785917.us-east-1.elb.amazonaws.com",
"plan--default"
],
[
"svc",
"dev",
"dev-IdGenService-2001-4-22-CJUFaMQ",
"dev-IdGen-WebServi-R7RVXSYAV92W-304073075.us-east-1.elb.amazonaws.com"
],
[
"svc",
"qa",
"qa-IdGenService-2001-4-22-Uhf9CTQ",
"internal-qa-IdGenS-WebServi-RT5BI5EEVZP3-665537643.us-east-1.elb.amazonaws.com"
]
]
With your input, the following filter yields the output as shown:
to_entries[] | select( .value | index("svc") ) | .key
Output:
"ContentKey--4-0-47--zjOkiQ"
"IdGenService--2001-4-22--CJUFaMQ"
"IdGenService--2001-4-22--Uhf9CTQ"
In cases like this, using index/1 is both simpler and (potentially much) faster than using any/2.
The top-level object in your json is an object, not an array. So .[] would only yield its values and discard the keys. Use with_entries/1 to filter that object. This converts an object to an array of key/value pairs and back with which you can apply filters to.
$ jq --arg key 'svc' 'with_entries(select(any(.value[]; . == $key)))' list.json
Also, you should avoid using contains/1 here. It's applied recursively so it will also match strings that contain the substring svc. i.e., "Foosvcbar" will be matched.
Related
I have some JSON data which is pretty typical CSV-style data, however it's represented in JSON. I am struggling to figure out the correct jq expression to convert the following JSON back to some JSON which can generate the appropriate CSV with #csv.
There's a fixed number of 'columns', i.e. the "AAA" values, but the number of values in each 'column' is dynamic yet fixed across columns. That is, the length of the arrays "AAA", "BBB", "CCC", etc are all the same, but the length is dynamic and can change between data sets.
Input (note invalid numbers present, to illustrate example):
{
"AAA": [
111.1,
111.2,
111.3,
111..,
111.n
],
"BBB": [
222.1,
222.2,
222.3,
222..,
222.n
],
"CCC": [
333.1,
333.2,
333.3,
333..,
333.n
],
"DDD": [
444.1,
444.2,
444.3,
444..,
444.n
],
"EEE": [
555.1,
555.2,
555.3,
555..,
555.n
]
}
Desired output (note invalid numbers present, to illustrate example):
{
[
"AAA",
"BBB",
"CCC",
"DDD",
"EEE"
],
[
111.1,
222.1,
333.1,
444.1,
555.1
],
[
111.2,
222.2,
333.2,
444.2,
555.2
],
[
111.3,
222.3,
333.3,
444.3,
555.3
],
[
111..,
222..,
333..,
444..,
555..
],
[
111.n,
222.n,
333.n,
444.n,
555.n
]
}
Here is the desired CSV, for illustration purposes (as converting with #csv is pretty straightforward):
AAA,BBB,CCC,DDD,EEE
111.1,222.1,333.1,444.1,555.1
111.2,222.2,333.2,444.2,555.2
111.3,222.3,333.3,444.3,555.3
111..,222..,333..,444..,555..
111.n,222.n,333.n,444.n,555.n
If the required expression is far easier without the first array in the result object containing the "AAA" 'header' values then I can easily live without them.
Thank you.
You can use the transpose function in jq to do the transposing of arrays, formed from keys/values.
jq '[ to_entries[] | [.key, .value[]] ] | transpose'
The bulk of the magic is performed by the transpose built-in, but before that you just need to collect the values into an array of arrays. The CSV result can be generated with the #csv function.
jq --raw-output '[ to_entries[] | [.key, .value[]] ] | transpose[] | #csv'
You could also use map() and be avoid the redundant [..]
jq 'to_entries | map([.key, .value[]]) | transpose'
jq --raw-output 'to_entries | map([.key, .value[]]) | transpose[] | #csv'
{
"Key": "value"
"Results": [
{
"KeyIwant":"value"
...
}
]
}
I want to get a list of objects that have only the keys and their values that i specifiy.
So far Ive found something from the internet, but it creates objects and not a list and there are no commas.
jq '.Results | .[] | with_entries(select([.key] | inside(["key","key2", "key3"])))' input.json
For efficiency, you could use IN:
[.Results[]|with_entries(select(.key|IN("KeyIwant","etc"))) ]
If you want the whitelist to be presented as a JSON array, say $w, then write IN($w[])
Given a bash array, how to convert it to a JSON array in order to output to a file with jq?
Additionnally: is there a way to keep the server_nohup array unchanged instead of re-writing the whole json file each time?
newArray=(100 200 300)
jq -n --arg newArray $newArray '{
client_nohup: [
$newArray
],
server_nohup: [
]
}' > $projectDir/.watch.json
Current output:
{
"client_nohup": [
"100"
],
"server_nohup": []
}
Desired output:
{
"client_nohup": [
100,
200,
300
],
"server_nohup": []
}
(1) If all the values in newArray are valid as JSON values without spaces, then you could get away with piping the values as a stream, e.g.
newArray=(100 200 300)
echo "${newArray[#]}" |
jq -s '{client_nohup: ., server_nohup: []}'
(2)
Now let's suppose you merely wish to update the "nohup" object in a file, say nohup.json:
{ "client_nohup": [], "server_nohup": [ "keep me" ] }
Since you are using bash, you can then write:
echo "${newArray[#]}" |
jq -s --argjson nohup "$(cat nohup.json)" '
. as $newArray | $nohup | .client_nohup = $newArray
'
Output
(1)
{
"client_nohup": [
100,
200,
300
],
"server_nohup": []
}
(2)
{
"client_nohup": [
100,
200,
300
],
"server_nohup": [
"keep me"
]
}
Other cases
Where there's a will, there's a jq way :-)
See for example the accepted answer at How to format a bash array as a JSON array (though this is not a completely generic solution).
For a generic solution, see 𝑸: How can a variable number of arguments be passed to jq? How can a bash array of values be passed in to jq as a single argument? at the jq FAQ https://github.com/stedolan/jq/wiki/FAQ
Generic Solutions
To be clear, if the array values are known to be valid JSON, there are several good options; if the array values are arbitrary bash strings, then the only efficient, generic way to handle them with jq is by using the -R jq option (e.g. in conjunction with -s), but then the (bash) strings will all be read in as JSON strings, so any intended type information will be lost. (The point here hinges on the technicality that bash strings cannot CONTAIN NUL characters.)
Often, to alleviate the latter concern, one can convert numeric strings to JSON numbers, e.g. using the jq idiom: (tonumber? // .).
In general, the only truly safe way to do this is with multiple invocations of jq, adding each element to the output of the previous command.
arr='[]' # Empty JSON array
for x in "${newArray[#]}"; do
arr=$(jq -n --arg x "$x" --argjson arr "$arr" '$arr + [$x]')
done
This ensures that each element x of your bash array is properly encoded prior to adding it to the JSON array.
This is complicated, though, by the fact that bash doesn't not distinguish between numbers and strings. This encodes your array as ["100", "200", "300"], not [100, 200, 300]. In the end, you need to have some awareness of what your array contains, and preprocess it accordingly.
I have, for complicated reasons involving a trip from an Apple plist through xml2json, a number of JSON files with data in this form:
{
"key": [ "key1", "key2", "key3" ],
"string": [ "value1", "value2", "value3" ]
}
And I would like to convert that to a normal JSON object:
{
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
After some head-banging, I came up with this jq program to do the trick:
jq '. as $d|[range(.key|length)|{"key":$d.key[.],"value":$d.string[.]}]|from_entries'
That works, but it seems a little convoluted. I was wondering if there were a cleaner solution?
This is indeed similar to this question, but the difference is that this is an object with named key and value elements instead just an array containing the two arrays directly.
The script you provide is already pretty good! This variation of your script saves the index into the variable instead of the input object, which feels more natural to read for me. It then creates an array of one key objects and adds them together.
jq '[range(.key | length) as $i | {(.key[$i]): .string[$i]}] | add'
When I first looked at this issue, I though that a zip builtin would improve the situation. Then I remembered: there is already a zip builtin! It's just called transpose. Using it, you can create a script such as this:
jq '[.key, .string] | transpose | map({key: .[0], value: .[1]}) | from_entries'
It seems easier to follow to me as well, although it is quite long too; I assume, however, that the focus is readability and not character count. Of course, you can also mix up both solutions:
jq '[.key, .string] | transpose | map({(.[0]): .[1]}) | add'
Here is a solution which uses reduce with a state object holding an iteration index and a result object. It iterates over .key setting corresponding values in the result from .string
.string as $v
| reduce .key[] as $k (
{idx:0, result:{}}; .result[$k] = $v[.idx] | .idx += 1
)
| .result
I have json data that looks like:
{"foo":"one", "bar":2, "baz":[0]}
{"foo":"two", "bar":3, "baz":[1]}
{"foo":"one", "bar":3, "baz":[2,3]}
{"foo":"one", "bar":2, "baz":[2,4]}
I want to group everything with the same "foo" and collect the unique values of bar, and then collect the unique values inside of the "baz" arrays:
[
{"foo":"one", "bar":[2, 3], "baz":[0,2,3,4]},
{"foo":"two", "bar":[3], "baz":[1]}
]
(I don't care if the result is in an array or just a raw sequence of whitespace separated JSON objects, and I don't care about the order of the items in the "baz" array)
I've got jq version 1.4 installed from source. I can properly group by "foo" and collect unique values of "bar" with:
jq -s 'group_by(.foo) | map({foo: .[0].foo, bar: map(.bar) | unique})'
yielding:
[
{"foo":"one","bar":[2,3]},
{"foo":"two","bar":[3]}
]
but I can't figure out how to do the collection of unique values of "baz".
What am I missing?
Edit: new "flatten" function not needed (thanks
#JeffMercado)
I can run
jq -s 'group_by(.foo) | map({foo: .[0].foo, bar: map(.bar) | unique, baz: map(.baz) | add | unique})
which produces:
[
{"foo":"one","bar":[2,3],"baz":[0,2,3,4]},
{"foo":"two","bar":[3],"baz":[1]}
]