Given a bash array, how to convert it to a JSON array in order to output to a file with jq?
Additionnally: is there a way to keep the server_nohup array unchanged instead of re-writing the whole json file each time?
newArray=(100 200 300)
jq -n --arg newArray $newArray '{
client_nohup: [
$newArray
],
server_nohup: [
]
}' > $projectDir/.watch.json
Current output:
{
"client_nohup": [
"100"
],
"server_nohup": []
}
Desired output:
{
"client_nohup": [
100,
200,
300
],
"server_nohup": []
}
(1) If all the values in newArray are valid as JSON values without spaces, then you could get away with piping the values as a stream, e.g.
newArray=(100 200 300)
echo "${newArray[#]}" |
jq -s '{client_nohup: ., server_nohup: []}'
(2)
Now let's suppose you merely wish to update the "nohup" object in a file, say nohup.json:
{ "client_nohup": [], "server_nohup": [ "keep me" ] }
Since you are using bash, you can then write:
echo "${newArray[#]}" |
jq -s --argjson nohup "$(cat nohup.json)" '
. as $newArray | $nohup | .client_nohup = $newArray
'
Output
(1)
{
"client_nohup": [
100,
200,
300
],
"server_nohup": []
}
(2)
{
"client_nohup": [
100,
200,
300
],
"server_nohup": [
"keep me"
]
}
Other cases
Where there's a will, there's a jq way :-)
See for example the accepted answer at How to format a bash array as a JSON array (though this is not a completely generic solution).
For a generic solution, see 𝑸: How can a variable number of arguments be passed to jq? How can a bash array of values be passed in to jq as a single argument? at the jq FAQ https://github.com/stedolan/jq/wiki/FAQ
Generic Solutions
To be clear, if the array values are known to be valid JSON, there are several good options; if the array values are arbitrary bash strings, then the only efficient, generic way to handle them with jq is by using the -R jq option (e.g. in conjunction with -s), but then the (bash) strings will all be read in as JSON strings, so any intended type information will be lost. (The point here hinges on the technicality that bash strings cannot CONTAIN NUL characters.)
Often, to alleviate the latter concern, one can convert numeric strings to JSON numbers, e.g. using the jq idiom: (tonumber? // .).
In general, the only truly safe way to do this is with multiple invocations of jq, adding each element to the output of the previous command.
arr='[]' # Empty JSON array
for x in "${newArray[#]}"; do
arr=$(jq -n --arg x "$x" --argjson arr "$arr" '$arr + [$x]')
done
This ensures that each element x of your bash array is properly encoded prior to adding it to the JSON array.
This is complicated, though, by the fact that bash doesn't not distinguish between numbers and strings. This encodes your array as ["100", "200", "300"], not [100, 200, 300]. In the end, you need to have some awareness of what your array contains, and preprocess it accordingly.
Related
This question already has answers here:
bash & jq: add attribute with object value
(2 answers)
Passing variable in jq to filter with select fails
(1 answer)
Closed 5 days ago.
I'm simply trying to replace an objects value in a json file with an array, using jq in a bash script.
The json file (truncated) looks like this:
{
"objects": {
"type": "foo",
"host": "1.1.1.1",
"port": "1234"
}
}
I want to replace the host objects value with an array of different values, so it looks like this:
{
"objects": {
"type": "foo",
"host": ["1.1.1.1","2.2.2.2"],
"port": "1234"
}
}
I tested around with this script. The Input comes from a simple, comma separated string which I convert into a proper json array (which seems to work).
But I'm not able replace the value with the array.
#!/bin/bash
objectshost="1.1.1.1,2.2.2.2"
objectshost_array=$(jq -c -n --arg arg $objectshost '$arg|split(",")')
jq --arg value "$objectshost_array" '.objects.host = $value' ./test.json > ./test.json.tmp
The best I ended up with, is this:
{
"objects": {
"type": "foo",
"host": "[\"1.1.1.1\",\"2.2.2.2\"]",
"port": "1234"
}
}
The result seems to be some logical result, as the script simply replaces the value with the arrays string. But it's not what I expected to get. ;)
I found some similar questions, but all of them were dealing with replacing values in existing arrays or key/value pairs, but my problem seems to be the conversion from a single value to an array.
Can somebody please push me into the right direction? Or should I forget about jq and threat the json file as a simple text file?
Thanks in advance,
André
It would work with a conditional assignment from arguments:
jq '
.objects.host = (
.objects.host |
if type == "array"
then .
else [ . ]
end + $ARGS.positional
)
' input.json --args 1.2.3.4 2.2.2.2 4.4.4.4
Or the same as a stand-alone jq script; which is more readable and maintainable:
myJQScript:
#!/usr/bin/env -S jq --from-file --args
.objects.host = (
.objects.host |
if type == "array"
then .
else [ . ]
end + $ARGS.positional
)
Make it executable:
chmod +x myJQScript
Run it with arguments to add array entries to host
$ ./myJQScript 1.2.3.4 2.2.2.2 4.4.4.4 < input.json
{
"objects": {
"type": "foo",
"host": [
"1.1.1.1",
"1.2.3.4",
"2.2.2.2",
"4.4.4.4"
],
"port": "1234"
}
}
You can do it with a single jq command:
#!/bin/sh
objectshost="1.1.1.1,2.2.2.2"
jq --arg value "$objectshost" '.objects.host = ($value / ",")' ./test.json > ./test.json.tmp
This has the added benefit of not requiring Bash arrays, but can be used with any shell.
If you already have a JSON array, you must use --argjson and not --arg. --arg always creates a variable of type string, --argjson however parses the value as JSON entity.
#!/bin/bash
objectshost="1.1.1.1,2.2.2.2"
objectshost_array=$(printf '%s\n' "$objectshost" | jq -c 'split(",")')
jq --argjson value "$objectshost_array" '.objects.host = $value' ./test.json > ./test.json.tmp
I have a json file named collection.jsonsuch as :
{
"info" : {
...
},
"item" : [
{...}, # A
{...}, # B
{...} # C
]
}
I want all the fields from the array item, like below:
{...}, # A
{...}, # B
{...} # C
What I have tried:
jq -r '.item' collection.json
With this, I still avec the squares brackets, at the beginning and the end.
jq -r '.item[]' collection.json
With this, the comma between the fields is removed.
Since your expected output is neither valid JSON (which would be your first suggestion jq -r '.item' collection.json that includes commas AND the array brackets) nor the plain raw content of the elements (which would be your second suggestion jq -r '.item[]' collection.json that removes all the surrounding JSON syntax), you will have to build the desired syntax yourself which may depend on what the .item array elements actually are.
For instance, convert them into strings using tostring and glue them together with the join builtin:
jq -r '.item | map(tostring) | join(",\n")' collection.json
First - a very simple code:
#!/bin/bash
a_string="string"
readarray -t a <<<"$a_string"
echo "${a[#]}"
echo "${#a[#]}"
# read empty string into array
emptystring=""
readarray -t b <<<"$emptystring"
echo "${b[#]}"
echo "${#b[#]}"
# and now - empty array
c=()
echo "${c[#]}"
echo "${#c[#]}"
And the output
string
1
1
0
So reading an empty string into bash array with readarray -t shows array length as 1. And only truly empty array's length is 0.
My question - why it is happening?
As my case - here is a fragment from my script - execute an API call, get JSON, and apply jq filter:
URL_list_json=$(curl -s --request GET "$curl_request" --header "Authorization: Bearer ${token}")
readarray -t URL_names <<<"$(echo "$URL_list_json" | jq -r '.[].Body.monitors[].name')"
Here is an example of JSON which results an empty array from the jq call:
[
{
"Header": {
"Region": "dc1",
"Tenant": "tenant1",
"Stage": "test"
},
"Body": {
"monitors": []
}
}
]
and here is JSON with populated content returning a non-empty array from jq call:
[
{
"Header": {
"Region": "dc2",
"Tenant": "tenant2",
"Stage": "qa"
},
"Body": {
"monitors": [
{
"enabled": true,
"entityId": "TEST-674BA97E74FC74AA",
"name": "production.example.com"
},
{
"enabled": false,
"entityId": "TEST-3E2D23438A973D1E",
"name": "test.example.com"
}
]
}
}
]
I need to check if URL_names array is empty or not. If not empty, then iterate over contents. If empty - means that jq did not return any results - white to log and move on.
If I use if ((${#URL_names[#]})) as a means to check if the array is empty or not, it will return 1 even if the array has just an empty string from the jq call, so this logic fails.
What are alternative approaches to deal with the case like above?
I can assign the output of the jq filter to a string and then use if statement to check if the string is empty and if non-empty then assign the string to an array, but this introduces additional elements. With this approach, I can skip using arrays entirely - I was hoping to use only arrays for this task.
Thanks
why it is happening?
Because it reads one line. From bash manual here document:
[n]<<< word
[...] The result is supplied as a single string, with a newline appended, to the command on its standard input (or file descriptor n if n is specified).
Because there is a newline, readarray reads one empty line. You may do:
readarray -t b < <(printf "%s" "$emptystring")
Notes:
echo "$var" is not preferred. Do printf "%s" "$var" when in posix shell, do <<<"$var" when in bash (and you do not care about the extra newline).
<<<"$(...)" always looks strange - the <<< has to allocate a subshell anyway. Do < <(...).
You want to do: readarray -t URL_names < <(<<<"$URL_list_json" jq -r '.[].Body.monitors[].name')
If you want to check if the array is empty, check that in jq. I see ex. jq --exit-status '.[].Body.monitors[].name' and just check the exit status.
the file is
{
"ContentKey--4-0-47--Vovb1BQ": ["infra", "qa", "qa-ContentKey-4-0-47-Vovb1BQ", "internal-qa-Conten-WebServi-19E4PUWHRGD44-460820639.us-east-1.elb.amazonaws.com", "plan--default"],
"ContentKey--4-0-47--zjOkiQ": ["svc", "dev", "dev-ContentKey-4-0-47-zjOkiQ", "dev-Conte-WebServi-KXJXZBDY113W-2116785917.us-east-1.elb.amazonaws.com", "plan--default"],
"IdGenService--2001-4-22--CJUFaMQ": ["svc", "dev", "dev-IdGenService-2001-4-22-CJUFaMQ", "dev-IdGen-WebServi-R7RVXSYAV92W-304073075.us-east-1.elb.amazonaws.com"],
"IdGenService--2001-4-22--Uhf9CTQ": ["svc", "qa", "qa-IdGenService-2001-4-22-Uhf9CTQ", "internal-qa-IdGenS-WebServi-RT5BI5EEVZP3-665537643.us-east-1.elb.amazonaws.com"]
}
I want to find the list of keys whose array value have the entry svc
i could get the following to work
cat list.json | jq '. | map(select (. | contains(["svc"])))'
But the output is the value array and not the key itself
[
[
"svc",
"dev",
"dev-ContentKey-4-0-47-zjOkiQ",
"dev-Conte-WebServi-KXJXZBDY113W-2116785917.us-east-1.elb.amazonaws.com",
"plan--default"
],
[
"svc",
"dev",
"dev-IdGenService-2001-4-22-CJUFaMQ",
"dev-IdGen-WebServi-R7RVXSYAV92W-304073075.us-east-1.elb.amazonaws.com"
],
[
"svc",
"qa",
"qa-IdGenService-2001-4-22-Uhf9CTQ",
"internal-qa-IdGenS-WebServi-RT5BI5EEVZP3-665537643.us-east-1.elb.amazonaws.com"
]
]
With your input, the following filter yields the output as shown:
to_entries[] | select( .value | index("svc") ) | .key
Output:
"ContentKey--4-0-47--zjOkiQ"
"IdGenService--2001-4-22--CJUFaMQ"
"IdGenService--2001-4-22--Uhf9CTQ"
In cases like this, using index/1 is both simpler and (potentially much) faster than using any/2.
The top-level object in your json is an object, not an array. So .[] would only yield its values and discard the keys. Use with_entries/1 to filter that object. This converts an object to an array of key/value pairs and back with which you can apply filters to.
$ jq --arg key 'svc' 'with_entries(select(any(.value[]; . == $key)))' list.json
Also, you should avoid using contains/1 here. It's applied recursively so it will also match strings that contain the substring svc. i.e., "Foosvcbar" will be matched.
I'm trying to use jq to combine two arrays and running into a bit of trouble.
I'm attempting to parse out the data from netdata (netdata.firehol.org) and the two pieces of data within the json response that I"m interested in are both part of an array. The first array is labels for the datapoints in the second array.
Sample Input
[
"time",
"guest_nice",
"guest",
"steal",
"softirq",
"irq",
"user",
"system",
"nice",
"iowait"
]
[
1460728600,
0,
0,
0,
0.45731,
0,
0.25108,
11.74702,
48.22465,
0
]
Input
If you want to grab fresh data yourself to test against, you can use the following:
curl -s -X GET --header 'Accept: application/json'
'http://netdata.firehol.org/api/v1/data?chart=system.cpu&after=-10&before=0&points=1&group=average&format=json&options=seconds%2Cjsonwrap' | jq '.result.labels, .result.data[]'
I've tried to use map() as well as trying to assign vars to both arrays and then print out the objects together, but have been unsuccessful (below).
Code
| jq '.result.labels as $labels | .result.data[] as $data | .result.data[] | Label: $labels[.], data: $data[.]}'
I appreciate anyone's insight in advance as I'm a little stuck, and would prefer to be able to do this all in jq rather than using for loops in bash (if possible).
Expected Ouput
{
"time": "1460728600",
"guest_nice": "0",
...
}
You haven't specified exactly how you want the arrays to be combined, but one approach is to use transpose, which in this case is effectively a kind of zip. For example:
$ jq -n -c '[["a","b"], [1,2]] | transpose'
yields: [["a",1],["b",2]]
If you wanted an array of objects, then with the same input,
transpose | map( { (.[0]) : .[1] } )
would yield: [{"a":1},{"b":2}]
If your jq does not have transpose, here is its definition:
# transpose a possibly jagged matrix, quickly;
# rows are padded with nulls so the result is always rectangular.
def transpose:
[range(0; (map(length) | max)) as $j
| [range(0; length) as $i | .[$i][$j] ] ] ;
Alternatively, if you would prefer a very brief zip:
def zip: [range(0; .[0]|length) as $i | [.[0][$i], .[1][$i]]];
Here is a solution that handles the general case where the first array contains the key names and the following arrays contain values using transpose and from_entries
{h:.[0], v:.[1:][]} # {h:[keys], v:[values]}
| [.h, .v] # [ [keys], [values] ] ...
| [ transpose[] | {key:.[0], value:.[1]} ] # [ {"key":key, "value":value}, ... ]
| from_entries # { key:value, key:value, ... }
For example, if this filter is in filter.jq and data.json contains
["time","guest_nice","guest","steal","softirq","irq","user","system","nice","iowait"]
[1460728600,0,0,0,0.45731,0,0.25108,11.74702,48.22465,0]
[1460728601,0,0,0,0.45732,0,0.25109,12.74703,49,0]
then the command
jq -M -s -c -f filter.jq data.json
produces
{"time":1460728600,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45731,"irq":0,"user":0.25108,"system":11.74702,"nice":48.22465,"iowait":0}
{"time":1460728601,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45732,"irq":0,"user":0.25109,"system":12.74703,"nice":49,"iowait":0}