In code below I'm trying to match values from a bash array with values from json array using jq.
PROJECTS=$(curl -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "${GITLAB_URL}/api/v4/projects")
for GITLAB_TAG in "${GITLAB_TAGS[#]}"; do
PROJECTS=`echo "${PROJECTS}" \
| jq --arg gitlab_tag "$GITLAB_TAG" '[ .[] | select(.tag_list[] | contains($gitlab_tag)) ]'`
done
PROJECTS=$(echo "$PROJECTS" | jq -r '.[]')
Consider the following JSON payload as a sample API response:
[{"id":31,"description":"","default_branch":null,"tag_list":["dev","app"],"archived":false,"visibility":"private"},{"id":28,"description":"","default_branch":"master","tag_list":["dev","app"],"archived":false,"visibility":"private"}]
This works only partially, because I can't iterate over results ($PROJECTS).
printf "${PROJECTS[0]}" prints the whole array.
Am I missing something here?
It appears that you should be using index/1 to check .tag_list (not .tag_list[]):
[ .[] | select(.tag_list | index($gitlab_tag))
jq has no knowledge of bash array variables. For these, you therefore have two basic options: 1) use bash to iterate over the bash array variable (calling jq as often as necessary); 2) present the contents of the bash array variable in a way that jq can handle.
There are many variations of (2). Consider for example:
$ a=(a "b c")
$ printf "%s\n" "${a[#]}" | jq -R | jq -s
[
"a",
"b c"
]
Related
Assuming I have the following JSON object (which is just an example):
{
"foo": 1,
"bar": 2,
"baz": 3
}
And the following JSON array (another example):
["foo", "baz"]
How could I use jq to output the following object?
{
"foo": 1,
"baz": 3
}
I hate asking this question because I feel certain it has been answered before, but google has failed me and my jq-fu is just not where it needs to be to figure this out.
Using a reduce to iteratively build up the result object would be one way:
echo '["foo", "baz"]' | jq --argjson index '{"foo":1,"bar":2,"baz":3}' '
reduce .[] as $x ({}; .[$x] = $index[$x])
'
Using JOIN, creating key-value pairs, and employing from_entries for assembly would be another way:
echo '["baz", "foo"]' | jq --argjson index '{"foo":1,"bar":2,"baz":3}' '
JOIN($index; .) | map({key:.[0], value:.[1]}) | from_entries
'
Output:
{
"foo": 1,
"baz": 3
}
Provided that . is the object and $arr is the array, the following does the trick
delpaths(keys - $arr | map([.]))
To achieve the desired result, one could write:
jq '{foo, baz}'
This can (with some considerable trepidation) be made into a solution for the given problem by text wrangling, e.g. along the lines of:
jq "$(echo '["foo", "baz"]' | jq -Rr '"{" + .[1:-1] + "}" ')"
or
jq "$(echo '["foo", "baz"]' | sed -e 's/\[/{/' -e 's/\]/}/')"
Here's a reduce-free solution that assumes $keys is the array of keys of interest and that is possibly more efficient than the one involving array subtraction:
. as $in | INDEX( $keys[]; $in[.] )
I parsed a json file with jq like this :
# cat test.json | jq '.logs' | jq '.[]' | jq '._id' | jq -s
It returns an array like this : [34,235,436,546,.....]
Using bash script i described an array :
# declare -a msgIds = ...
This array uses () instead of [] so when I pass the array given above to this array it won't work.
([324,32,45..]) this causes problem. If i remove the jq -s, an array forms with only 1 member in it.
Is there a way to solve this issue?
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use #sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | #sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | #sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket().
It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | #sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that have spaces, newlines (or any other arbitrary characters) just use jq's #sh filter and bash's declare -a. (No need for a while read loop or any other pre-processing)
// foo.json
{"data": ["A B", "C'D", ""]}
str=$(jq -r '.data | #sh' foo.json)
declare -a arr="($str)" # must be quoted like this
$ declare -p arr
declare -a arr=([0]="A B" [1]="C'D" [2]="")
The reason that this works correctly is that #sh will produce a space-separated list of shell-quoted words:
$ echo "$str"
'A B' 'C'\''D' ''
and this is exactly the format that declare expects for an array definition.
Use jq -r to output a string "raw", without JSON formatting, and use the #sh formatter to format your results as a string for shell consumption. Per the jq docs:
#sh:
The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings.
So can do e.g.
msgids=($(<test.json jq -r '.logs[]._id | #sh'))
and get the result you want.
From the jq FAQ (https://github.com/stedolan/jq/wiki/FAQ):
𝑸: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
A: One option would be to use mapfile (aka readarray), for example:
mapfile -t array <<< $(jq -c '.[]' input.json)
An alternative that might be indicative of what to do in other shells is to use read -r within a while loop. The following bash script populates an array, x, with JSON texts. The key points are the use of the -c option, and the use of the bash idiom while read -r value; do ... done < <(jq .......):
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
++ To resolve this, we can use a very simple approach:
++ Since I am not aware of you input file, I am creating a file input.json with the following contents:
input.json:
{
"keys": ["key1","key2","key3"]
}
++ Use jq to get the value from the above file input.json:
Command: cat input.json | jq -r '.keys | #sh'
Output: 'key1' 'key2' 'key3'
Explanation: | #sh removes [ and "
++ To remove ' ' as well we use tr
command: cat input.json | jq -r '.keys | #sh' | tr -d \'
Explanation: use tr delete -d to remove '
++ To store this in a bash array we use () with `` and print it:
command:
KEYS=(`cat input.json | jq -r '.keys | #sh' | tr -d \'`)
To print all the entries of the array: echo "${KEYS[*]}"
I'm trying to use jq to combine two arrays and running into a bit of trouble.
I'm attempting to parse out the data from netdata (netdata.firehol.org) and the two pieces of data within the json response that I"m interested in are both part of an array. The first array is labels for the datapoints in the second array.
Sample Input
[
"time",
"guest_nice",
"guest",
"steal",
"softirq",
"irq",
"user",
"system",
"nice",
"iowait"
]
[
1460728600,
0,
0,
0,
0.45731,
0,
0.25108,
11.74702,
48.22465,
0
]
Input
If you want to grab fresh data yourself to test against, you can use the following:
curl -s -X GET --header 'Accept: application/json'
'http://netdata.firehol.org/api/v1/data?chart=system.cpu&after=-10&before=0&points=1&group=average&format=json&options=seconds%2Cjsonwrap' | jq '.result.labels, .result.data[]'
I've tried to use map() as well as trying to assign vars to both arrays and then print out the objects together, but have been unsuccessful (below).
Code
| jq '.result.labels as $labels | .result.data[] as $data | .result.data[] | Label: $labels[.], data: $data[.]}'
I appreciate anyone's insight in advance as I'm a little stuck, and would prefer to be able to do this all in jq rather than using for loops in bash (if possible).
Expected Ouput
{
"time": "1460728600",
"guest_nice": "0",
...
}
You haven't specified exactly how you want the arrays to be combined, but one approach is to use transpose, which in this case is effectively a kind of zip. For example:
$ jq -n -c '[["a","b"], [1,2]] | transpose'
yields: [["a",1],["b",2]]
If you wanted an array of objects, then with the same input,
transpose | map( { (.[0]) : .[1] } )
would yield: [{"a":1},{"b":2}]
If your jq does not have transpose, here is its definition:
# transpose a possibly jagged matrix, quickly;
# rows are padded with nulls so the result is always rectangular.
def transpose:
[range(0; (map(length) | max)) as $j
| [range(0; length) as $i | .[$i][$j] ] ] ;
Alternatively, if you would prefer a very brief zip:
def zip: [range(0; .[0]|length) as $i | [.[0][$i], .[1][$i]]];
Here is a solution that handles the general case where the first array contains the key names and the following arrays contain values using transpose and from_entries
{h:.[0], v:.[1:][]} # {h:[keys], v:[values]}
| [.h, .v] # [ [keys], [values] ] ...
| [ transpose[] | {key:.[0], value:.[1]} ] # [ {"key":key, "value":value}, ... ]
| from_entries # { key:value, key:value, ... }
For example, if this filter is in filter.jq and data.json contains
["time","guest_nice","guest","steal","softirq","irq","user","system","nice","iowait"]
[1460728600,0,0,0,0.45731,0,0.25108,11.74702,48.22465,0]
[1460728601,0,0,0,0.45732,0,0.25109,12.74703,49,0]
then the command
jq -M -s -c -f filter.jq data.json
produces
{"time":1460728600,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45731,"irq":0,"user":0.25108,"system":11.74702,"nice":48.22465,"iowait":0}
{"time":1460728601,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45732,"irq":0,"user":0.25109,"system":12.74703,"nice":49,"iowait":0}
i have a json file
{
"file" : {
"a" : 1,
"b" : 2
}
}
I am using jq to count number of keys file value have in this json object.
then using on bash
arr=($(cat jsonfile.json | jq '.file' | jq -r 'keys'))
echo ${#arr[#]}
here i get output 4 whereas there is only 2 keys a,b
Why is that so, and how do i get arr only have two elements a and b.?
To understand why you get an array of 4 elements, look at the output of the sub-shell:
cat jsonfile.json | jq '.file' | jq -r 'keys'
This produces:
[
"a",
"b"
]
Each line there becomes an element of the array -> 4 lines.
Try this instead:
jq -r '.file | keys | .[]' jsonfile.json
Output:
a
b
I also simplified your original expression. (Thanks #JeffMercado!)
I have a Bash script which gets data in JSON, I want to be able to convert the JSON into an accessible structure - array / list / or other model which would be easy to parse the nested data.
Example:
{
"SALUTATION": "Hello world",
"SOMETHING": "bla bla bla Mr. Freeman"
}
I want to get the value like the following: echo ${arr[SOMETHING]}
[ Different approach is optional as well. ]
If you want key and value, and based on How do i convert a json object to key=value format in JQ, you can do:
$ jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" file
SALUTATION=Hello world
SOMETHING=bla bla bla Mr. Freeman
In a more general way, you can store the values into an array myarray[key] = value like this, just by providing jq to the while with the while ... do; ... done < <(command) syntax:
declare -A myarray
while IFS="=" read -r key value
do
myarray[$key]="$value"
done < <(jq -r 'to_entries|map("(.key)=(.value)")|.[]' file)
And then you can loop through the values like this:
for key in "${!myarray[#]}"
do
echo "$key = ${myarray[$key]}"
done
For this given input, it returns:
SALUTATION = Hello world
SOMETHING = bla bla bla Mr. Freeman
Although this question is answered, I wasn't able to fully satiate my
requirements from the posted answer. Here is a little write up that'll help any
bash-newcomers.
Foreknowledge
A basic associative array declaration
#!/bin/bash
declare -A associativeArray=([key1]=val1 [key2]=val2)
You can also use quotes (', ") around the declaration, its keys, and
values.
#!/bin/bash
declare -A 'associativeArray=([key1]=val1 [key2]=val2)'
And you can delimit each [key]=value pair via space or newline.
#!/bin/bash
declare -A associativeArray([key1]=value1
['key2']=value2 [key3]='value3'
['key4']='value2' ["key5"]="value3"
["key6"]='value4'
['key7']="value5"
)
Depending on your quote variation, you may need to escape your string.
Using Indirection to access both key and value in an associative array
example () {
local -A associativeArray=([key1]=val1 [key2]=val2)
# print associative array
local key value
for key in "${!associativeArray[#]}"; do
value="${associativeArray["$key"]}"
printf '%s = %s' "$key" "$value"
done
}
Running the example function
$ example
key2 = val2
key1 = val1
Knowing the aforementioned tidbits allows you to derive the following snippets:
The following examples will all have the result as the example above
String evaluation
#!/usr/bin/env bash
example () {
local arrayAsString='associativeArray=([key1]=val1 [key2]=val2)'
local -A "$arrayAsString"
# print associative array
}
Piping your JSON into JQ
#!/usr/bin/env bash
# Note: usage of single quotes instead of double quotes for the jq
# filter. The former is preferred to avoid issues with shell
# substitution of quoted strings.
example () {
# Given the following JSON
local json='{ "key1": "val1", "key2": "val2" }'
# filter using `map` && `reduce`
local filter='to_entries | map("[\(.key)]=\(.value)") |
reduce .[] as $item ("associativeArray=("; . + ($item|#sh) + " ") + ")"'
# Declare and assign separately to avoid masking return values.
local arrayAsString;
# Note: no encompassing quotation (")
arrayAsString=$(jq --raw-output "${filter}" <<< "$json")
local -A "$arrayAsString"
# print associative array
}
jq -n / --null-input option + --argfile && redirection
#!/usr/bin/env bash
example () {
# /path/to/file.json contains the same json as the first two examples
local filter filename='/path/to/file.json'
# including bash variable name in reduction
filter='to_entries | map("[\(.key | #sh)]=\(.value | #sh) ")
| "associativeArray=(" + add + ")"'
# using --argfile && --null-input
local -A "$(jq --raw-output --null-input --argfile file "$filename" \
"\$filename | ${filter}")"
# or for a more traceable declaration (using shellcheck or other) this
# variation moves the variable name outside of the string
# map definition && reduce replacement
filter='[to_entries[]|"["+(.key|#sh)+"]="+(.value|#sh)]|"("+join(" ")+")"'
# input redirection && --join-output
local -A associativeArray=$(jq --join-output "${filter}" < "${filename}")
# print associative array
}
Reviewing previous answers
#Ján Lalinský
To load JSON object into a bash associative array efficiently
(without using loops in bash), one can use tool 'jq', as follows.
# first, load the json text into a variable:
json='{"SALUTATION": "Hello world", "SOMETHING": "bla bla bla Mr. Freeman"}'
# then, prepare associative array, I use 'aa':
unset aa
declare -A aa
# use jq to produce text defining name:value pairs in the bash format
# using #sh to properly escape the values
aacontent=$(jq -r '. | to_entries | .[] | "[\"" + .key + "\"]=" + (.value | #sh)' <<< "$json")
# string containing whole definition of aa in bash
aadef="aa=($aacontent)"
# load the definition (because values may contain LF characters, aadef must be in double quotes)
eval "$aadef"
# now we can access the values like this: echo "${aa[SOMETHING]}"
Warning: this uses eval, which is dangerous if the json input is from unknown source (may contain malicious shell commands that eval may execute).
This could be reduced to the following
example () {
local json='{ "key1": "val1", "key2": "val2" }'
local -A associativeArray="($(jq -r '. | to_entries | .[] |
"[\"" + .key + "\"]=" + (.value | #sh)' <<< "$json"))"
# print associative array
}
#fedorqui
If you want key and value, and based on How do i convert a json object to key=value format in JQ, you can do:
$ jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" file
SALUTATION=Hello world
SOMETHING=bla bla bla Mr. Freeman
In a more general way, you can store the values into an array myarray[key] = value like this, just by providing jq to the while with the while ... do; ... done < <(command) syntax:
declare -A myarray
while IFS="=" read -r key value
do
myarray[$key]="$value"
done < <(jq -r "to_entries|map(\"\(.key)=\(.value)\")|.[]" file)
And then you can loop through the values like this:
for key in "${!myarray[#]}"
do
echo "$key = ${myarray[$key]}"
done
For this given input, it returns:
SALUTATION = Hello world
SOMETHING = bla bla bla Mr. Freeman
The main difference between this solution and my own is looping through the
array in bash or in jq.
Each solution is valid and depending on your use case, one may be more useful
then the other.
Context: This answer was written to be responsive to a question title which no longer exists..
The OP's question actually describes objects, vs arrays.
To be sure that we help other people coming in who are actually looking for help with JSON arrays, though, it's worth covering them explicitly.
For the safe-ish case where strings can't contain newlines (and when bash 4.0 or newer is in use), this works:
str='["Hello world", "bla bla bla Mr. Freeman"]'
readarray -t array <<<"$(jq -r '.[]' <<<"$str")"
To support older versions of bash, and strings with newlines, we get a bit fancier, using a NUL-delimited stream to read from jq:
str='["Hello world", "bla bla bla Mr. Freeman", "this is\ntwo lines"]'
array=( )
while IFS= read -r -d '' line; do
array+=( "$line" )
done < <(jq -j '.[] | (. + "\u0000")')
This is how can it be done recursively:
#!/bin/bash
SOURCE="$PWD"
SETTINGS_FILE="$SOURCE/settings.json"
SETTINGS_JSON=`cat "$SETTINGS_FILE"`
declare -A SETTINGS
function get_settings() {
local PARAMS="$#"
local JSON=`jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" <<< "$1"`
local KEYS=''
if [ $# -gt 1 ]; then
KEYS="$2"
fi
while read -r PAIR; do
local KEY=''
if [ -z "$PAIR" ]; then
break
fi
IFS== read PAIR_KEY PAIR_VALUE <<< "$PAIR"
if [ -z "$KEYS" ]; then
KEY="$PAIR_KEY"
else
KEY="$KEYS:$PAIR_KEY"
fi
if jq -e . >/dev/null 2>&1 <<< "$PAIR_VALUE"; then
get_settings "$PAIR_VALUE" "$KEY"
else
SETTINGS["$KEY"]="$PAIR_VALUE"
fi
done <<< "$JSON"
}
To call it:
get_settings "$SETTINGS_JSON"
The array will be accessed like so:
${SETTINGS[grandparent:parent:child]}
To load JSON object into a bash associative array efficiently (without using loops in bash), one can use tool 'jq', as follows.
# first, load the json text into a variable:
json='{"SALUTATION": "Hello world", "SOMETHING": "bla bla bla Mr. Freeman"}'
# then, prepare associative array, I use 'aa':
unset aa
declare -A aa
# use jq to produce text defining name:value pairs in the bash format
# using #sh to properly escape the values
aacontent=$(jq -r '. | to_entries | .[] | "[\"" + .key + "\"]=" + (.value | #sh)' <<< "$json")
# string containing whole definition of aa in bash
aadef="aa=($aacontent)"
# load the definition (because values may contain LF characters, aadef must be in double quotes)
eval "$aadef"
# now we can access the values like this: echo "${aa[SOMETHING]}"
Warning: this uses eval, which is dangerous if the json input is from unknown source (may contain malicious shell commands that eval may execute).
Building on #HelpNeeder's solution (nice one btw)
His solution wasn't really working with integers, so i made some additions. Extended amount of condition checks, so it's fair to say some performance is sacrificed.
This version works with integers and also floating point values.
SOURCE="$PWD"
SETTINGS_FILE="./test2.json"
SETTINGS_JSON=`cat "$SETTINGS_FILE"`
declare -A SETTINGS
get_settings() {
local PARAMS="$#"
local JSON=`jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" <<< "$1"`
local KEYS=''
if [ $# -gt 1 ]; then
KEYS="$2"
fi
while read -r PAIR; do
local KEY=''
if [ -z "$PAIR" ]; then
break
fi
IFS== read PAIR_KEY PAIR_VALUE <<< "$PAIR"
if [ -z "$KEYS" ]; then
KEY="$PAIR_KEY"
else
KEY="$KEYS:$PAIR_KEY"
fi
res=$(jq -e . 2>/dev/null <<< "$PAIR_VALUE")
exitCode=$?
check=`echo "$PAIR_VALUE" | grep -E ^\-?[0-9]*\.?[0-9]+$`
# if [ "${res}" ] && [ $exitCode -eq "0" ] && [[ ! "${PAIR_VALUE}" == ?(-)+([0-9]) ]] ALTERNATIVE, works only for integer (not floating point)
if [ "${res}" ] && [ $exitCode -eq "0" ] && [[ "$check" == '' ]]
then
get_settings "$PAIR_VALUE" "$KEY"
else
SETTINGS["$KEY"]="$PAIR_VALUE"
fi
done <<< "$JSON"
}
get_settings "$SETTINGS_JSON"
Solution: use jq( it's a lightweight and flexible command-line JSON processor.).
In bash I'd rather assign JSONs object to a variable and use jq in order to access and parse the right result from it. It's more convenient than parse this structure with arrays and it comes out of the box with multiple functionalities and features such as accessing nested and complex objects, select methods, builtin operators and functions, regex support ,comparisons etc...
example:
example='{"SALUTATION": "Hello world","SOMETHING": "bla bla bla Mr. Freeman"}'
echo $example | jq .SOMETHING
# output:
"bla bla bla Mr. Freeman"