I have script
#!/bin/bash
ARR=("a" "b")
collection_init_msg=$( jq -n --arg arr $ARR '{arr: [$arr]}')
echo some command "$collection_init_msg"
that should convert the ARR and print it as a JSON array.
Current result
some command {
"arr": [
"a"
]
}
What I want is:
some command {
"arr": [
"a", "b"
]
}
#!/bin/bash
ARR=("a" "b")
collection_init_msg=$( jq -nc '{arr: $ARGS.positional}' --args "${ARR[#]}" )
echo "$collection_init_msg"
In answer to the supplementary question in the comment: one could do worse than:
jq -n --argjson a1 $(jq -nc '$ARGS.positional' --args "${A1[#]}") '
{$a1, a2: $ARGS.positional}' --args "${A2[#]}"
Related
First a working example with arrays
json_array() {
local -n array="${1}"
readarray -d $'\0' -t array < <(
# Create nul delimited array entry using jq
jq -cjn --argjson arr "$array" '$arr|map(tostring)|.[]+"\u0000"'
)
}
> unset arr; arr='["a", "b", "c"]'; json_array arr; echo "${arr[0]} ${arr[1]} ${arr[2]}"
a b c
Now I'm trying to do something similar with dict, convert a json dict into a bash associative array
json_dict() {
local -n dict="${1}"
declare -A hash_table
append_to_hash_table() {
shift
{ read -r key; read -r value; } <<<"$1"
hash_table+=([$key]="$value")
}
readarray -d $'\0' -c 1 -C append_to_hash_table < <(
# Create nul delimited dict entry using jq
jq -cjn --argjson d "$dict" '$d|to_entries|map("\(.key)\n\(.value|tostring|#sh)")|.[]+"\u0000"'
)
# Here hash_table contain the correct output
dict=""
dict="$hash_table"
}
> unset arr; arr='{"a": "aa", "l": "bb", "c": "ccccc"}'; json_dict arr; echo "${arr[#]}"
Nothing
It seems dict="$hash_table" doesn't correctly update the refname,
How can I make bash dict refname point to hash_table?
There's no need for readarray here: You can have two separate NUL-delimited reads as part of your while loop.
See the below answer demonstrated at https://replit.com/#CharlesDuffy2/GrandioseDraftyArguments#main.sh
while IFS= read -r -d '' key && IFS= read -r -d '' value; do
hash_table[$key]=$value
done < <(jq -cjn --argjson d "$arr" \
'$d | to_entries[] | ( .key, "\u0000", .value, "\u0000")')
Putting this into context:
json_dict() {
declare key value in_value="${!1}"
unset "$1" # FIXME: Better to take a $2 for output variable
declare -g -A "$1"
declare -n hash_table="$1"
while IFS= read -r -d '' key && IFS= read -r -d '' value; do
hash_table[$key]=$value
done < <(
jq -cjn --argjson d "$in_value" \
'$d | to_entries[] | ( .key, "\u0000", .value, "\u0000")'
)
}
arr='{"a": "aa", "l": "bb", "c": "ccccc"}'
json_dict arr
declare -p arr
...emits as output:
declare -A arr=([a]="aa" [c]="ccccc" [l]="bb" )
That said, to answer the question exactly as-asked, thus using readarray:
json_dict() {
declare -a pieces=()
readarray -d '' pieces < <(
jq -cjn --argjson d "${!1}" \
'$d | to_entries[] | ( .key, "\u0000", .value, "\u0000")'
)
unset "$1"
declare -g -A "$1"
declare -n hash_table="$1"
set -- "${pieces[#]}"
while (( $# )); do
hash_table[$1]=$2
{ shift && shift; } || return
done
}
arr='{"a": "aa", "l": "bb", "c": "ccccc"}'
json_dict arr
declare -p arr
Wouldn't it be simpler to just use declare and have jq create the contents using #sh for shell conformity?
Indexed array:
unset arr; arr='["a", "b", "c"]'
declare -a arr="($(jq -r #sh <<< "$arr"))"
Associative array:
unset arr; arr='{"a": "aa", "l": "bb", "c": "ccccc"}'
declare -A arr="($(jq -r 'to_entries[] | #sh "[\(.key)]=\(.value)"' <<< "$arr"))"
Reacting to an edit request: The above requires the values to be strings (numbers and booleans will kind of work, too), but other structures need to be brought into a stringy format first. In particular, following the request, #json could be used to encode an arbitrary JSON content as a string. However, keep in mind that in doing so, the bash array's items will be JSON-encoded, which means that also the simple cases (strings like aa from above) will be encoded (e.g. as "aa", including the quotes, as required by JSON). If this is what you want, go for it:
Indexed array with JSON-encoded values:
unset arr; arr='["a", {}, [null, {"a": true}]]'
declare -a arr="($(jq -r 'map(#json) | #sh' <<< "$arr"))"
Associative array with JSON-encoded values:
unset arr; arr='{"a": "aa", "b": {"l": "bb", "c": "ccccc"}}'
declare -A arr="($(jq -r 'to_entries[] | #sh "[\(.key)]=\(.value | #json)"' <<< "$arr"))"
I actually have 2 arrays in bash that contains string values.
Something like that :
Array1=(Kevin Paul)
Array2=(OK DANGER)
I would like to create a json with 2 attributes, something like that if possible
{
"results":[
{
"nom":"Kevin",
"status":"OK"
},
{
"nom":"Paul",
"status":"Danger"
}
]
}
I read a lot speaking about JQ that i alreay use for my arrays, but no one speak about something like i want :(
One of my test ( that does not respond to what i would like ) :
declare -a test_array
declare -a test_array2
test_array=(apple orange lemon)
test_array2=(DANGER OK WARNING)
echo ${test_array[0]}
echo '['
printf '{"CVEC": "%s", "LVL" : "%s"},\n' "${test_array[#]}, ${test_array2[#]}" | sed '$s/,$//'
echo ']'
Display
[
{"CVEC": "apple", "LVL" : "orange"},
{"CVEC": "lemon, DANGER", "LVL" : "OK"},
{"CVEC": "WARNING", "LVL" : ""}
]
Using a template engine: perl's Template::Toolkit command line tool: tpage:
Files
header:
{
"results":[
footer:
]
}
file.tpl (template):
{
"nom": "[% x1 %]",
"status": "[% x2 %]"
}[% sep %]
Bash script
#!/bin/bash
arr1=( Kevin Paul )
arr2=( OK danger )
{
cat header
for i in "${!arr1[#]}"; do
((i==${#arr1[#]}-1)) && sep='' || sep=','
tpage --define x1="${arr1[i]}" \
--define x2="${arr2[i]}" \
--define sep=$sep file.tpl
done
cat footer
} | tee file.json
Validation
$ jq . file.json
{
"results": [
{
"nom": "Kevin",
"status": "OK"
},
{
"nom": "Paul",
"status": "danger"
}
]
}
Package
For debian and debian like:
apt install libtemplate-perl
Via CPAN:
cpan -i Template::Toolkit
Check http://www.template-toolkit.org/docs/tools/tpage.html
One way:
paste <(printf "%s\n" "${Array1[#]}") <(printf "%s\n" "${Array2[#]}") |
jq -nRc '{ results: [inputs] | map(split("\t") | { nom: .[0], status: .[1] }) }'
produces
{"results":[{"nom":"Kevin","status":"OK"},{"nom":"Paul","status":"DANGER"}]}
This assumes that the elements of your arrays do not have tabs or newlines in them. It uses paste to generate pairs of corresponding array elements, separated by tabs, one pair per line, and then uses jq to create the JSON output from that.
If the objective is to stick with a non-jq solution - and chepner's comments about needing to validate the array entries is not an issue for this situation - one idea would be to loop through the array indices.
Test data:
$ declare -a test_array=(apple orange lemon)
$ typeset -p test_array
declare -a test_array=([0]="apple" [1]="orange" [2]="lemon")
$ declare -a test_array2=(DANGER OK WARNING)
$ typeset -p test_array2
declare -a test_array2=([0]="DANGER" [1]="OK" [2]="WARNING")
A simple loop through the indices (0,1,2):
sfx=',' # default printf suffix is a comma
for (( i=0 ; i<${#test_array[#]} ; i++ ))
do
(( ${i} == ( ${#test_array[#]} - 1 ) )) && sfx='' # clear suffix for last pass through loop
printf '{"CVEC": "%s", "LVL" : "%s"}%s\n' "${test_array[${i}]}" "${test_array2[${i}]}" "${sfx}"
done
Which generates the following:
{"CVEC": "apple", "LVL" : "DANGER"},
{"CVEC": "orange", "LVL" : "OK"},
{"CVEC": "lemon", "LVL" : "WARNING"}
I parsed a json file with jq like this :
# cat test.json | jq '.logs' | jq '.[]' | jq '._id' | jq -s
It returns an array like this : [34,235,436,546,.....]
Using bash script i described an array :
# declare -a msgIds = ...
This array uses () instead of [] so when I pass the array given above to this array it won't work.
([324,32,45..]) this causes problem. If i remove the jq -s, an array forms with only 1 member in it.
Is there a way to solve this issue?
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use #sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | #sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | #sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket().
It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | #sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that have spaces, newlines (or any other arbitrary characters) just use jq's #sh filter and bash's declare -a. (No need for a while read loop or any other pre-processing)
// foo.json
{"data": ["A B", "C'D", ""]}
str=$(jq -r '.data | #sh' foo.json)
declare -a arr="($str)" # must be quoted like this
$ declare -p arr
declare -a arr=([0]="A B" [1]="C'D" [2]="")
The reason that this works correctly is that #sh will produce a space-separated list of shell-quoted words:
$ echo "$str"
'A B' 'C'\''D' ''
and this is exactly the format that declare expects for an array definition.
Use jq -r to output a string "raw", without JSON formatting, and use the #sh formatter to format your results as a string for shell consumption. Per the jq docs:
#sh:
The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings.
So can do e.g.
msgids=($(<test.json jq -r '.logs[]._id | #sh'))
and get the result you want.
From the jq FAQ (https://github.com/stedolan/jq/wiki/FAQ):
𝑸: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
A: One option would be to use mapfile (aka readarray), for example:
mapfile -t array <<< $(jq -c '.[]' input.json)
An alternative that might be indicative of what to do in other shells is to use read -r within a while loop. The following bash script populates an array, x, with JSON texts. The key points are the use of the -c option, and the use of the bash idiom while read -r value; do ... done < <(jq .......):
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
++ To resolve this, we can use a very simple approach:
++ Since I am not aware of you input file, I am creating a file input.json with the following contents:
input.json:
{
"keys": ["key1","key2","key3"]
}
++ Use jq to get the value from the above file input.json:
Command: cat input.json | jq -r '.keys | #sh'
Output: 'key1' 'key2' 'key3'
Explanation: | #sh removes [ and "
++ To remove ' ' as well we use tr
command: cat input.json | jq -r '.keys | #sh' | tr -d \'
Explanation: use tr delete -d to remove '
++ To store this in a bash array we use () with `` and print it:
command:
KEYS=(`cat input.json | jq -r '.keys | #sh' | tr -d \'`)
To print all the entries of the array: echo "${KEYS[*]}"
I'm trying to run a script for all files in a directory with a common ID.
ls -1 *.vcf
a1.sourceA.vcf
a1.sourceB.vcf
a1.sourceC.vcf
a2.sourceA.vcf
a2.sourceB.vcf
a2.sourceC.vcf
a3.sourceA.vcf
a3.sourceC.vcf
The ID in each case precedes the first . (a1, a2 or a3) and for each ID I want to have all the sources for that ID in an associative array, keyed by the ID, e.g.;
a1 => [a1.sourceA.vcf, a1.sourceB.vcf, a1.sourceC.vcf]
I've attempted this as follows:
for file in $(ls *.vcf | sort)
do
id=$(echo $file | cut -d '.' -f 1)
vcfs[$id]+=$file
done
for i in "${!vcfs[#]}"
do
echo "key : $i"
echo "value: ${vcfs[$i]}"
echo " "
done
But I can't figure out how to get it working.
In Perl I would push values onto a hash of arrays in the loop:
push #{$vcfs{$id}}, $file;
to give me a data structure like this:
'a1' => [
'a1.sourceA.vcf',
'a1.sourceB.vcf',
'a1.sourceC.vcf'
],
'a3' => [
'a3.sourceA.vcf',
'a3.sourceC.vcf'
],
'a2' => [
'a2.sourceA.vcf',
'a2.sourceB.vcf',
'a2.sourceC.vcf'
]
How can I achieve this in bash?
From another answer given in question's comments
unset a1 a2 a3
function push {
local arr_name=$1
shift
if [[ $(declare -p "$arr_name" 2>&1) != "declare -a "* ]]
then
declare -g -a "$arr_name"
fi
declare -n array=$arr_name
array+=($#)
}
for file in *.vcf; do [[ -e $file ]] && push "${file%%.*}" "$file"; done
(IFS=,;echo "${a1[*]}")
(IFS=,;echo "${a2[*]}")
(IFS=,;echo "${a3[*]}")
But depending on needs maybe for with pattern is sufficient
for file in a1.*.vcf; do ... ; done
Finally $(ls ) must not be used in for loops as seen in other answers.
Why you shouldn't parse the output of ls
I'm parsing a JSON response with a tool called jq.
The output from jq will give me a list of full names in my command line.
I have the variable getNames which contains JSON, for example:
{
"count": 49,
"user": [{
"username": "jamesbrown",
"name": "James Brown",
"id": 1
}, {
"username": "matthewthompson",
"name": "Matthew Thompson",
"id": 2
}]
}
I pass this through JQ to filter the json using the following command:
echo $getNames | jq -r .user[].name
Which gives me a list like this:
James Brown
Matthew Thompson
I want to put each one of these entries into a bash array, so I enter the following commands:
declare -a myArray
myArray=( `echo $getNames | jq -r .user[].name` )
However, when I try to print the array using:
printf '%s\n' "${myArray[#]}"
I get the following:
James
Brown
Matthew
Thompson
How do I ensure that a new index is created after a new line and not a space? Why are the names being separated?
Thanks.
A simple script in bash to feed each line of the output into the array myArray.
#!/bin/bash
myArray=()
while IFS= read -r line; do
[[ $line ]] || break # break if line is empty
myArray+=("$line")
done < <(jq -r .user[].name <<< "$getNames")
# To print the array
printf '%s\n' "${myArray[#]}"
Just use mapfile command to read multiple lines into an array like this:
mapfile -t myArray < <(jq -r .user[].name <<< "$getNames")