I have a json with a reasons array in the format [] or ["a","b","c"]. Basically i want to replace
drop_reasons=["a","b","c"] in json to drop_reasons="a,b,c" . I know we can use join(",") for this with jq .However , don't know how to modify it in the json .
I have tried this - cat test.json | jq ' .drop_reasons = .drop_reasons | join(",") ' | sponge test.json
, but doesn't see, to work , it tries to join the entire json rather than just drop_reasons. How do i tackle it ? Any help will be appreciated. Thanks
Sample json is :
{"id":11828997,"user":"8ddbceef-c374-44be-82f6-996b9d3f9cbd","timestamp":"2020-08-12T05:50:00+05:30","claim_timestamp":"2020-08-12T20:30:58+05:30","unique_key":"d56af2a7-10b8-4a98-b12c-a8aeab9fc56e","platform":"android","location_type":"indoor","activity_type":"unknown","activity_confidence":0,"total_day_steps":151744,"gf_total_steps":0,"step_count":122,"session_id":"1792b79c-1490-4b13-83e2-3c50ebce28f4","label":"indoor","is_claimed":false,"is_dropped":false,"drop_reasons":[],"is_valid":false,"invalid_reason":["steps>allowed_freq"],"conversion":null,"createdAt":"2020-08-12T20:30:58.385285+05:30","updatedAt":"2020-08-12T20:30:58.385285+05:30","location_uuid":null,"location_latitude":28.6673,"location_longitude":77.3915,"location_accuracy":1000,"location_speed":0,"location_timestamp":"2020-08-12T05:46:40+05:30","location_altitude":0,"location_ios_distance_filler":null,"location_ios_desired_accuracy":null,"location_distance_filter":0,"location_desired_accuracy":0,"location_course":0,"location_floor":null,"meta_data_geo_string":"28.6672867,77.3914746","meta_data_timezone":"Asia/Kolkata","meta_data_device_model":"Redmi Note 8 Pro","meta_data_device_brand":"redmi","meta_data_device_manufacturer":"xiaomi","meta_data_app_version":"0.9.31","meta_data_bundle_id":"com.pepkit.ssg","meta_data_build_no":"213","meta_data_plan_id":"a562ad72-54a9-4aea-941c-7f075e2a8b18"}
Using a pared-down sample JSON object with just the relevant key:
$ cat test.json
{
"drop_reasons": ["a","b","c"]
}
$ jq '.drop_reasons |= join(",")' test.json
{
"drop_reasons": "a,b,c"
}
Your sample with an empty array would change to an empty string.
x |= y is essentially shorthand for x = (x | y). The parens are what you were missing in your attempt; they're needed because of jq precedence rules.
I'm not familiar with jq, but if you just replace every "," with ,, it does the trick. It's a hack, so it could break if the input data changes format.
$ cat temp.txt
drop_reasons=["a","b","c"]
$ perl -pe 's/","/,/g' temp.txt
drop_reasons=["a,b,c"]
Related
I am trying to curl some URL which returns a json file, as follows
[
10,
20,
30,
40,
]
now I am trying to save these value in file with a variable assigned to it.
need output in file as
a=10
b=20
c=30
d=40
thanks in advance.
You could use implode to create the letters by array index, which is provided in .key when to_entries is applied to an array.
jq -r 'to_entries[] | "\([.key + 97] | implode)=\(.value)"'
a=10
b=20
c=30
d=40
Demo
To provide individual names, you can introduce an array of names and use transpose to make the alignment:
jq -r '
[["Price", "Amount", "Value", "Tax"], .]
| transpose[] | "\(first)=\(last)"
'
Price=10
Amount=20
Value=30
Tax=40
Demo
Note: The list could also be provided from outside jq:
using a JSON array
jq -r --argjson names '["Price", "Amount", "Value", "Tax"]' \
'[$names, .] | transpose[] | "\(first)=\(last)"'
using positional parameters
jq -r '[$ARGS.positional, .] | transpose[] | "\(first)=\(last)"' \
--args Price Amount Value Tax
Assuming I have the following JSON object (which is just an example):
{
"foo": 1,
"bar": 2,
"baz": 3
}
And the following JSON array (another example):
["foo", "baz"]
How could I use jq to output the following object?
{
"foo": 1,
"baz": 3
}
I hate asking this question because I feel certain it has been answered before, but google has failed me and my jq-fu is just not where it needs to be to figure this out.
Using a reduce to iteratively build up the result object would be one way:
echo '["foo", "baz"]' | jq --argjson index '{"foo":1,"bar":2,"baz":3}' '
reduce .[] as $x ({}; .[$x] = $index[$x])
'
Using JOIN, creating key-value pairs, and employing from_entries for assembly would be another way:
echo '["baz", "foo"]' | jq --argjson index '{"foo":1,"bar":2,"baz":3}' '
JOIN($index; .) | map({key:.[0], value:.[1]}) | from_entries
'
Output:
{
"foo": 1,
"baz": 3
}
Provided that . is the object and $arr is the array, the following does the trick
delpaths(keys - $arr | map([.]))
To achieve the desired result, one could write:
jq '{foo, baz}'
This can (with some considerable trepidation) be made into a solution for the given problem by text wrangling, e.g. along the lines of:
jq "$(echo '["foo", "baz"]' | jq -Rr '"{" + .[1:-1] + "}" ')"
or
jq "$(echo '["foo", "baz"]' | sed -e 's/\[/{/' -e 's/\]/}/')"
Here's a reduce-free solution that assumes $keys is the array of keys of interest and that is possibly more efficient than the one involving array subtraction:
. as $in | INDEX( $keys[]; $in[.] )
I have an array that is being stored in a bash variable that is a json array of dates (it's currently being produced by a separate script). It looks like the following
["2019-09-19 03:13:29", "2019-09-19 20:20:18", "2019-09-19 18:19:50", "2019-09-19 06:07:17", "2019-09-19 11:53:25"]
I want to iterate over these dates and preserve the white space in the dates so I can use them with the gdate command. I'm currently using jq but that is splitting up the date part from the time part. i.e.
dates=$(python date_producing_script.py | jq -c -r '.[]')
for date in ${dates[#]}; do
echo $date
#end goal is to do something here w/ gdate ex gdate -d $date ...
done
However this gives something like
2019-09-19
03:13:29
2019-09-19
20:20:18
2019-09-19
18:19:50
2019-09-19
06:07:17
2019-09-19
11:53:25
Where I'm looking for something like the following
2019-09-19 03:13:29
2019-09-19 20:20:18
2019-09-19 18:19:50
2019-09-19 06:07:17
2019-09-19 11:53:25
(one confusing thing for me here is that if I just do the python date_producing_script.py | jq -c -r '.[]' command in a terminal it looks the way I want it to)
However ideally I would want something like. Is there anyway to get this result from the input in shell script
Your jq is fine; the problem is in the way you're looping through the bash array, as can be seen from this bash typescript:
$ ary=("a b" "c d")
$ for x in ${ary[#]} ; do echo "$x" ; done
a
b
c
d
$ for x in "${ary[#]}" ; do echo "$x" ; done
a b
c d
$
IOW
"${dates[#]}"
Here is one solution:
#!/bin/bash
#
IFS=','
dates='["2019-09-19 03:13:29", "2019-09-19 20:20:18", "2019-09-19 18:19:50", "2019-09-19 06:07:17", "2019-09-19 11:53:25"]'
for date in ${dates[#]}
do
# Cleanup $date
thedate=$(echo $date | tr -d '\[\]"' | sed -e 's/^[[:space:]]*//')
echo "-->$thedate<--"
done
IFS: sets the separator as ,
$dates is now separated on ,
$thedate is $date with [, ], " removed. The sed is to remove prefix spaces.
I put --> and <-- to show that $thedate does not include any spaces.
There are most probably solutions with awk too.
One of your issue is when you populate dates array ,
and the other is when you want to use the array for loop .
My proposal is :
change jq to generate a long string of date with = as a separator
tell bash to use = as separator instead of <space><tab><line return>
you must use the syntax "$dates[#]" instead of $dates[#]
Your script will be like :
OLDIFS=$IFS ;
IFS="=" ;
dates=( python date_producing_script.py | jq -c -j -r '.[]|(.,"=")') );
IFS=$OLDIFS
for d in "${dates[#]}" ;
do
echo $d ;
done
I parsed a json file with jq like this :
# cat test.json | jq '.logs' | jq '.[]' | jq '._id' | jq -s
It returns an array like this : [34,235,436,546,.....]
Using bash script i described an array :
# declare -a msgIds = ...
This array uses () instead of [] so when I pass the array given above to this array it won't work.
([324,32,45..]) this causes problem. If i remove the jq -s, an array forms with only 1 member in it.
Is there a way to solve this issue?
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use #sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | #sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | #sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket().
It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | #sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that have spaces, newlines (or any other arbitrary characters) just use jq's #sh filter and bash's declare -a. (No need for a while read loop or any other pre-processing)
// foo.json
{"data": ["A B", "C'D", ""]}
str=$(jq -r '.data | #sh' foo.json)
declare -a arr="($str)" # must be quoted like this
$ declare -p arr
declare -a arr=([0]="A B" [1]="C'D" [2]="")
The reason that this works correctly is that #sh will produce a space-separated list of shell-quoted words:
$ echo "$str"
'A B' 'C'\''D' ''
and this is exactly the format that declare expects for an array definition.
Use jq -r to output a string "raw", without JSON formatting, and use the #sh formatter to format your results as a string for shell consumption. Per the jq docs:
#sh:
The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings.
So can do e.g.
msgids=($(<test.json jq -r '.logs[]._id | #sh'))
and get the result you want.
From the jq FAQ (https://github.com/stedolan/jq/wiki/FAQ):
𝑸: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
A: One option would be to use mapfile (aka readarray), for example:
mapfile -t array <<< $(jq -c '.[]' input.json)
An alternative that might be indicative of what to do in other shells is to use read -r within a while loop. The following bash script populates an array, x, with JSON texts. The key points are the use of the -c option, and the use of the bash idiom while read -r value; do ... done < <(jq .......):
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
++ To resolve this, we can use a very simple approach:
++ Since I am not aware of you input file, I am creating a file input.json with the following contents:
input.json:
{
"keys": ["key1","key2","key3"]
}
++ Use jq to get the value from the above file input.json:
Command: cat input.json | jq -r '.keys | #sh'
Output: 'key1' 'key2' 'key3'
Explanation: | #sh removes [ and "
++ To remove ' ' as well we use tr
command: cat input.json | jq -r '.keys | #sh' | tr -d \'
Explanation: use tr delete -d to remove '
++ To store this in a bash array we use () with `` and print it:
command:
KEYS=(`cat input.json | jq -r '.keys | #sh' | tr -d \'`)
To print all the entries of the array: echo "${KEYS[*]}"
In code below I'm trying to match values from a bash array with values from json array using jq.
PROJECTS=$(curl -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "${GITLAB_URL}/api/v4/projects")
for GITLAB_TAG in "${GITLAB_TAGS[#]}"; do
PROJECTS=`echo "${PROJECTS}" \
| jq --arg gitlab_tag "$GITLAB_TAG" '[ .[] | select(.tag_list[] | contains($gitlab_tag)) ]'`
done
PROJECTS=$(echo "$PROJECTS" | jq -r '.[]')
Consider the following JSON payload as a sample API response:
[{"id":31,"description":"","default_branch":null,"tag_list":["dev","app"],"archived":false,"visibility":"private"},{"id":28,"description":"","default_branch":"master","tag_list":["dev","app"],"archived":false,"visibility":"private"}]
This works only partially, because I can't iterate over results ($PROJECTS).
printf "${PROJECTS[0]}" prints the whole array.
Am I missing something here?
It appears that you should be using index/1 to check .tag_list (not .tag_list[]):
[ .[] | select(.tag_list | index($gitlab_tag))
jq has no knowledge of bash array variables. For these, you therefore have two basic options: 1) use bash to iterate over the bash array variable (calling jq as often as necessary); 2) present the contents of the bash array variable in a way that jq can handle.
There are many variations of (2). Consider for example:
$ a=(a "b c")
$ printf "%s\n" "${a[#]}" | jq -R | jq -s
[
"a",
"b c"
]