Use jq to replace txt file array string values from dictionary - arrays

I have a dictionary that looks like this:
{
"uid": "d6fc3e2b-0001a",
"name": "ABC Mgmt",
"type": "host"
}
{
"uid": "d6fc3e2b-0002a",
"name": "Server XYZ",
"type": "group"
}
{
"uid": "d6fc3e2b-0003a",
"name": "NTP Primary",
"type": "host"
}
{
"uid": "d6fc3e2b-0004a",
"name": "H-10.10.10.10",
"type": "host"
}
Then I have a txt file:
"d6fc3e2b-0001a"
"d6fc3e2b-0001a","d6fc3e2b-0002a","d6fc3e2b-0003a"
"d6fc3e2b-0004a"
Expected Output:
"ABC Mgmt"
"ABC Mgmt","Server XYZ","NTP Primary"
"H-10.10.10.10"
I have some trouble to make jq using an array which is not json format. I tried various solutions that I found, but none of them worked. I am rather new to scripting, need some help.
input=file.txt
while IFS= read -r line
do
{
value=$(jq -r --arg line "$line" \
'from_entries | .[($line | split(","))[]]' \
dictionary.json)
echo $name
}
done < "$input"

In the following solution, the dictionary file is read using the --slurpfile command-line option, and the lines of "text" are read using inputs in conjunction with the -n command-line option. The -r command-line option is used in conjunction with the #csv filter to produce the desired output.
Invocation
jq -n -R -r --slurpfile dict stream.json -f program.jq stream.txt
program.jq
(INDEX($dict[]; .uid) | map_values(.name)) as $d
| inputs
| split(",")
| map(fromjson)
| map($d[.])
| #csv
Caveat
The above assumes that the quoted values in stream.txt do not themselves contain commas.
If the quoted values in stream.txt do contain commas, then it would be much easier if the values given on each line in stream.txt were given as JSON entities, e.g. as an array of strings, or as a sequence of JSON strings with no separator character.
Solution to problem described in a comment
Invocation
< original.json jq -r --slurpfile dict stream.json -f program.jq
program.jq
(INDEX($dict[]; .uid) | map_values(.name)) as $d
| .source
| map($d[.])
| #csv

Related

How to replace an objects value with array in a json file using jq? [duplicate]

This question already has answers here:
bash & jq: add attribute with object value
(2 answers)
Passing variable in jq to filter with select fails
(1 answer)
Closed 5 days ago.
I'm simply trying to replace an objects value in a json file with an array, using jq in a bash script.
The json file (truncated) looks like this:
{
"objects": {
"type": "foo",
"host": "1.1.1.1",
"port": "1234"
}
}
I want to replace the host objects value with an array of different values, so it looks like this:
{
"objects": {
"type": "foo",
"host": ["1.1.1.1","2.2.2.2"],
"port": "1234"
}
}
I tested around with this script. The Input comes from a simple, comma separated string which I convert into a proper json array (which seems to work).
But I'm not able replace the value with the array.
#!/bin/bash
objectshost="1.1.1.1,2.2.2.2"
objectshost_array=$(jq -c -n --arg arg $objectshost '$arg|split(",")')
jq --arg value "$objectshost_array" '.objects.host = $value' ./test.json > ./test.json.tmp
The best I ended up with, is this:
{
"objects": {
"type": "foo",
"host": "[\"1.1.1.1\",\"2.2.2.2\"]",
"port": "1234"
}
}
The result seems to be some logical result, as the script simply replaces the value with the arrays string. But it's not what I expected to get. ;)
I found some similar questions, but all of them were dealing with replacing values in existing arrays or key/value pairs, but my problem seems to be the conversion from a single value to an array.
Can somebody please push me into the right direction? Or should I forget about jq and threat the json file as a simple text file?
Thanks in advance,
André
It would work with a conditional assignment from arguments:
jq '
.objects.host = (
.objects.host |
if type == "array"
then .
else [ . ]
end + $ARGS.positional
)
' input.json --args 1.2.3.4 2.2.2.2 4.4.4.4
Or the same as a stand-alone jq script; which is more readable and maintainable:
myJQScript:
#!/usr/bin/env -S jq --from-file --args
.objects.host = (
.objects.host |
if type == "array"
then .
else [ . ]
end + $ARGS.positional
)
Make it executable:
chmod +x myJQScript
Run it with arguments to add array entries to host
$ ./myJQScript 1.2.3.4 2.2.2.2 4.4.4.4 < input.json
{
"objects": {
"type": "foo",
"host": [
"1.1.1.1",
"1.2.3.4",
"2.2.2.2",
"4.4.4.4"
],
"port": "1234"
}
}
You can do it with a single jq command:
#!/bin/sh
objectshost="1.1.1.1,2.2.2.2"
jq --arg value "$objectshost" '.objects.host = ($value / ",")' ./test.json > ./test.json.tmp
This has the added benefit of not requiring Bash arrays, but can be used with any shell.
If you already have a JSON array, you must use --argjson and not --arg. --arg always creates a variable of type string, --argjson however parses the value as JSON entity.
#!/bin/bash
objectshost="1.1.1.1,2.2.2.2"
objectshost_array=$(printf '%s\n' "$objectshost" | jq -c 'split(",")')
jq --argjson value "$objectshost_array" '.objects.host = $value' ./test.json > ./test.json.tmp

Use jq to replace array values from dictionary

I have a dictionary which looks like:
cat dictionary.json
[
{
"key": "key01",
"value": "value01"
},
{
"key": "key02",
"value": "value02"
},
{
"key": "key03",
"value": "value03",
"extraProperty": {
"foo": "bar"
}
},
{
"key": "key04",
"value": "value04"
}
]
Then, I have an array which is:
echo $array
key01 key02 key03
Expected output:
value01 value02 value03
I have some trouble to make jq using an array which is not json format.
I tried various solutions that I found, but none of them worked.
This post jq - How to select objects based on a 'whitelist' of property values seems to solve a similar problem but it doesn't work with my input:
echo $array | jq --argfile whitelist dictionary.json 'select(any(.key== $whitelist[]; .value))'
parse error: Invalid numeric literal at line 1, column 6
I also tried to use
jq -n --arg array $array --argfile whitelist dico.json 'select(any(.key== $whitelist[]; .valuee))'
jq: error: key02/0 is not defined at <top-level>, line 1:
key02
jq: 1 compile error
Thanks!
Here
jq -r --arg array "$array" \
'from_entries | .[($array | split(" "))[]]' \
dictionary.json
Output
value01
value02
value03
See man jq for further information.
Using INDEX/2, which constructs a dictionary:
echo 'key01 key02 key03' |
jq -Rr --argfile dict dictionary.json '
INDEX($dict[]; .key) as $d
| split(" ") | map( $d[.]|.value )
| join(" ")'
yields:
value01 value02 value03
If your jq does not have INDEX, then now would be an excellent time to upgrade to jq 1.6; alternatively, you can simply snarf its def by googling: jq "def INDEX"

How to compare Json data with array using Shell Script?

I have one json e.g
var bbview ={
"tbl_am_api":[
{
"Modified_User":"user1",
"Modified_Time":"04-Jul-2018 01:40:05",
"Line_Number":"SS001",
"Service_Type":"BB3",
"Status":"Yes",
"ID":3144526000014337832,
"Added_Time":"04-May-2018 11:37:29"
},
{
"Modified_User":"user2",
"Modified_Time":"04-Jul-2018 01:40:05",
"Line_Number":"SS002",
"Service_Type":"BB2",
"Status":"Yes",
"ID":3144526000014337832,
"Added_Time":"04-May-2018 11:37:29"
},
{
"Modified_User":"user3",
"Modified_Time":"04-Jul-2018 01:40:05",
"Line_Number":"SS004",
"Service_Type":"BB1",
"Status":"No",
"ID":3144526000014337832,
"Added_Time":"04-May-2018 11:37:29"
}
]
};
I want to compare this json data and array. Primary key as Line Number.
arrayA = {[{Line_Number : SS001, Service_Type : BB3; Status : Yes}]}
arrayA have Line_Number SS001. Find this Line_Number in json and compare Service_Type value and Status value are same or not. I want to write with Shell Script in bash file. I am not proficient in shell script. Please help me.
Update:
I tried with following bash code. But still fail. Please advice me
echo "Download FMS AM API File"
rm -rf tbl_am_api_Report?authtoken=da84d49f334c33b88d30dd2c947a4ff0 && wget -q https://creator.zoho.com/api/json/fixed-management-system/view/tbl_am_api_Report?authtoken=da84d49f334c33b88d30dd2c947a4ff0&scope=creatorapi&zc_ownername=tmlbroadband < /dev/null
cat > tbl_api_Report?authtoken=da84d49f334c33b88d30dd2c947a4ff0 //read json file
for row in $(echo "${apiview}" | jq -r '.[] | #base64'); do
_jq() {
echo ${row} | base64 --decode | jq -r ${1}
}
echo $(_jq '.name') >> info.txt
done
mail -s "Test email" aa#gmail.com -A info.txt < /dev/null
The value of arrayA is not JSON so I am going to let you figure out how to extract the values of Line_Number and Status from arrayA (but see below). Once those values are available, one could proceed as illustrated here:
#!/bin/bash
bbview='...' # as above
echo "$bbview" |
jq --arg Line_Number SS001 --arg Status Yes '
.tbl_am_api
| map(select(.Line_Number==$Line_Number and .Status==$Status)) '
Output
[
{
"Modified_User": "user1",
"Modified_Time": "04-Jul-2018 01:40:05",
"Line_Number": "SS001",
"Service_Type": "BB3",
"Status": "Yes",
"ID": 3144526000014338000,
"Added_Time": "04-May-2018 11:37:29"
}
]
true/false
Under a different reading of the question, the following variant might be relevant:
echo "$bbview" |
jq --arg Line_Number SS001 --arg Status Yes '
.tbl_am_api
| map(select(.Line_Number==$Line_Number) | .Status==$Status) '
arrayA
If you are using a version of bash that supports associative arrays, you could define arrayA as an associative array, like so:
declare -A arrayA
arrayA=([Line_Number]=SS001 [Service_Type]=BB3 [Status]=Yes)
Then to retrieve the value associated with Line_Number, you would write: ${arrayA[Line_Number]}; etc.

Bash - Only go next index when new line occurs, instead of white space?

I'm parsing a JSON response with a tool called jq.
The output from jq will give me a list of full names in my command line.
I have the variable getNames which contains JSON, for example:
{
"count": 49,
"user": [{
"username": "jamesbrown",
"name": "James Brown",
"id": 1
}, {
"username": "matthewthompson",
"name": "Matthew Thompson",
"id": 2
}]
}
I pass this through JQ to filter the json using the following command:
echo $getNames | jq -r .user[].name
Which gives me a list like this:
James Brown
Matthew Thompson
I want to put each one of these entries into a bash array, so I enter the following commands:
declare -a myArray
myArray=( `echo $getNames | jq -r .user[].name` )
However, when I try to print the array using:
printf '%s\n' "${myArray[#]}"
I get the following:
James
Brown
Matthew
Thompson
How do I ensure that a new index is created after a new line and not a space? Why are the names being separated?
Thanks.
A simple script in bash to feed each line of the output into the array myArray.
#!/bin/bash
myArray=()
while IFS= read -r line; do
[[ $line ]] || break # break if line is empty
myArray+=("$line")
done < <(jq -r .user[].name <<< "$getNames")
# To print the array
printf '%s\n' "${myArray[#]}"
Just use mapfile command to read multiple lines into an array like this:
mapfile -t myArray < <(jq -r .user[].name <<< "$getNames")

Given a json array, how do I extract a list of key values by key, using jq?

I have a json array that looks like this:
{
"StackSummaries": [
{
"CreationTime": "2016-06-01T22:22:49.890Z",
"StackName": "foo-control-eu-west-1",
"StackStatus": "UPDATE_COMPLETE",
"LastUpdatedTime": "2016-06-01T22:47:58.433Z"
},
{
"CreationTime": "2016-04-13T11:22:04.250Z",
"StackName": "foo-bar-testing",
"StackStatus": "UPDATE_COMPLETE",
"LastUpdatedTime": "2016-04-26T16:17:07.570Z"
},
{
"CreationTime": "2016-04-10T01:09:49.428Z",
"StackName": "foo-ldap-eu-west-1",
"StackStatus": "UPDATE_COMPLETE",
"LastUpdatedTime": "2016-04-17T13:44:04.758Z"
}
]
}
I am looking to create text output that looks like this:
foo-control-eu-west-1
foo-bar-testing
foo-ldap-eu-west-1
Is jq able to do this? Specifically, what would the jq command line be that would select each StackName in the array and output each key one per line?
jq --raw-output '.StackSummaries[].StackName'
$ jq -r '[.StackSummaries[] | .StackName] | unique[]' input.json
foo-bar-testing
foo-control-eu-west-1
foo-ldap-eu-west-1
The -r option strips the quotation marks from the output. You might not want the call to 'unique'.
For reference, if you wanted all the key names:
$ jq '[.StackSummaries[] | keys[]] | unique' input.json
[
"CreationTime",
"LastUpdatedTime",
"StackName",
"StackStatus"
]
Here is another solution
jq -M -r '..|.StackName?|values' input.json

Resources