Convert JSON dictionary into Bash one? - arrays

I am trying to receive output from an aws-cli command into Bash, and use it as an input to the second.
I successfully saved the output into the variable with this helpful answer:
iid=$(aws ec2 run-instances ...)
and receive output like that:
{ "ImageId": "ami-0abcdef1234567890", "InstanceId": "i-1231231230abcdef0", "InstanceType": "t2.micro", ... }
I know that Bash since v4 supports associative arrays but I'm struggling converting one into one.
I tried to parse the dictionary in Bash but received error:
must use subscript when assigning associative array
This is because the right key syntax for Bash dicts is =, not :.
Finally I accessed the members by using this marvelous answer with sed:
echo $iid|sed 's/{//g;s/}//g;s/"//g'|cut -d ':' -f2
My question: is there any standard way of creating a Bash dictionary from JSON or text besides regex? Best-practice?
Considering the snippet from the answer I used, the sed-approach can be very verbose and the verbosity increases exponentially with the number of keys/members:
for items in `echo $db|sed 's/{//;s/}//'`
do
echo one${count} = `echo $items|sed 's/^.*\[//;s/\].*$//'|cut -d ',' -f1`
echo two${count} = `echo $items|sed 's/^.*\[//;s/\].*$//'|cut -d ',' -f2`
echo three${count} = `echo $items|sed 's/^.*\[//;s/\].*$//'|cut -d ',' -f3`
echo four${count} = `echo $items|sed 's/^.*\[//;s/\].*$//'|cut -d ',' -f4`
...
done
For simple dicts it is OK, but for the complex dictionaries with hundreds of keys and a big nesting level it is almost inapplicable.
Is there any unified approach for arbitrary dictionary?
P.S. I found answers about solving the opposite (receiving Bash dict in Python), solving the task through jq, creating dict from Bash to Bash and from non-common input but nothing about specifically JSON. I prefer not to use jq and python and stick to the standard Bash toolset, most of the answers of this collective answer use 3rd-party tools. Is it possible at all?

One way to turn your JSON object members into a Bash4+'s associative array:
#!/usr/bin/env bash
# shellcheck disable=SC2155 # Associative array declaration from JSON
declare -A assoc=$(
jq -r '"(",(to_entries | .[] | "["+(.key|#sh)+"]="+(.value|#sh)),")"' \
input.json
)
# Debug dump the Associative array declaration
typeset -p assoc
Sample output:
declare -A assoc=([InstanceId]="i-1231231230abcdef0" [InstanceType]="t2.micro" [ImageId]="ami-0abcdef1234567890" )

Related

Bash: Is there a way to add UUIDs and their Mount Points into an array?

I'm trying to write a backup script that takes a specific list of disk's UUIDs, mounts them to specified points, rsyncs the data to a specified end point, and then also does a bunch of other conditional checks when that's done. But since there will be a lot of checking after rsync, it would be nice to set each disk's UUID and associated/desired mount point as strings to save hardcoding them throughout the script, and also if say the UUIDs change in future (if a drive is updated or swapped out), this will be easier to maintain the script...
I've been looking at arrays in bash, to make the list of wanted disks, but have questions about how to make this possible, as my experience with arrays is non-existent!
The list of disks---in order of priority wanted to backup---are:
# sdc1 2.7T UUID 8C1CC0C19D012E29 /media/user/Documents/
# sdb1 1.8T UUID 39CD106C6FDA5907 /media/user/Photos/
# sdd1 3.7T UUID 5104D5B708E102C0 /media/user/Video/
... and notice I want to go sdc1, sdb1, sdd1 and so on, (i.e. custom order).
Is it possible to create this list, in order of priority, so it's something like this?
DisksToBackup
└─1
└─UUID => '8C1CC0C19D012E29'
└─MountPoint => '/media/user/Documents/'
└─2
└─UUID => '39CD106C6FDA5907'
└─MountPoint => '/media/user/Photos/'
└─3
└─UUID => '5104D5B708E102C0'
└─MountPoint => '/media/user/Video/'
OR some obviously better idea than this...
And then how to actually use this?
Let's say for example, how to go through our list and mount each disk (I know this is incorrect syntax, but again, I know nothing about arrays:
mount --uuid $DisksToBackup[*][UUID] $DisksToBackup[*][MountPoint]?
Update: Using Linux Mint 19.3
Output of bash --version gives: GNU bash, version 4.4.20(1)
Bash starting with version 4 provides associative arrays but only using a single dimension. Multiple dimensions you would have to simulate using keys such as 'sdc1-uuid' as shown in the following interactive bash examples (remove leading $ and > and bash output when putting into a script).
$ declare -A disks
$ disks=([0-uuid]=8C1CC0C19D012E29 [0-mount]=/media/user/Documents/
> [1-uuid]=39CD106C6FDA5907 [1-mount]=/media/user/Photos/)
$ echo ${disks[sdc1-uuid]}
8C1CC0C19D012E29
$ echo ${disks[*]}
/media/user/Documents/ 39CD106C6FDA5907 8C1CC0C19D012E29 /media/user/Photos/
$ echo ${!disks[*]}
0-mount 0-uuid 1-uuid 1-mount
However, there is no ordering for the keys (the order of the keys differs from the order in which we defined them). You may want to use a second array as in the following example which allows you to break down the multiple dimensions as well:
$ disks_order=(0 1)
$ for i in ${disks_order[*]}; do
> echo "${disks[$i-uuid]} ${disks[$i-mount]}"
> done
8C1CC0C19D012E29 /media/user/Documents/
39CD106C6FDA5907 /media/user/Photos/
In case you use bash version 3, you need to simulate the associative array using other means. See the question on associative arrays in bash 3 or simply represent your structure in a simple array such as which makes everything more readable anyway:
$ disks=(8C1CC0C19D012E29=/media/user/Documents/
> 39CD106C6FDA5907=/media/user/Photos/)
$ for disk in "${disks[#]}"; do
> uuid="${disk%=*}"
> path="${disk##*=}"
> echo "$uuid $path"
> done
8C1CC0C19D012E29 /media/user/Documents/
39CD106C6FDA5907 /media/user/Photos/
The %=* is a fancy way of saying remove everything after (and including) the = sign. And ##*= to remove everything before (and including) the = sign.
As an example of how one could read this data into a series of arrays:
#!/usr/bin/env bash
i=0
declare -g -A "disk$i"
declare -n currDisk="disk$i"
while IFS= read -r line || (( ${#currDisk[#]} )); do : "line=$line"
if [[ $line ]]; then
if [[ $line = *=* ]]; then
currDisk[${line%%=*}]=${line#*=}
else
printf 'WARNING: Ignoring unrecognized line: %q\n' "$line" >&2
fi
else
if [[ ${#currDisk[#]} ]]; then
declare -p "disk$i" >&2 # for debugging/demo: print out what we created
(( ++i ))
unset -n currDisk
declare -g -A "disk$i=( )"
declare -n currDisk="disk$i"
fi
fi
done < <(blkid -o export)
This gives you something like:
declare -g -A disk0=( [PARTLABEL]="primary" [UUID]="1111-2222-3333" [TYPE]=btrfs ...)
declare -g -A disk1=( [PARTLABEL]="esp" [LABEL]="boot" [TYPE]="vfat" ...)
...so you can write code iterating over them doing whatever search/comparison/etc you want. For example:
for _diskVar in "${!disk#}"; do # iterates over variable names starting with "disk"
declare -n _currDisk="$_diskVar" # refer to each such variable as _currDisk in turn
# replace the below with your actual application logic, whatever that is
if [[ ${_currDisk[LABEL]} = "something" ]] && [[ ${_currDisk[TYPE]} = "something_else" ]]; then
echo "Found ${_currDisk[DEVNAME]}"
fi
unset -n _currDisk # clear the nameref when done with it
done

Creating an array from JSON data of file/directory locations; bash not registering if element is a directory

I have a JSON file which holds data like this: "path/to/git/directory/location": "path/to/local/location". A minimum example of the file might be this:
{
"${HOME}/dotfiles/.bashrc": "${HOME}/.bashrc",
"${HOME}/dotfiles/.atom/": "${HOME}/.atom/"
}
I have a script that systematically reads the above JSON (called locations.json) and creates an array, and then prints elements of the array that are directories. MWE:
#!/usr/bin/env bash
unset sysarray
declare -A sysarray
while IFS=: read -r field data
do
sysarray["${field}"]="${data}"
done <<< $(sed '/^[{}]$/d;s/\s*"/"/g;s/,$//' locations.json)
for file in "${sysarray[#]}"
do
if [ -d "${file}" ]
then
echo "${file}"
fi
done
However, this does not print the directory (i.e., ${HOME}/.atom).
I don't understand why this is happening, because
I have tried creating an array manually (i.e., not from a JSON) and checking if its elements are directories, and that works fine.
I have tried echoing each element in the array into a temporary file and reading each line in the file to see if it was a product of how the information was stored in the array, but no luck.
I have tried adding | tr -d "[:blank:]" | tr -d '\"' after using sed on the JSON (to see if it was a product of unintended whitespace or quotes), but no luck.
I have tried simply running [ -d "${HOME}/.atom/" ] && echo '.atom is a directory', and that works (so indeed it is a directory). I'm unsure what might be causing this.
Help on this would be great!
You could use a tool to process json files properly, which will deal with any valid json.
#!/usr/bin/env bash
unset sysarray
declare -A sysarray
while IFS="=" read -r field data
do
sysarray["${field}"]=$(eval echo "${data}")
done <<< $(jq -r 'keys[] as $k | "\($k)=\(.[$k])"' locations.json)
for file in "${sysarray[#]}"
do
if [ -d "${file}" ]
then
echo "${file}"
fi
done
Another problem is that, once the extra quote signs are properly processed, we have a literal ${HOME} that is not expanded. The only solution I came is using eval to force the expansion. It is not the nicest way, but right now I cannot find a better solution.

Making array from values requested with jq

I am trying to make an array with Jq
My code is:
all='('$(cat players_temp.json | jq -r '.item1.items[1].firstName, .item1.items[1].lastName')')'
It gives the output
$ echo $all
(Luka Modrić)
$ echo $all[1]
(Luka Modrić)[1]
as you can see the array does not work like an array. I was expecting this:
$ echo $all[1]
Modrić
To create a bash array from jq output, see e.g. the following SO page:
How do I convert json array to bash array of strings with jq?
To understand why your approach failed, consider this transcript of a session with the bash shell:
$ all='('"Luka Modrić"')'
$ echo $all
(Luka Modrić)
$ echo $all[1]
(Luka Modrić)[1]
This essentailly shows that your question has nothing to do with jq at all.
If you want $all to be an array consisting of the two strings "Luka" and "Modrić" then you could write:
$ all=("Luca" "Modrić")
echo ${all[1]}
Modrić
$ echo ${all[0]}
Luca
Notice the correct bash syntax for arrays, and that the index origin is 0.
Summary
See the above-mentioned SO page for alternative ways to create a bash array from jq output.
The syntax for creating a bash array from a collection of strings can be summarized by:
ary=( v0 ... )
If ary is a bash array, ${ary[i]} is the i-th element, where i ranges from 0 to ${#ary[#]} - 1.

Store values from arrays in config-file to variables

I've got the following problem:
I got a config-file (written in bash) with multiple arrays, the amount of these arrays is different from config to config. Each array contains three values.
declare -a array0
array0=(value1 value2 value3)
#
declare -a array1
array1=(value1 value2 value3)
#
declare -a array2
array2=(value1 value2 value3)
Now, this config file is sourced into the main bash script. I want to go from array to array and store the values into single variables. My actual solution:
for ((i=0;i=2;i++))
do
if [ "$i" = 0 ]
then
wantedvalue1="${array0["$i"]}"
fi
if [ "$i" = 1 ]
then
wantedvalue2="${array0["$i"]}"
fi
if [ "$i" = 2 ]
then
wantedvalue3="${array0["$i"]}"
fi
done
I guess, this will work for one specific array. But how can I tell the script to analyze every array in the config file like this?
Thanks for any help!
You can find the arrays in your environment via set. This extracts the names of the arrays which have exactly three elements:
set | sed -n 's/^\([_A_Za-z][_A-Za-z0-9]*\)=(\[0]=.*\[2]="[^"]*")$/\1/p'
(The number of backslashes depends on your sed dialect. This worked for me on Debian, where backslashed parentheses are metacharacters for grouping, and bare parentheses are matched literally.)
I don't really see why you want to use a loop to extract just three elements, but the wacky indirect reference syntax in bash kind of forces it here.
for array in $(set |
sed -n 's/^\([_A_Za-z][_A-Za-z0-9]*\)=(\[0]=.*\[2]="[^"]*")$/\1/p'); do
for((i=0, j=1; i<3; ++i, ++j)); do
k="$array[$i]"
eval wantedvalue$j=\'${!k}\'
done
:
: code which uses the wantedvalues here
done
It would be a tad simpler if you just used another array for the wantedvalues. Then the pesky eval could be avoided, too.

bash grep results into array

In bash I'm trying to collect my grep results in array, each cell holding each line.
I'm downloaing urls with this line
wget -O index -E $CurrentURL
and then i want to grep the 'index' file results (other urls) into array each line per cell,
what should be the correct syntax?
Array=(grep "some expression" index)
??
readarray GREPPED < <(grep "some expression" index)
for item in "${GREPPED[#]}"
do
# echo
echo "${item}"
done
Oh, and combine those -v greps like so:
egrep -v '\.(jpg|gif|xml|zip|asp|php|pdf|rar|cgi|html?)'
Probably most elegant among several poor alternatives would be to use a temp file.
wget $blah | grep 'whatever' > $TMPFILE
declare -a arr
declare -i i=0
while read; do
arr[$i]="$REPLY"
((i = i + 1))
done < $TMPFILE
I don't have time to explain why, but do not pipe directly into read.
No Unix shell is an appropriate tool for this task. Perl, Groovy, Java, Python... lots of languages could handle this elegantly, but none of the Unix shells.

Resources