I have a bash script that uses filenames as keys in an associative array. Some of the filenames have quotes in them and I can't seem to find any way to unset them.
Here's an example replicating the problem from the terminal:
$ declare -A x
$ y="key with spaces"
$ z="key with spaces and ' quote"
$ x[$y]=5 # this works fine
$ x[$z]=44 # as does this
$ echo "${x[$y]}" "${x[$z]}" # no problems here
5 44
$ unset x["$y"] # works
$ unset x["$z"] # does not work
bash: unset: `x[key with spaces and ' quote]': not a valid identifier
$ echo "${x[$y]}" "${x[$z]}" # second key was not deleted
44
The file names processed in my script are arbitrary and need to work regardless of what characters they have in them (within reason, at least needs to work with printable characters.) The unset is used to clear a flag on files with certain properties.
How can I get bash to unset these particular keys when they might contain quote symbols?
I find this works for me:
unset 'x[$z]'
This works for other special characters:
$ y="key with spaces"
$ v="\$ ' \" # # * & \`"
$ x[$y]=5
$ x[$v]=10
$ echo ${x[*]}
5 10
$ unset 'x[$v]'
$ echo ${x[*]}
5
This is the case where single quotes around associative array key work as well so this should work:
$> declare -p x
declare -A x='(["key with spaces and '\'' quote"]="44" )'
$> unset x['$z']
$> declare -p x
declare -A x='()'
It may be a bug (at the very least, it is inconvenient). Until a fix is available, you can work around this by getting a shell-quoted version of $z with
$ unset x["$(printf '%q' "$z")"]
In bash 4.4, this can more briefly be written as
$ unset x["${z#Q}"]
Related
I'm trying to write a backup script that takes a specific list of disk's UUIDs, mounts them to specified points, rsyncs the data to a specified end point, and then also does a bunch of other conditional checks when that's done. But since there will be a lot of checking after rsync, it would be nice to set each disk's UUID and associated/desired mount point as strings to save hardcoding them throughout the script, and also if say the UUIDs change in future (if a drive is updated or swapped out), this will be easier to maintain the script...
I've been looking at arrays in bash, to make the list of wanted disks, but have questions about how to make this possible, as my experience with arrays is non-existent!
The list of disks---in order of priority wanted to backup---are:
# sdc1 2.7T UUID 8C1CC0C19D012E29 /media/user/Documents/
# sdb1 1.8T UUID 39CD106C6FDA5907 /media/user/Photos/
# sdd1 3.7T UUID 5104D5B708E102C0 /media/user/Video/
... and notice I want to go sdc1, sdb1, sdd1 and so on, (i.e. custom order).
Is it possible to create this list, in order of priority, so it's something like this?
DisksToBackup
└─1
└─UUID => '8C1CC0C19D012E29'
└─MountPoint => '/media/user/Documents/'
└─2
└─UUID => '39CD106C6FDA5907'
└─MountPoint => '/media/user/Photos/'
└─3
└─UUID => '5104D5B708E102C0'
└─MountPoint => '/media/user/Video/'
OR some obviously better idea than this...
And then how to actually use this?
Let's say for example, how to go through our list and mount each disk (I know this is incorrect syntax, but again, I know nothing about arrays:
mount --uuid $DisksToBackup[*][UUID] $DisksToBackup[*][MountPoint]?
Update: Using Linux Mint 19.3
Output of bash --version gives: GNU bash, version 4.4.20(1)
Bash starting with version 4 provides associative arrays but only using a single dimension. Multiple dimensions you would have to simulate using keys such as 'sdc1-uuid' as shown in the following interactive bash examples (remove leading $ and > and bash output when putting into a script).
$ declare -A disks
$ disks=([0-uuid]=8C1CC0C19D012E29 [0-mount]=/media/user/Documents/
> [1-uuid]=39CD106C6FDA5907 [1-mount]=/media/user/Photos/)
$ echo ${disks[sdc1-uuid]}
8C1CC0C19D012E29
$ echo ${disks[*]}
/media/user/Documents/ 39CD106C6FDA5907 8C1CC0C19D012E29 /media/user/Photos/
$ echo ${!disks[*]}
0-mount 0-uuid 1-uuid 1-mount
However, there is no ordering for the keys (the order of the keys differs from the order in which we defined them). You may want to use a second array as in the following example which allows you to break down the multiple dimensions as well:
$ disks_order=(0 1)
$ for i in ${disks_order[*]}; do
> echo "${disks[$i-uuid]} ${disks[$i-mount]}"
> done
8C1CC0C19D012E29 /media/user/Documents/
39CD106C6FDA5907 /media/user/Photos/
In case you use bash version 3, you need to simulate the associative array using other means. See the question on associative arrays in bash 3 or simply represent your structure in a simple array such as which makes everything more readable anyway:
$ disks=(8C1CC0C19D012E29=/media/user/Documents/
> 39CD106C6FDA5907=/media/user/Photos/)
$ for disk in "${disks[#]}"; do
> uuid="${disk%=*}"
> path="${disk##*=}"
> echo "$uuid $path"
> done
8C1CC0C19D012E29 /media/user/Documents/
39CD106C6FDA5907 /media/user/Photos/
The %=* is a fancy way of saying remove everything after (and including) the = sign. And ##*= to remove everything before (and including) the = sign.
As an example of how one could read this data into a series of arrays:
#!/usr/bin/env bash
i=0
declare -g -A "disk$i"
declare -n currDisk="disk$i"
while IFS= read -r line || (( ${#currDisk[#]} )); do : "line=$line"
if [[ $line ]]; then
if [[ $line = *=* ]]; then
currDisk[${line%%=*}]=${line#*=}
else
printf 'WARNING: Ignoring unrecognized line: %q\n' "$line" >&2
fi
else
if [[ ${#currDisk[#]} ]]; then
declare -p "disk$i" >&2 # for debugging/demo: print out what we created
(( ++i ))
unset -n currDisk
declare -g -A "disk$i=( )"
declare -n currDisk="disk$i"
fi
fi
done < <(blkid -o export)
This gives you something like:
declare -g -A disk0=( [PARTLABEL]="primary" [UUID]="1111-2222-3333" [TYPE]=btrfs ...)
declare -g -A disk1=( [PARTLABEL]="esp" [LABEL]="boot" [TYPE]="vfat" ...)
...so you can write code iterating over them doing whatever search/comparison/etc you want. For example:
for _diskVar in "${!disk#}"; do # iterates over variable names starting with "disk"
declare -n _currDisk="$_diskVar" # refer to each such variable as _currDisk in turn
# replace the below with your actual application logic, whatever that is
if [[ ${_currDisk[LABEL]} = "something" ]] && [[ ${_currDisk[TYPE]} = "something_else" ]]; then
echo "Found ${_currDisk[DEVNAME]}"
fi
unset -n _currDisk # clear the nameref when done with it
done
I need to read the output of a command in my script into an array. The command is, for example:
ps aux | grep | grep | x
and it gives the output line by line like this:
10
20
30
I need to read the values from the command output into an array, and then I will do some work if the size of the array is less than three.
The other answers will break if output of command contains spaces (which is rather frequent) or glob characters like *, ?, [...].
To get the output of a command in an array, with one line per element, there are essentially 3 ways:
With Bash≥4 use mapfile—it's the most efficient:
mapfile -t my_array < <( my_command )
Otherwise, a loop reading the output (slower, but safe):
my_array=()
while IFS= read -r line; do
my_array+=( "$line" )
done < <( my_command )
As suggested by Charles Duffy in the comments (thanks!), the following might perform better than the loop method in number 2:
IFS=$'\n' read -r -d '' -a my_array < <( my_command && printf '\0' )
Please make sure you use exactly this form, i.e., make sure you have the following:
IFS=$'\n' on the same line as the read statement: this will only set the environment variable IFS for the read statement only. So it won't affect the rest of your script at all. The purpose of this variable is to tell read to break the stream at the EOL character \n.
-r: this is important. It tells read to not interpret the backslashes as escape sequences.
-d '': please note the space between the -d option and its argument ''. If you don't leave a space here, the '' will never be seen, as it will disappear in the quote removal step when Bash parses the statement. This tells read to stop reading at the nil byte. Some people write it as -d $'\0', but it is not really necessary. -d '' is better.
-a my_array tells read to populate the array my_array while reading the stream.
You must use the printf '\0' statement after my_command, so that read returns 0; it's actually not a big deal if you don't (you'll just get an return code 1, which is okay if you don't use set -e – which you shouldn't anyway), but just bear that in mind. It's cleaner and more semantically correct. Note that this is different from printf '', which doesn't output anything. printf '\0' prints a null byte, needed by read to happily stop reading there (remember the -d '' option?).
If you can, i.e., if you're sure your code will run on Bash≥4, use the first method. And you can see it's shorter too.
If you want to use read, the loop (method 2) might have an advantage over method 3 if you want to do some processing as the lines are read: you have direct access to it (via the $line variable in the example I gave), and you also have access to the lines already read (via the array ${my_array[#]} in the example I gave).
Note that mapfile provides a way to have a callback eval'd on each line read, and in fact you can even tell it to only call this callback every N lines read; have a look at help mapfile and the options -C and -c therein. (My opinion about this is that it's a little bit clunky, but can be used sometimes if you only have simple things to do — I don't really understand why this was even implemented in the first place!).
Now I'm going to tell you why the following method:
my_array=( $( my_command) )
is broken when there are spaces:
$ # I'm using this command to test:
$ echo "one two"; echo "three four"
one two
three four
$ # Now I'm going to use the broken method:
$ my_array=( $( echo "one two"; echo "three four" ) )
$ declare -p my_array
declare -a my_array='([0]="one" [1]="two" [2]="three" [3]="four")'
$ # As you can see, the fields are not the lines
$
$ # Now look at the correct method:
$ mapfile -t my_array < <(echo "one two"; echo "three four")
$ declare -p my_array
declare -a my_array='([0]="one two" [1]="three four")'
$ # Good!
Then some people will then recommend using IFS=$'\n' to fix it:
$ IFS=$'\n'
$ my_array=( $(echo "one two"; echo "three four") )
$ declare -p my_array
declare -a my_array='([0]="one two" [1]="three four")'
$ # It works!
But now let's use another command, with globs:
$ echo "* one two"; echo "[three four]"
* one two
[three four]
$ IFS=$'\n'
$ my_array=( $(echo "* one two"; echo "[three four]") )
$ declare -p my_array
declare -a my_array='([0]="* one two" [1]="t")'
$ # What?
That's because I have a file called t in the current directory… and this filename is matched by the glob [three four]… at this point some people would recommend using set -f to disable globbing: but look at it: you have to change IFS and use set -f to be able to fix a broken technique (and you're not even fixing it really)! when doing that we're really fighting against the shell, not working with the shell.
$ mapfile -t my_array < <( echo "* one two"; echo "[three four]")
$ declare -p my_array
declare -a my_array='([0]="* one two" [1]="[three four]")'
here we're working with the shell!
You can use
my_array=( $(<command>) )
to store the output of command <command> into the array my_array.
You can access the length of that array using
my_array_length=${#my_array[#]}
Now the length is stored in my_array_length.
Here is a simple example. Imagine that you are going to put the files and directory names (under the current folder) to an array and count them. The script would be like;
my_array=( `ls` )
my_array_length=${#my_array[#]}
echo $my_array_length
Or, you can iterate over this array by adding the following script:
for element in "${my_array[#]}"
do
echo "${element}"
done
Please note that this is the core concept and the input must be sanitized before the processing, i.e. removing extra characters, handling empty Strings, and etc. (which is out of the topic of this thread).
It helps me all the time suppose you want to copy whole list of directories into current directory into an array
bucketlist=($(ls))
#then print them one by one
for bucket in "${bucketlist[#]}"; do
echo " here is bucket: ${bucket}"
done
here https://stackoverflow.com/a/19915925/4673197 I learned I can split a string into array by setting IFS.
here https://stackoverflow.com/a/9429887 I learned I can join a array by IFS delimiter.
But in my test below:
0:~ $ a=(1 2 3)
0:~ $ echo "${a[*]}"
1 2 3
0:~ $ IFS=. echo "${a[*]}" # IFS=. not work
1 2 3
0:~ $ (IFS=.; echo "${a[*]}") # this works
1.2.3
0:~ $ echo $IFS # the original IFS is not change
0:~ $ v=1.2.3
0:~ $ IFS=. b=($v) # change string to array
0:~ $ echo ${b[*]}
1 2 3
0:~ $ echo "${b[*]}" # the array join by `.`!
1.2.3
0:~ $ echo ${b}
1
0:~ $ (IFS=,; echo "${b[*]}") # this still work
1,2,3
0:~ $ IFS=, echo "${b[*]}" # this not work, b array still join by .
1.2.3
0:~ $ c=(1 2 3)
0:~ $ echo "${c[*]}" # a new array join by '.' !
1.2.3
0:~ $ IFS=, echo "${c[*]}" # IFS=, not work, still join by '.'
1.2.3
0:~ $ (IFS=,; echo "${c[*]}") # this works
1,2,3
0:~ $ echo $IFS # original IFS is space
0:~ $ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.
from the above, I have the following guessing/questions:
IFS=. echo "${a[*]}" this change the echo environment, but not change the quoting expanding environment, so it's not work?
if 1 is true, why IFS=. b=($v) successful create the array?
why echo "${c[*]}" joined by .? the IFS should be default whitespace.
The problem is with the IFS setting when you do IFS=. b=($v) two shell assignments take place in the context of the current shell. The modification to IFS to . will make all subsequent array expansions of type "${arr[*]}" to make use of this new value. Note the double quotes here, same array expansion without the quotes won't use IFS i.e. ${arr[*]}
So to answer your questions
IFS=. echo "${a[*]}" this change the echo environment, but not change the quoting expanding environment, so it's not work?
You are wrong, though you are passing IFS to the environment that echo receives. However, echo doesn't do anything with its environment list. The result happens because you've earlier set the value of IFS to .
if 1 is true, why IFS=. b=($v) successful create the array?
This command runs as if you were executing the two commands separately. Modify the IFS to . and expand the result of unquoted string v into the array which undergoes word-splitting with the value of IFS. Since you've defined it to be . previously, the string is split into individual components and stored into the array.
why echo "${c[*]}" joined by .? the IFS should be default whitespace.
It is already explained in the first answer. You've modified in the current shell to use IFS to be . and subsequent array expansions will take these for joining strings. If you start a new shell and run the same expansion it won't work.
Just to clarify one more point on when passing a variable to the local environment of the command will work. If instead of
var=val echo "$var"
if you had written
var=val sh -c 'echo "$var"'
you see the value of var written, because in this case, you are passing the value of var in the environment, the sh shell is run. But unlike echo, sh reads from the environment and sees the value of var and uses it for variable expansion.
I need to read the output of a command in my script into an array. The command is, for example:
ps aux | grep | grep | x
and it gives the output line by line like this:
10
20
30
I need to read the values from the command output into an array, and then I will do some work if the size of the array is less than three.
The other answers will break if output of command contains spaces (which is rather frequent) or glob characters like *, ?, [...].
To get the output of a command in an array, with one line per element, there are essentially 3 ways:
With Bash≥4 use mapfile—it's the most efficient:
mapfile -t my_array < <( my_command )
Otherwise, a loop reading the output (slower, but safe):
my_array=()
while IFS= read -r line; do
my_array+=( "$line" )
done < <( my_command )
As suggested by Charles Duffy in the comments (thanks!), the following might perform better than the loop method in number 2:
IFS=$'\n' read -r -d '' -a my_array < <( my_command && printf '\0' )
Please make sure you use exactly this form, i.e., make sure you have the following:
IFS=$'\n' on the same line as the read statement: this will only set the environment variable IFS for the read statement only. So it won't affect the rest of your script at all. The purpose of this variable is to tell read to break the stream at the EOL character \n.
-r: this is important. It tells read to not interpret the backslashes as escape sequences.
-d '': please note the space between the -d option and its argument ''. If you don't leave a space here, the '' will never be seen, as it will disappear in the quote removal step when Bash parses the statement. This tells read to stop reading at the nil byte. Some people write it as -d $'\0', but it is not really necessary. -d '' is better.
-a my_array tells read to populate the array my_array while reading the stream.
You must use the printf '\0' statement after my_command, so that read returns 0; it's actually not a big deal if you don't (you'll just get an return code 1, which is okay if you don't use set -e – which you shouldn't anyway), but just bear that in mind. It's cleaner and more semantically correct. Note that this is different from printf '', which doesn't output anything. printf '\0' prints a null byte, needed by read to happily stop reading there (remember the -d '' option?).
If you can, i.e., if you're sure your code will run on Bash≥4, use the first method. And you can see it's shorter too.
If you want to use read, the loop (method 2) might have an advantage over method 3 if you want to do some processing as the lines are read: you have direct access to it (via the $line variable in the example I gave), and you also have access to the lines already read (via the array ${my_array[#]} in the example I gave).
Note that mapfile provides a way to have a callback eval'd on each line read, and in fact you can even tell it to only call this callback every N lines read; have a look at help mapfile and the options -C and -c therein. (My opinion about this is that it's a little bit clunky, but can be used sometimes if you only have simple things to do — I don't really understand why this was even implemented in the first place!).
Now I'm going to tell you why the following method:
my_array=( $( my_command) )
is broken when there are spaces:
$ # I'm using this command to test:
$ echo "one two"; echo "three four"
one two
three four
$ # Now I'm going to use the broken method:
$ my_array=( $( echo "one two"; echo "three four" ) )
$ declare -p my_array
declare -a my_array='([0]="one" [1]="two" [2]="three" [3]="four")'
$ # As you can see, the fields are not the lines
$
$ # Now look at the correct method:
$ mapfile -t my_array < <(echo "one two"; echo "three four")
$ declare -p my_array
declare -a my_array='([0]="one two" [1]="three four")'
$ # Good!
Then some people will then recommend using IFS=$'\n' to fix it:
$ IFS=$'\n'
$ my_array=( $(echo "one two"; echo "three four") )
$ declare -p my_array
declare -a my_array='([0]="one two" [1]="three four")'
$ # It works!
But now let's use another command, with globs:
$ echo "* one two"; echo "[three four]"
* one two
[three four]
$ IFS=$'\n'
$ my_array=( $(echo "* one two"; echo "[three four]") )
$ declare -p my_array
declare -a my_array='([0]="* one two" [1]="t")'
$ # What?
That's because I have a file called t in the current directory… and this filename is matched by the glob [three four]… at this point some people would recommend using set -f to disable globbing: but look at it: you have to change IFS and use set -f to be able to fix a broken technique (and you're not even fixing it really)! when doing that we're really fighting against the shell, not working with the shell.
$ mapfile -t my_array < <( echo "* one two"; echo "[three four]")
$ declare -p my_array
declare -a my_array='([0]="* one two" [1]="[three four]")'
here we're working with the shell!
You can use
my_array=( $(<command>) )
to store the output of command <command> into the array my_array.
You can access the length of that array using
my_array_length=${#my_array[#]}
Now the length is stored in my_array_length.
Here is a simple example. Imagine that you are going to put the files and directory names (under the current folder) to an array and count them. The script would be like;
my_array=( `ls` )
my_array_length=${#my_array[#]}
echo $my_array_length
Or, you can iterate over this array by adding the following script:
for element in "${my_array[#]}"
do
echo "${element}"
done
Please note that this is the core concept and the input must be sanitized before the processing, i.e. removing extra characters, handling empty Strings, and etc. (which is out of the topic of this thread).
It helps me all the time suppose you want to copy whole list of directories into current directory into an array
bucketlist=($(ls))
#then print them one by one
for bucket in "${bucketlist[#]}"; do
echo " here is bucket: ${bucket}"
done
I need to read the output of a command in my script into an array. The command is, for example:
ps aux | grep | grep | x
and it gives the output line by line like this:
10
20
30
I need to read the values from the command output into an array, and then I will do some work if the size of the array is less than three.
The other answers will break if output of command contains spaces (which is rather frequent) or glob characters like *, ?, [...].
To get the output of a command in an array, with one line per element, there are essentially 3 ways:
With Bash≥4 use mapfile—it's the most efficient:
mapfile -t my_array < <( my_command )
Otherwise, a loop reading the output (slower, but safe):
my_array=()
while IFS= read -r line; do
my_array+=( "$line" )
done < <( my_command )
As suggested by Charles Duffy in the comments (thanks!), the following might perform better than the loop method in number 2:
IFS=$'\n' read -r -d '' -a my_array < <( my_command && printf '\0' )
Please make sure you use exactly this form, i.e., make sure you have the following:
IFS=$'\n' on the same line as the read statement: this will only set the environment variable IFS for the read statement only. So it won't affect the rest of your script at all. The purpose of this variable is to tell read to break the stream at the EOL character \n.
-r: this is important. It tells read to not interpret the backslashes as escape sequences.
-d '': please note the space between the -d option and its argument ''. If you don't leave a space here, the '' will never be seen, as it will disappear in the quote removal step when Bash parses the statement. This tells read to stop reading at the nil byte. Some people write it as -d $'\0', but it is not really necessary. -d '' is better.
-a my_array tells read to populate the array my_array while reading the stream.
You must use the printf '\0' statement after my_command, so that read returns 0; it's actually not a big deal if you don't (you'll just get an return code 1, which is okay if you don't use set -e – which you shouldn't anyway), but just bear that in mind. It's cleaner and more semantically correct. Note that this is different from printf '', which doesn't output anything. printf '\0' prints a null byte, needed by read to happily stop reading there (remember the -d '' option?).
If you can, i.e., if you're sure your code will run on Bash≥4, use the first method. And you can see it's shorter too.
If you want to use read, the loop (method 2) might have an advantage over method 3 if you want to do some processing as the lines are read: you have direct access to it (via the $line variable in the example I gave), and you also have access to the lines already read (via the array ${my_array[#]} in the example I gave).
Note that mapfile provides a way to have a callback eval'd on each line read, and in fact you can even tell it to only call this callback every N lines read; have a look at help mapfile and the options -C and -c therein. (My opinion about this is that it's a little bit clunky, but can be used sometimes if you only have simple things to do — I don't really understand why this was even implemented in the first place!).
Now I'm going to tell you why the following method:
my_array=( $( my_command) )
is broken when there are spaces:
$ # I'm using this command to test:
$ echo "one two"; echo "three four"
one two
three four
$ # Now I'm going to use the broken method:
$ my_array=( $( echo "one two"; echo "three four" ) )
$ declare -p my_array
declare -a my_array='([0]="one" [1]="two" [2]="three" [3]="four")'
$ # As you can see, the fields are not the lines
$
$ # Now look at the correct method:
$ mapfile -t my_array < <(echo "one two"; echo "three four")
$ declare -p my_array
declare -a my_array='([0]="one two" [1]="three four")'
$ # Good!
Then some people will then recommend using IFS=$'\n' to fix it:
$ IFS=$'\n'
$ my_array=( $(echo "one two"; echo "three four") )
$ declare -p my_array
declare -a my_array='([0]="one two" [1]="three four")'
$ # It works!
But now let's use another command, with globs:
$ echo "* one two"; echo "[three four]"
* one two
[three four]
$ IFS=$'\n'
$ my_array=( $(echo "* one two"; echo "[three four]") )
$ declare -p my_array
declare -a my_array='([0]="* one two" [1]="t")'
$ # What?
That's because I have a file called t in the current directory… and this filename is matched by the glob [three four]… at this point some people would recommend using set -f to disable globbing: but look at it: you have to change IFS and use set -f to be able to fix a broken technique (and you're not even fixing it really)! when doing that we're really fighting against the shell, not working with the shell.
$ mapfile -t my_array < <( echo "* one two"; echo "[three four]")
$ declare -p my_array
declare -a my_array='([0]="* one two" [1]="[three four]")'
here we're working with the shell!
You can use
my_array=( $(<command>) )
to store the output of command <command> into the array my_array.
You can access the length of that array using
my_array_length=${#my_array[#]}
Now the length is stored in my_array_length.
Here is a simple example. Imagine that you are going to put the files and directory names (under the current folder) to an array and count them. The script would be like;
my_array=( `ls` )
my_array_length=${#my_array[#]}
echo $my_array_length
Or, you can iterate over this array by adding the following script:
for element in "${my_array[#]}"
do
echo "${element}"
done
Please note that this is the core concept and the input must be sanitized before the processing, i.e. removing extra characters, handling empty Strings, and etc. (which is out of the topic of this thread).
It helps me all the time suppose you want to copy whole list of directories into current directory into an array
bucketlist=($(ls))
#then print them one by one
for bucket in "${bucketlist[#]}"; do
echo " here is bucket: ${bucket}"
done