I need to generate ipset rules dynamically. so i created following script and having error try to get the urls from array.
#!/bin/bash
# urlList
allow_SMTP_OUT_URLS=(mtp.sendgrid.net)
allow_HTTP_OUT_URLS=(archive.mariadb.org)
allow_HTTPS_OUT_URLS=(ppa.launchpad.net repo.mongodb.org www.google.com)
allow_SSH_OUT_URLS=(bitbucket.org)
ipsetNames=(allow_HTTP_OUT allow_HTTPS_OUT allow_SMTP_OUT allow_SSH_OUT)
for ipSET in "${ipsetNames[#]}"
do
ipset create -exist $ipSET hash:ip timeout 86400 comment family inet
chkIPSETexsit="$(ipset list -n | grep $ipSET)"
# Check / Adding IPSET rules
if [ -z "$chkIPSETexsit" ]; then
echo "$ipSET is empty"
exit 1
else
echo "$ipSET is present. Adding URLs to ipset"
urlList=$ipSET
urlList+="_URLS"
echo "urlList: $urlList"
echo URLs: ${(echo $urlList)[#]}
fi
done
its gives error as bad substitution
allow_HTTP_OUT is present. Adding URLs to ipset
urlList: allow_HTTP_OUT_URLS
/root/salt-cron-scripts/test2.sh: line 30: ${(echo $urlList)[#]}: bad substitution
Any suggestions to correct this please
You can use a nameref declared with declare -n. This gives you a slightly better and more intuitive manipulation than using indirect parameter expansion, such as using: ${urls[#]}, ${urls[0]}, ${urls[5]//a/b} etc.
#!/bin/bash
# urlList
allow_SMTP_OUT_URLS=(mtp.sendgrid.net)
allow_HTTP_OUT_URLS=(archive.mariadb.org)
allow_HTTPS_OUT_URLS=(ppa.launchpad.net repo.mongodb.org www.google.com)
allow_SSH_OUT_URLS=(bitbucket.org)
ipsetNames=(allow_HTTP_OUT allow_HTTPS_OUT allow_SMTP_OUT allow_SSH_OUT)
for ipSET in "${ipsetNames[#]}"
do
ipset create -exist $ipSET hash:ip timeout 86400 comment family inet
chkIPSETexsit="$(ipset list -n | grep $ipSET)"
# Check / Adding IPSET rules
if [ -z "$chkIPSETexsit" ]; then
echo "$ipSET is empty"
exit 1
else
echo "$ipSET is present. Adding URLs to ipset"
urlList=${ipSET}_URLS
echo "urlList: $urlList"
declare -n urls=$urlList
echo "URLs: ${urls[#]}"
fi
done
You need to use indirect parameter expansion to expand the array named by urlList.
allow_SMTP_OUT_URLS=(mtp.sendgrid.net)
allow_HTTP_OUT_URLS=(archive.mariadb.org)
allow_HTTPS_OUT_URLS=(ppa.launchpad.net repo.mongodb.org www.google.com)
allow_SSH_OUT_URLS=(bitbucket.org)
ipsetNames=(allow_HTTP_OUT allow_HTTPS_OUT allow_SMTP_OUT allow_SSH_OUT)
for ipSET in "${ipsetNames[#]}"
do
ipset create -exist "$ipSET" hash:ip timeout 86400 comment family inet
chkIPSETexsit="$(ipset list -n | grep $ipSET)"
# Check / Adding IPSET rules
if [ -z "$chkIPSETexsit" ]; then
echo "$ipSET is empty"
exit 1
fi
echo "$ipSET is present. Adding URLs to ipset"
urlList=${ipSET}_URLS
echo "urlList: $urlList"
urlList=$urlList[#]
echo "URLs: ${!urlList}"
done
Related
Extending this question and answer, I'd like some help exploring some solutions to making this exercise of using source to bring a config file into a Bash file more "safely." I say "more safely" because I recognize it may be impossible to do with 100% safety.
I want to use a config file to set variables and arrays and have some comments throughout. Everything else should be disallowed.
The above Q&A suggested starting a regex line to check for things we want, versus what we don't want, before passing it to source.
For example, the regex could be:
(^\s*#|^\s*$|^\s*[a-z_][^[:space:]]*=[^;&\(\`]*$|[a-z_][^[:space:]]*\+?=\([^;&\(\`]*\)$)
But I'm looking for help in both refactoring that regex, or considering other pathways to get what we're after in the Bash script below, especially after wondering if this approach is futile in the first place?
Example
This is what the desired config file would look like:
#!/bin/bash
disks=([0-UUID]=1234567890123 [0-MountPoint]='/some/path/')
disks+=([1-UUID]=4567890123456 [1-MountPoint]='/some/other/path')
# ...
someNumber=1
rsyncExclude=('.*/' '/dev/' '/proc/' '/sys/' '/tmp/' '/mnt/' '/media/' '/lost+found' '.Trash-*/' '[$]RECYCLE.BIN/' '/System Volume Information/' 'pagefile.sys' '/temp/' '/Temp/' '/Adobe/')
remote='this#123.123.123.123'
# there should be nothing in the config more complicated than above
And this is a simplified version of the bash script it will go into, using the example from #Erman in the Q/A linked to above, to do the checking:
#!/bin/bash
configFile='/blah/blah/config.file'
if [[ -f "${configFile}" ]]; then
# check if the config file contains any commands because that is unexpected and unsafe
disallowedSyntax="(^\s*#|^\s*$|^\s*[a-z_][^[:space:]]*=[^;&\(\`]*$|[a-z_][^[:space:]]*\+?=\([^;&\(\`]*\)$)"
if egrep -q -iv "${disallowedSyntax}" "${configFile}"; then
printf "%s\n" 'The configuration file is not safe!' >&2 # print to STDERR
exit 1
else
# config file might be okay
if result=$( bash -n "${configFile}" 2>&1 ); then
# set up the 'disk' associative array first and then import
declare -A disks
source <(awk '/^\s*\w++?=/' "${configFile}")
# ...
else
# config has syntax error
printf '%s\n' 'The configuration file has a syntax error.' >&2
exit 1
fi
fi
else
# config file doesn't exist?
printf '%s\n' "The configuration file doesn't exist." >&2
exit 1
fi
I imagine below is ideally what we want to be allowed and disallowed as a starting point?
Allowed
# whole numbers only
var=1
var=123
# quoted stuff
var='foo bar'
var="foo bar"
# arrays
var=('foo' 'bar')
var=("foo" "bar")
var=([0-foo]=1 [0-bar]='blah' ...
var+=(...
# vars with underscores, same format as above
foo_bar=1
...
foo_bar+=(...
# and that's it?
Not allowed*
* Not an exhaustive list (and I'm certain I'm missing things) but the idea is to at least disallow anything not quoted (unless it's a number), and then also anything else that would allow unleash_virus to be run:
var=notquoted
...
var=notquoted unleash_virus
var=`unleash_virus`
...
var='foo bar' | unleash_virus
...
var="foo bar"; unleash_virus
var="foo bar" && unleash_virus
var="foo bar $(unleash_virus)"
...
At least one issue you might encounter is the ${configFile} changing between the syntax check and the subsequent sourcing:
# configFile might seem save according to your syntax rules:
if egrep -q -iv "${disallowedSyntax}" "${configFile}"; then
printf "%s\n" 'The backup configuration file is not safe!' >&2
exit 1
else
if result=$( bash -n "${configFile}" 2>&1 ); then
declare -A disks
# Warning: config file might have changed
source "${configFile}"
If you cannot guarantee that the contents of the config file remain the same then your regex-check won't help you much.
Since you wanted specific feedback on the regex; Here is a variable assignment with a quoted value that is not allowed by the regex:
some_regex_config='\s'
Note that this was the regex of time of answering:
(^\s*#|^\s*$|^\s*[a-z_][^[:space:]]*=[^;&\(\`]*$|[a-z_][^[:space:]]*\+?=\([^;&\(\`]*\)$)
Here's a start, thanks to #SasaKanjuh.
Instead of checking for disallowed syntax, we could use awk to only pass parts of the config file that match formatting we expect to eval, and nothing else.
For example, we expect that variables must have some kind of quoting (unless they solely contain a number); arrays start and end with () as usual; and everything else should be ignored...
Here's the awk line that does this:
awk '/^\s*\w+\+?=(\(|[0-9]+$|["'\''][^0-9]+)/ && !/(\$\(|&&|;|\||`)/ { print gensub("(.*[\"'\''\\)]).*", "\\1", 1) }' ./example.conf
first part captures line starting with variable name, until =
then after = sign, it is looking for (, numerical value, or ' or " followed by a string
second part excludes lines with $(), &&, ; and |
and gensub captures everything including last occurrence of ' or " or ), ignoring everything after.
#!/bin/bash
configFile='./example.conf'
if [[ -f "${configFile}" ]]; then
# config file exists, check if it has OK bash syntax
if result=$( bash -n "${configFile}" 2>&1 ); then
# seems parsable, import the config file
# filter the contents using `awk` first so we're only accepting vars formatted like this:
# var=1
# var='foo bar'
# var="foo bar"
# var=('array' 'etc')
# var+=('and' "so on")
# and everything else should be ignored:
# var=unquoted
# var='foo bar' | unleash_virus
# var='foo bar'; unleash_virus
# var='foo' && unleash_virus
# var=$(unleash_virus)
# var="$(unleash_virus)"
# ...etc
if config=$(awk '/^\s*\w+\+?=(\(|[0-9]+$|["'\''][^0-9]+)/ && !/(\$\(|&&|;|\||`)/ { print gensub("(.*[\"'\''\\)]).*", "\\1", 1) }' "${configFile}"); then
# something matched
# now actually insert the config data into this session by passing it to `eval`
eval "${config}"
else
# no matches from awk
echo "No config content to work with."
exit 1
fi
else
# config file didn't pass the `bash -n` test
echo "Config contains invalid syntax."
exit 1
fi
else
# config file doesn't exist or isn't a file
echo "There is no config file."
exit 1
fi
How to append return status of a command and loop through the associative array and print the results. It seems to be simple and working with sample code but not working in a function.
I tried quoting the keys and values to push to array but values are not printed. I did checked the return status after appending to array and it is success. However, it is not having the values in the associative array.
The array is not getting the values assigned correctly and hence values are not printed as expected.
The same code is working in my bash version.
#!/bin/bash
declare -A combo
combo+=(['foo']='bar')
combo+=(['hello']='world')
for window in "${!combo[#]}"
do
echo "The key: ${window}" # foo
echo "The value: ${combo[${window}]}" # bar
done
How to capture the return status of 0 when maven build succeeded and 1 when failure and assign to associative array and print the results from associative array after build ran for all the repositories.
#!/bin/bash
declare -A GITARRAY=(
[git_token]=user-devops:ggc4ktalwfbf5jiqdsdhmmgj2jvvhj3ltfzdujxzxnmhj45qk525kq
[git_branch]='development'
)
declare -A GITREPOS=(
[repository1]=org.gitrepo.com/LIBS/_git/repository1
[repository2]=org.gitrepo.com/LIBS/_git/repository2
)
declare -A BUILDSTATUS
target_dir=$HOME/womsrc
BASEDIR=`dirname "$(readlink -f "$0")"`
HELLOTO=`whoami`
maven_goal=compile
#repositories modifed to dummy values
REPOS=(reposiroty1 repository2)
main() {
echo "In main fuction: maven goal: $maven_goal"
if [ -n "$maven_goal" ] ; then
for REPO in "${REPOS[#]}"
do
if [ -d "$target_dir/$REPO/.git" ] ; then
echo "[INFO] invoking compilation for service in $REPO ..."
service_dir="$target_dir/$REPO/$REPO"
build_status=$(buildService $REPO $maven_goal $service_dir)
echo "[STATUS] mvn:$maven_goal $REPO: $build_status"
fi
done
fi ## mavengoal end here
echo "printing build status"
for bt in "${!BUILDSTATUS[#]}"
do
echo "key: ${bt} result: ${BUILDSTATUS[${bt}]}"
done
}
buildService() {
servicename="$1"
mavengoal="$2"
servicedir="$3"
cd "${servicedir}" || echo "[ERROR] cd to $servicename failed with $?"
echo "[INFO] starting compilation in `pwd` ..."
LOG_FILE="$COMPILE_LOGPATH/project-$servicename.log"
mvn $mavengoal -l $LOG_FILE
if [ "$?" -eq 0 ] ; then
buildstatus=success
else
buildstatus=failure
fi
BUILDSTATUS+=([${servicename}]=${buildstatus})
echo "Adding BUILDSTATUS return value: $?"
echo "${BUILDSTATUS[${servicename}]}"
}
main
Your call of the function buildService() via command substitution ($()) is executed in a child process and cannot affect the BUILDSTATUS variable in the parent process, or as the bash(1) manual page puts it:
Command substitution, [...] are invoked in a subshell environment that is a duplicate of the shell environment [...] Changes made to the subshell environment cannot affect the shell's execution environment.
I would suggest outputting only the build status of the current maven call inside buildService() and adding that to BUILDSTATUS in the calling process. You currently have several other echo statements in there. You need to redirect them somewhere else, or they will mess up your assignment to build_status in the calling process.
Im am struggeling with a script, and are newbie to shell scripting.
I am trying to create a script, that will run curl on several websites, from a source file.
arr_values=()
#Getting list of domains/websites to check, into array.
read -ra arr_values <<< $(head -n 999 web.csv)
for s in "${arr_values[#]}"; do
#Running curl on each websites from list.
res=$(curl -s -o /dev/null -w "%{http_code}" $s)
if [ $res == "200" ]; then
echo "OK";
elif
[ $res == "302" ]; then
echo "OK";
else
echo "Error";
fi
done
But i get code 000 when i run without the if statement.
if i run it manually, it all works fine.
And results in a 200 or 302.
Assuming the CSV file is this:
www.yahoo.com
www.google.ca
I.e. one site per line. Also run dos2unix on the file, to ensure it does not have "\r\n" characters at the end of each line.
Then, run this code:
#!/bin/bash
arr_values=()
#Getting list of domains/websites to check, into array.
read -ra arr_values <<< $(head -n 999 web.csv)
for s in "${arr_values[#]}"
do
#Running curl on each websites from list.
res=$(curl -s -o /dev/null -w "%{http_code}" $s)
if [ $res == "200" ]
then
echo "OK $s $res";
elif [ $res == "302" ]
then
echo "OK $s $res";
else
echo "Error $s $res";
fi
done
And got this output:
./t.bash
Error www.yahoo.com 301
OK www.google.ca 200
What was modified:
if conditions are verified using '=='
Modified the echo statements a bit to make it easier to debug
You can use single brackets around the expression for it i.e. [ and ], not [[ and ]]. Also you can remove the asterisk from the value to check i.e. "200", not *"200"*.
I'm trying to write a script that performs actions of files with different extensions. To make it as easy to add different actions as possible, the extensions are read from an array, files are found via the "find" command, and results returned to a dynamically generated array named after the file extension that was searched.
To add a new extension to search for I can simply add to the file_ext array.
I create the array like this:
file_exts=("dsd" "dsdd" "dmesg")
for ext in "${file_exts[#]}"
do
echo "Finding $ext files"
eval var="exts_$ext"
declare -a $var="($( find "$dirs_target" -name "*.$ext"))"
done
The arrays are created correctly, and I can manually echo "${exts_dsd[0]} ${exts_dsd[1]}" and see the entries, However, I can't find a way of looping through each entry in the dynamically assigned arrays.
I have tried a few combinations using eval, and I can print out the first entry in the array, IE just referencing "$exts_dsd" Here are two things I've already tried:
for varname in "${!exts_#}"
do
for entry in ${varname[#]}
do
echo "$varname : $entry"
done
eval value=\$${varname[#]}
echo "$varname=$value"
done
How can I loop through each entry in the above for loop, so I can print out all the entries in all the dynamically created arrays?
Here is a complete test script:
#! /bin/bash
file_exts=("dsd" "dsdd" "dmesg")
dirs_target="/tmp/arraytest/"
echo "Creating $dirs_target"
if [[ ! -d "$dirs_target" ]]; then
if ! mkdir "$dirs_target"; then
echo "Couldn't create temp dir"
exit 1
fi
fi
echo "Creating test files"
for tmpfile in $( seq 0 5 )
do
echo -e "\tCreating $dirs_target$tmpfile.dsd"
if ! touch "$dirs_target/$tmpfile.dsd"; then
echo "Coudn't create $dirs_target/test$tmpfile.dsd"
exit 1
fi
done
echo ""
echo "-----Finding Files-----"
for ext in "${file_exts[#]}"
do
echo "Finding $ext files"
eval var="exts_$ext"
declare -a $var="($( find "$dirs_target" -name "*.$ext"))"
done
echo ""
echo "-----File Extensions-----"
for varname in "${!exts_#}"
do
for entry in ${varname[#]}
do
echo "$varname : $entry"
done
eval value=\$${varname[#]}
#echo "$varname=$value"
done
echo ""
echo "Finishing."
rm -rf "$dirs_target"
To loop over the entries, you have to use the same trick as when creating them: just store the variable name in a variable. The point is to include the [#] index, too, which will be correctly recognized in the indirection:
for varname in "${!exts_#}" ; do
arr=$varname'[#]'
for entry in "${!arr}" ; do
echo "$varname : $entry"
done
done
Also note that eval isn't needed in
# eval var="exts_$ext"
var=exts_$ext # works even better!
I've found the answer. I had the eval statement slightly wrong.
echo "-----File Extensions-----"
for varname in "${!exts_#}"
do
echo "varname=$varname"
eval testvalue="\${$varname[#]}"
for entry in $testvalue
do
echo -e "\tFile: $entry"
done
done
As a bonus, I've also figured out how to add to a dynamically created array
var="table_$entry"
declare -a $var
while read -r line
do
eval $var+=\(\'"$line"\'\)
done < "$dirs_table"
I am currently attempting to create a bash script that will check inside of each users /Library/Mail folder to see if a folder named V2 exists. The script should create an array with each item in the array being a user and then iterate through each of these users checking their home folder for the above captioned contents. This is what I have so far:
#!/bin/bash
cd /Users
array=($(ls))
for i in ${array[#]}
do
if [ -d /$i/Library/Mail/V2 ]
then
echo "$i mail has been upgraded."
else
echo "$i FAIL"
fi
done
Populating your array from the output of ls is going to make for serious problems when you have a username with spaces. Use a glob expression instead. Also, using [ -d $i/... ] will similarly break on names with spaces -- either use [[ -d $i/... ]] (the [[ ]] construct has its own syntax rules and doesn't require quoting) or [ -d "$i/..." ] (with the quotes).
Similarly, you need to double-quote "${array[#]}" to avoid string-splitting from splitting names with spaces in two, as follows:
cd /Users
array=(*)
for i in "${array[#]}"; do
if [[ -d $i/Library/Mail/V2 ]]; then
echo "$i mail has been upgraded."
else
echo "$i FAIL"
fi
done
That said, you don't really need an array here at all:
for i in *; do
...check for $i/Library/Mail/V2...
done