Situation: I have 2 lists.
list_all_process.ini
Listing all of my bash scripts
exclude_proccess.ini
listing all of my "excluding" bash scripts
These contain
list_all_process.ini
script-1.0.sh
script-2.0.sh
script-3.0.sh
script-4.0.sh
-
exclude_proccess.ini
script-3.0.sh
What I would like to have:
If the value doesn't correspond to the exclude proccess -> execute the script. If match -> nothing
My code:
except_proccess(){
readarray a -t < ${INC}/full_proccess_list.ini
readarray b -t < ${INC}/exclude_list.ini
for ACCEPT in ${a[#]} ; do
for DENNY in ${b[#]} ; do
if [[ $ACCEPT == $DENNY ]]; then
continue;
else
bash ${INC}/${ACCEPT}
fi
done
done
My problem:
False positive and ID contain in exclude_list is launched ...
I am wondering why you do it in such a complicated way. Try this:
for p in $(cat list_all_process.ini) ; do
grep -q "$p" exclude_proccess.ini || echo "execute $p"
done
Related
in my usecase i'm filtering certain IPv4s from the list and putting them into array for further tasks:
readarray -t firstarray < <(grep -ni '^ser*' IPbook.file | cut -f 2 -d "-")
As a result the output is:
10.8.61.10
10.0.10.15
172.0.20.30
678.0.0.10
As you see the last row is not an IP, therefore i faced an urge to add some regex check on the FIRSTARRAY.
I do not want to save a collateral files to work with them, so i'm looking for some "on-the-fly" option to regex the firstarray. I tried the following:
for X in "${FIRSTARRAY[#]}"; do
readarray -t SECONDARRAY < <(grep -E '\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.|$)){4}\b' "$X")
done
But in the output I see that system thinks that $X is a file/dir, and didn't process the value, even though it clearly sees it:
line ABC: 172.0.20.30: No such file or directory
line ABC: 678.0.0.10: No such file or directory
What am I doing wrong and what would be the best approach to proceed?
You are passing "$X" as an argument to grep and hence it is being treated as a file. Use herestring <<< instead:
for X in "${firstarray[#]}"; do
readarray -t secondarray < <(grep -E '\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.|$)){4}\b' <<< "$X")
done
You are better off writing a function to validate the IP instead of relying on just a regex match:
#!/bin/bash
validate_ip() {
local arr element
IFS=. read -r -a arr <<< "$1" # convert ip string to array
[[ ${#arr[#]} != 4 ]] && return 1 # doesn't have four parts
for element in "${arr[#]}"; do
[[ $element =~ ^[0-9]+$ ]] || return 1 # non numeric characters found
[[ $element =~ ^0[1-9]+$ ]] || return 1 # 0 not allowed in leading position if followed by other digits, to prevent it from being interpreted as on octal number
((element < 0 || element > 255)) && return 1 # number out of range
done
return 0
}
And then loop through your array:
for x in "${firstarray[#]}"; do
validate_ip "$x" && secondarray+=("$x") # add to second array if element is a valid IP
done
The problem is, that you passing an argument to the grep command and it expects reading standard input instead.
You can use your regex to filter the IP addresses right in the first command:
readarray -t firstarray < <(grep -ni '^ser*' IPbook.file | cut -f 2 -d "-" | grep -E '\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.|$)){4}\b' )
Then you have only IP addresses in firstarray.
I've looked, but have only seen answers to one array being passed in a script.
I want to pass multiple arrays to a bash script that assigns them as individual variables as follows:
./myScript.sh ${array1[#]} ${array2[#]} ${array3[#]}
such that: var1=array1 and var2=array2 and var3=array3
I've tried multiple options, but doing variableName=("$#") combines all arrays together into each variable. I hope to have in my bash script a variable that represents each array.
The shell passes a single argument vector (that is to say, a simple C array of strings) off to a program being run. This is an OS-level limitation: There exists no method to pass structured data between two programs (any two programs, written in any language!) in an argument list, except by encoding that structure in the contents of the members of this array of C strings.
Approach: Length Prefixes
If efficiency is a goal (both in terms of ease-of-parsing and amount of space used out of the ARG_MAX limit on command-line and environment storage), one approach to consider is prefixing each array with an argument describing its length.
By providing length arguments, however, you can indicate which sections of that argument list are supposed to be part of a given array:
./myScript \
"${#array1[#]}" "${array1[#]}" \
"${#array2[#]}" "${array2[#]}" \
"${#array3[#]}" "${array3[#]}"
...then, inside the script, you can use the length arguments to split content back into arrays:
#!/usr/bin/env bash
array1=( "${#:2:$1}" ); shift "$(( $1 + 1 ))"
array2=( "${#:2:$1}" ); shift "$(( $1 + 1 ))"
array3=( "${#:2:$1}" ); shift "$(( $1 + 1 ))"
declare -p array1 array2 array3
If run as ./myScript 3 a b c 2 X Y 1 z, this has the output:
declare -a array1='([0]="a" [1]="b" [2]="c")'
declare -a array2='([0]="X" [1]="Y")'
declare -a array3='([0]="z")'
Approach: Per-Argument Array Name Prefixes
Incidentally, a practice common in the Python world (particularly with users of the argparse library) is to allow an argument to be passed more than once to amend to a given array. In shell, this would look like:
./myScript \
"${array1[#]/#/--array1=}" \
"${array2[#]/#/--array2=}" \
"${array3[#]/#/--array3=}"
and then the code to parse it might look like:
#!/usr/bin/env bash
declare -a args array1 array2 array3
while (( $# )); do
case $1 in
--array1=*) array1+=( "${1#*=}" );;
--array2=*) array2+=( "${1#*=}" );;
--array3=*) array3+=( "${1#*=}" );;
*) args+=( "$1" );;
esac
shift
done
Thus, if your original value were array1=( one two three ) array2=( aye bee ) array3=( "hello world" ), the calling convention would be:
./myScript --array1=one --array1=two --array1=three \
--array2=aye --array2=bee \
--array3="hello world"
Approach: NUL-Delimited Streams
Another approach is to pass a filename for each array from which a NUL-delimited list of its contents can be read. One chief advantage of this approach is that the size of array contents does not count against ARG_MAX, the OS-enforced command-line length limit. Moreover, with an operating system where such is available, the below does not create real on-disk files but instead creates /dev/fd-style links to FIFOs written to by subshells writing the contents of each array.
./myScript \
<( (( ${#array1[#]} )) && printf '%s\0' "${array1[#]}") \
<( (( ${#array2[#]} )) && printf '%s\0' "${array2[#]}") \
<( (( ${#array3[#]} )) && printf '%s\0' "${array3[#]}")
...and, to read (with bash 4.4 or newer, providing mapfile -d):
#!/usr/bin/env bash
mapfile -d '' array1 <"$1"
mapfile -d '' array2 <"$2"
mapfile -d '' array3 <"$3"
...or, to support older bash releases:
#!/usr/bin/env bash
declare -a array1 array2 array3
while IFS= read -r -d '' entry; do array1+=( "$entry" ); done <"$1"
while IFS= read -r -d '' entry; do array2+=( "$entry" ); done <"$2"
while IFS= read -r -d '' entry; do array3+=( "$entry" ); done <"$3"
Charles Duffy's response works perfectly well, but I would go about it a different way that makes it simpler to initialize var1, var2 and var3 in your script:
./myScript.sh "${#array1[#]} ${#array2[#]} ${#array3[#]}" \
"${array1[#]}" "${array2[#]}" "${array3[#]}"
Then in myScript.sh
#!/bin/bash
declare -ai lens=($1);
declare -a var1=("${#:2:lens[0]}") var2=("${#:2+lens[0]:lens[1]}") var3=("${#:2+lens[0]+lens[1]:lens[2]}");
Edit: Since Charles has simplified his solution, it is probably a better and more clear solution than mine.
Here is a code sample, which shows how to pass 2 arrays to a function. There is nothing more than in previous answers except it provides a full code example.
This is coded in bash 4.4.12, i.e. after bash 4.3 which would require a different coding approach. One array contains the texts to be colorized, and the other array contains the colors to be used for each of the text elements :
function cecho_multitext () {
# usage : cecho_multitext message_array color_array
# what it does : Multiple Colored-echo.
local -n array_msgs=$1
local -n array_colors=$2
# printf '1: %q\n' "${array_msgs[#]}"
# printf '2: %q\n' "${array_colors[#]}"
local i=0
local coloredstring=""
local normalcoloredstring=""
# check array counts
# echo "msg size : "${#array_msgs[#]}
# echo "col size : "${#array_colors[#]}
[[ "${#array_msgs[#]}" -ne "${#array_colors[#]}" ]] && exit 2
# build the colored string
for msg in "${array_msgs[#]}"
do
color=${array_colors[$i]}
coloredstring="$coloredstring $color $msg "
normalcoloredstring="$normalcoloredstring $msg"
# echo -e "coloredstring ($i): $coloredstring"
i=$((i+1))
done
# DEBUG
# echo -e "colored string : $coloredstring"
# echo -e "normal color string : $normal $normalcoloredstring"
# use either echo or printf as follows :
# echo -e "$coloredstring"
printf '%b\n' "${coloredstring}"
return
}
Calling the function :
#!/bin/bash
green='\E[32m'
cyan='\E[36m'
white='\E[37m'
normal=$(tput sgr0)
declare -a text=("one" "two" "three" )
declare -a color=("$white" "$green" "$cyan")
cecho_multitext text color
Job done :-)
I do prefer using base64 to encode and decode arrays like:
encode_array(){
local array=($#)
echo -n "${array[#]}" | base64
}
decode_array(){
echo -n "$#" | base64 -d
}
some_func(){
local arr1=($(decode_array $1))
local arr2=($(decode_array $2))
local arr3=($(decode_array $3))
echo arr1 has ${#arr1[#]} items, the second item is ${arr1[2]}
echo arr2 has ${#arr2[#]} items, the third item is ${arr2[3]}
echo arr3 has ${#arr3[#]} items, the here the contents ${arr3[#]}
}
a1=(ab cd ef)
a2=(gh ij kl nm)
a3=(op ql)
some_func "$(encode_array "${a1[#]}")" "$(encode_array "${a2[#]}")" "$(encode_array "${a3[#]}")"
The output is
arr1 has 3 items, the second item is cd
arr2 has 4 items, the third item is kl
arr3 has 2 items, the here the contents op ql
Anyway, that will not work with values that have tabs or spaces. If required, we need a more elaborated solution. something like:
encode_array()
{
for item in "$#";
do
echo -n "$item" | base64
done | paste -s -d , -
}
decode_array()
{
local IFS=$'\2'
local -a arr=($(echo "$1" | tr , "\n" |
while read encoded_array_item;
do
echo "$encoded_array_item" | base64 -d;
echo "$IFS"
done))
echo "${arr[*]}";
}
test_arrays_step1()
{
local IFS=$'\2'
local -a arr1=($(decode_array $1))
local -a arr2=($(decode_array $2))
local -a arr3=($(decode_array $3))
unset IFS
echo arr1 has ${#arr1[#]} items, the second item is ${arr1[1]}
echo arr2 has ${#arr2[#]} items, the third item is ${arr2[2]}
echo arr3 has ${#arr3[#]} items, the here the contents ${arr3[#]}
}
test_arrays()
{
local a1_2="$(echo -en "c\td")";
local a1=("a b" "$a1_2" "e f");
local a2=(gh ij kl nm);
local a3=(op ql );
a1_size=${#a1[#])};
resp=$(test_arrays_step1 "$(encode_array "${a1[#]}")" "$(encode_array "${a2[#]}")" "$(encode_array "${a3[#]}")");
echo -e "$resp" | grep arr1 | grep "arr1 has $a1_size, the second item is $a1_2" || echo but it should have only $a1_size items, with the second item as $a1_2
echo "$resp"
}
Based on the answers to this question you could try the following.
Define the arrays as variable on the shell:
array1=(1 2 3)
array2=(3 4 5)
array3=(6 7 8)
Have a script like this:
arg1=("${!1}")
arg2=("${!2}")
arg3=("${!3}")
echo "arg1 array=${arg1[#]}"
echo "arg1 #elem=${#arg1[#]}"
echo "arg2 array=${arg2[#]}"
echo "arg2 #elem=${#arg2[#]}"
echo "arg3 array=${arg3[#]}"
echo "arg3 #elem=${#arg3[#]}"
And call it like this:
. ./test.sh "array1[#]" "array2[#]" "array3[#]"
Note that the script will need to be sourced (. or source) so that it is executed in the current shell environment and not a sub shell.
I'm trying to write a small script that will take the 4th columns of a file and store it in an array then do a little comparison. If the element in the array is greater than 0 and less than 500 I have to increment the counter. However when I run the script the counter always shows 0. Here's my script
#!/bin/bash
mapfile -t my_array < <(cat file1.txt | awk '{ print $4 }' > test.txt)
COUNTER=0
for i in ${my_array[#]}; do
if [["${my_array[$i]}" -gt 0 -a "${my_array[$i]}" -lt 500 ]]
then
COUNTER=$((COUNTER + 1))
fi
printf "%s\t%s\n" "%i" "${my_array[$i]}"//just to test if the mapfile command is working
done
echo $COUNTER
output:
./script1.bash
0
#!/bin/bash
mapfile -t my_array < <(awk '{ print $4 }' file1.txt | tee test.txt)
COUNTER=0
for idx in "${!my_array[#]}"; do
value=${my_array[$idx]}
if (( value > 0 )) && (( value < 500 )); then
COUNTER=$((COUNTER + 1))
fi
printf "%s\t%s\n" "$idx" "$value"
done
echo "$COUNTER"
The use of cat here is needless: It added nothing but inefficiency (requiring an extra process to be started, and forcing awk to read from a pipe rather than direct from a file).
mapfile had nothing to read because the output of awk was redirected to test.txt. If you want it to go to both a file and stdout, then you need to use tee.
-a is not valid in [[ ]]; use && instead there. However, since you're doing only arithmetic, (( )) is more appropriate. Incidentally, -a is officially marked obsolescent even for [ ] and test; see the current POSIX standard.
${my_array[#]} iterates over values. If you want to iterate over indexes, you need ${!my_array[#]} instead.
Whitespace is mandatory in separating command names. [["$foo" is a different command from [[, unless $foo is empty or starts with a character in $IFS.
If you redirect the output to a file: > test.txt then there is no output in "standard output" because it is consumed by the file. So, first, you need to remove that redirection. You may use:
mapfile -t my_array < <(cat file1.txt | awk '{ print $4 }' )
But since awk could perfectly well read a file, this is better:
mapfile -t my_array < <(awk '{ print $4 }' file1.txt)
And since you are using awk, it could do the comparison to 0 and 500 and output the whole count.
counter=$(awk '{if($4>0 && $4<500){c++}}END{print c}' file1.txt)
echo "$counter"
Simpler, faster.
That will also avoid some simple mistakes in your script, like missing an space in the […] construct:
if [[ "${my … # NOT "if [["${my …"
And some missing quotes:
for i in "${my_array[#]}" # NOT for i in ${my_array[#]}
In general, it is a good idea to check your script with ShellCheck.net to remove some simple mistakes.
Before you get concerned, this (hopefully) isn't another of the many threads about having variable access problems from a while loop in BASH. I looked at those and implemented the redirect method, but I'm having some issues.
Code:
#!/bin/bash
rm /tmp/sched.now
rm /tmp/sched.later
NOWURL="schedNOWurl"
LATERURL="schedLATERurl"
curl -s $NOWURL --output /tmp/sched.now
curl -s $LATERURL --output /tmp/sched.later
re='^[0-9]+$'
if grep -q 'no shifts' /tmp/sched.now ; then
nowpeople[0]="No Shifts Assigned For This Time"
nowtime[0]="Now"
else
while read line
do
if [[ ${line:0:1} =~ $re ]] ; then
nowtime[$nowcount]=$line
((nowcount++))
elif [[ ${line:0:1} == '-' ]] ; then
nowpeople[$nowcount]=$line
((nowcount++))
fi
done < /tmp/sched.now
fi
declare -a latertime
declare -a laterpeople
if grep -q 'no shifts' /tmp/sched.later ; then
laterpeople[0]="No Shifts Assigned For This Time"
latertime[0]="Until 23:59"
else
while read line
do
if [[ ${line:0:1} =~ $re ]] ; then
latertime[$latercount]=$line
((latercount++))
elif [[ ${line:0:1} == '-' ]] ; then
laterpeople[$latercount]=$line
((latercount++))
fi
done < /tmp/sched.later
fi
echo ${nowpeople[#]}
echo ${nowtime[#]}
echo ${laterpeople[#]}
echo ${latertime[#]}
The URLs that I'm getting are HTML files. The text I'm attempting to pull in the loops looks something like this
11a-1:30p
- John Doe
with some HTML thrown above and below it. Note that these are all on their own lines.
The issue I'm facing is that the loops aren't saving all of the matches to the array. In the second loop, it should pull three entries into each array. Right now it pulls one with malformed text as follows:
- John Doe
1:30p-4pp
Notice the missing m in pm and the replaced m with a p in pp.
The strange thing is that if I echo $line before the array statement, the line is pulled from the file fine, so it sounds like it's something in the way I'm saving to the array.
Suggestions? Please let me know if you need more details.
I have a flat file which contains the following
INDIA USA SA NZ AUS ARG GER BRA
so there are eight columns altogether . Now I want to store the indexes of those columns only which starts with A into an array.
For that I tried the following statement
awk '{for(i=1;i<=NF;i++){if($i~/^A/){set -A newArray $i}}}' testUnix.txt
when I echo the file using
echo "${newArray[*]}"
it's printing 5 6 but whenever I am trying to get the length of that array
echo ${#newArray[#]}
its length is being shown as 1 only. Should not it be 2 ?
I also tried
awk '{y = 0;for(i=1;i<=NF;i++){if($i~/^A/){newArray[y] = $i ; y++}}}' testUnix.txt
but also it's producing the same result.
What am I missing ?Please explain. I intend to get the desired output 2.
No need for awk. You can loop through the elements and check if they start with A:
r="INDIA USA SA NZ AUS ARG GER BRA"
arr=()
for w in $r
do
[[ $w == A* ]] && arr+=("$w")
done
If you execute it then the arr array contains:
$ for i in "${arr[#]}"; do echo "$i"; done
AUS
ARG
And to confirm that is has two elements, let's count them:
$ echo "${#arr[#]}"
2
What is happening with your aproach?
awk '{for(i=1;i<=NF;i++){if($i~/^A/){set -A newArray $i}}}' testUnix.txt
this says set -A newArray but it is not really defining the variable in bash, because you are in awk.
What I would do to have a bash array :
bash_arr=( $(awk '{for(i=1;i<=NF;i++){if($i~/^A/){print $i}}}' file) )
echo "${bash_arr[#]}"
AUS ARG
And you don't even need awk in reality, bash is capable of doing regex :
for word in $(<file); do [[ $word =~ ^A ]] && basharr+=( "$word" ); done