Appending to elements in a Bash associative array - arrays

My name is Mike and I am a total noob at Bash programming. With that being said, please don't chew me to a pulp. Here's my scenario. I'm importing a csv and processing it line by line. It looks up information in AD and pipes that result to a variable $ADresult1.
I then take that output and send it into the array.
declare -A arrmail
arrmail[$ADresult1]="$v1, $v3"
appendvar+="; ${arrmail[$ADresult1]}"
if [ ! -z "${arrmail[$ADresult1]}" ]; then
echo ${arrmail[#]}
else
echo ${arrmail[#]}="$appendvar"
fi
Now in my CSV file, the var $ADresult1 has more than one entry that is the same, however each element should be different.
For example: The first line might contain arrmail[p100]=Y2K, Y2J
The second line might arrmail[p100]=GGG, GG1
My question would be how to combine the elements from the first line and second line into one line of output and/or if this is possible.
For instance arrmail[p100]=Y2K, Y2J; GGG, GG1
If needs be, I can post the whole entire script for a bigger picture.
Here's the script.
#!/bin/bash
###--------------------------------------------------------------------------
# Author : Volha Zhdanava (p146819)
# Date : 11-FEB-2013
# Purpose: To automate the process of sending out notification emails to developers when
# Assigned DEV Machine is on the Expired Devices List.
#----------------------------------------------------------------------------
# ----MODIFIED----
#----------------------------------------------------------------------------------------------------
# Author Revsn Date Revsn ID Revsn Purpose
# Mike Bell 02-28-14 Rev.03 Add ability to email j or p number
# Mike Bell 01-02-15 Rev.04 Add verbiage about 90 day licensing
###----------------------------------------------------------------------------------------------------
Purple='\e[1;35m'
ColorOFF='\e[0m'
TO_TEAM="dev-tools#edwardjones.com"
#-----------------------------------------
#Ask for input CSV file
#-----------------------------------------
FILEDIR="/export/share/is/dev_env/expdev"
FILENAME="expired_devices.csv"
#if the directory is not empty, find the csv file
if [[ $(ls -A $FILEDIR) ]]
then
#check if the csv file was stored with default name <expired_devices.csv>
if [ -f $FILEDIR/$FILENAME ]
then
#absolute path to the downloaded file
CSVFILE=`echo $FILEDIR/$FILENAME`
#if the csv file was stored under a user-defined name,
#re-define the absolute path to that file
else
#count how many files are saved in the expdev directory
NAME=( `ls -A $FILEDIR` )
CNT=${#NAME[#]}
echo -e "There is/are ${Purple}${CNT}${ColorOFF} file(s) in the ${Purple}$FILEDIR${ColorOFF} directory:"
echo ${NAME[#]}
#if there is only one file saved within the expdev directory
if [ ${CNT} -eq 1 ]
then
FILENAME=${NAME[0]}
else
#it will prompt to enter the name of the file
REPLY="n"
while [ "$REPLY" != "y" ]
do
echo -e "Please enter the full name of the downloaded csv file: \c"
read FILENAME
echo -e "You entered ${Purple}$FILENAME${ColorOFF}. Is this correct? y/n: \c\n"
read ANSWER
REPLY=`echo $ANSWER | tr [:upper:] [:lower:]`
done
fi
#verify the specified file exists in <expdev> directory
echo `ls -Al $FILEDIR/$FILENAME`
echo -e "\nThe script will run against the listed above ${Purple}$FILENAME${ColorOFF} file.\n${Purple}NOTE${ColorOFF}: Do NOT proceed if it says ${Purple}No such file or directory${ColorOFF}.\nWould you like to proceed? y/n -->:\c "
read PROCEED
PROCEED=`echo $PROCEED | tr [:upper:] [:lower:]`
if [ "$PROCEED" == "y" ]
#user agrees
then
CSVFILE=`echo $FILEDIR/$FILENAME`
else
echo -e "Make sure you saved your CSV file in ${Purple}$FILEDIR${ColorOFF} directory and run the script again."
exit 1
fi
fi
#----------------------------------------------
#Remove header from CSVFILE
#and create a temp csvfile.txt
#for "WHILE READ" loop
#----------------------------------------------
Nrows=`cat ${CSVFILE}| wc -l`
N=`expr $Nrows - 1`
tail -$N ${CSVFILE} > csvfile.txt
#----------------------------------------------
#Determine the Extension Date
#----------------------------------------------
TODAYsDATE=`date +"%a"`
case $TODAYsDATE in
Mon ) ExtDate=`date --date='4 day' +"%A, %d-%b-%Y"`
;;
Tue ) ExtDate=`date --date='6 day' +"%A, %d-%b-%Y"`
;;
Wed ) ExtDate=`date --date='6 day' +"%A, %d-%b-%Y"`
;;
Thu ) ExtDate=`date --date='6 day' +"%A, %d-%b-%Y"`
;;
Fri ) ExtDate=`date --date='6 day' +"%A, %d-%b-%Y"`
;;
* ) echo "Cannot determine the Extention Date."
REPLY="n"
while [ "$REPLY" != "y" ]
do
echo -e "Please enter Extended Date: \c"
read ExtDate
echo -e "You entered ${Purple}$ExtDate${ColorOFF}. Is this correct? y/n: \c "
read ANSWER
REPLY=`echo $ANSWER | tr [:upper:] [:lower:]`
done
esac
echo -e "The Extended Date for the Application Owners will be ${Purple}$ExtDate${ColorOFF}.\nPlease make a note for End Date in the APEX Workstation Tracking Tool."
#create a temp confirmation file that will be sent to dev-tools
echo -e "\nThe Extended Date is $ExtDate.\nSent emails to: " > confirmation.txt
#----------------------------------------------
#Read the csvfile.txt
#----------------------------------------------
while IFS=',' read v1 v2 v3 v4 other
do
#----------------------------------------------
#Determine email address
#----------------------------------------------
v2=` echo $v2 | tr '"/,\&' ' ' `
NAME=( ${v2} )
cnt=${#NAME[#]}
#-----------------------------------------------
# Filter j or p<number> userids from table V2
#--------------------------------------------------
if [[ "$NAME" =~ [P|p][[:digit:]]{6}$|[J|j][[:digit:]]{5}$ ]]
then
ADresult1=`getent passwd "$NAME" | awk -F":" '{print $1}'`
echo ${ADresult1}
#------------------------------------------------------------------------------------------------------
# Processing Array
#-----------------------------------------------------------------------------------------------------
declare -A arrmail
arrmail[$ADresult1]="$v1, $v3"
appendvar+="; ${arrmail[$ADresult1]}"
if [ ! -z "${arrmail[$ADresult1]}" ]
then
echo ${arrmail[#]}
else
echo ${arrmail[#]}="$appendvar"
fi
#--------------------------------------------------------------------------------------
# If query matches j or p#, then email user
#------------------------------------------------------------------------------------------
#
#
# v1=` echo $v1 | tr '"' ' ' `
# SUBJECT=" ${v1}, ${v3} - Your Development Machine Status Update"
#
# TO="${ADresult1}#edwardjones.com"
#
# v4=` echo $v4 | tr '"' ' ' `
#
# devmailmessage="\nGreetings ${NAME[0]}, \n\nI am in the process of cleaning up the machines in our lab.\n\nAssigned device: ${v1} \n\nEffort: ${v3}\n\nThe above machine have been assigned to you since ${v4}.\n\nDo you still need this machine or can I reclaim it?\nIf this machine is still required, please let us know how much longer you will need it.\nPlease keep in mind that Dev. workstations or laptops can only be assigned for 90 days due to licensing restrictions.\n\nIf this machine is no longer needed, please let us know also.\n\nThis machine will be rebuilt on or after $ExtDate.\n\nPlease reply to dev-tools#edwardjones.com by $ExtDate.\n\n\nThank you,\n\nDevelopment Environment Support team\ndev-tools#edwardjones.com "
#
#echo -e ${devmailmessage}|mail -s "$SUBJECT" $TO -- -f $TO_TEAM
#--------------------------------------------------------------------------------------------
# If query does not match j or p number, then lookup username via first and last name
#----------------------------------------------------------------------------------------------
elif [ ${cnt} -eq 2 ]
then
ADresult=`getent passwd | grep -i "${NAME[0]}" | grep -i "${NAME[1]}"`
echo ${ADresult}
#count search results from Active Directory; only email if there is one unique result
ADsearch1=`echo ${ADresult} | grep -i "${NAME[0]}" | wc -l`
ADsearch2=`echo ${ADresult} | grep -i "${NAME[1]}" | wc -l`
if [ ${ADsearch1} -eq 1 ] && [ ${ADsearch2} -eq 1 ]
then
pNumber=`echo ${ADresult} | awk -F":" '{print $1}'`
#------------------------------------------------------------------------------------------------------
# Processing Array
#-----------------------------------------------------------------------------------------------------
#declare -A arrmail
#arrmail[$pNumber]="$v1, $v3"
#appendvar+="; ${arrmail[$pNumber]}"
#if [ ! -z "${arrmail[$pNumber]}" ]
# then
# {arrmail[$pNumber]}="${arrmail[$ADresult1]}"
#else
# {arrmail[$ADresult1]}="$appendvar"
#fi
#------------------------------------------------------------------------------------------------------------
# v1=` echo $v1 | tr '"' ' ' `
# SUBJECT=" ${v1}, ${v3} - Your Development Machine Status Update"
# TO="${pNumber}#edwardjones.com"
# v4=` echo $v4 | tr '"' ' ' `
# devmailmessage="\nGreetings ${NAME[0]}, \n\nI am in the process of cleaning up the machines in our lab.\n\nAssigned device: ${v1} \n\nEffort: ${v3}\n\nThe above machine have been assigned to you since ${v4}.\n\nDo you still need this machine or can I reclaim it?\nIf this machine is still required, please let us know how much longer you will need it.\nPlease keep in mind that Dev. workstations or laptops can only be assigned for 90 days due to licensing restrictions.\n\nIf this machine is no longer needed, please let us know also.\n\nThis machine will be rebuilt on or after $ExtDate.\n\nPlease reply to dev-tools#edwardjones.com by $ExtDate.\n\n\nThank you,\n\nDevelopment Environment Support team\ndev-tools#edwardjones.com "
# echo -e ${devmailmessage}| mail -s "$SUBJECT" $TO -- -f $TO_TEAM
# else
# TO=`echo "ACTION REQUIRED: Email address cannot be determined for the following user: ${v2}, with assigned device: ${v1}. Please email this user yourself."`
fi
# else
# TO=`echo "ACTION REQUIRED: Email address cannot be determined for the following user: ${v2}, with assigned device: ${v1}. Please email this user yourself."`
fi
#echo -e "\n$TO ${v2}, ${v1}" >> confirmation.txt
echo "NEXT ROW"
done < csvfile.txt
#----------------------------------------------
#sending out confirmation email to dev-tools
#----------------------------------------------
#subject="WW - Notification Emails to Developers - Status"
#cat confirmation.txt | mail -s "$subject" $TO_TEAM -- -f $TO_TEAM
#echo -e "Result: Processing is COMPLETED. Please check the ${Purple}$subject${ColorOFF} email. Note: ${Purple}$FILENAME${ColorOFF} file was deleted from ${Purple}$FILEDIR${ColorOFF} directory to avoid duplicates in the future."
rm ${CSVFILE}
rm csvfile.txt
rm confirmation.txt
else
#----------------------------------------------------
#if the directory is empty
#------------------------------------------------------
echo -e "Please make sure the CSV file was downloaded to ${Purple}$FILEDIR${ColorOFF} directory and run the script again."
fi
###
#--------THE END---------------------------------
###
Here's some of the output of the script.
IFS=,
read v1 v2 v3 v4 other
++ echo p098650
++ tr '"/,\&' ' '
v2=p098650
NAME=(${v2})
cnt=1
[[ p098650 =~ [P|p][[:digit:]]{6}$|[J|j][[:digit:]]{5}$ ]]
++ getent passwd p098650
++ awk -F: '{print $1}'
ADresult1=p098650
echo p098650
p098650
declare -A arrmail
arrmail[$ADresult1]='CNU327B1Y3, EDGARLITE_0340_COR_00'
appendvar+='; CNU327B1Y3, EDGARLITE_0340_COR_00'
'[' '!' -z 'CNU327B1Y3, EDGARLITE_0340_COR_00' ']'
echo CNU327B1Y3, EDGARLITE_0340_COR_00
CNU327B1Y3, EDGARLITE_0340_COR_00
echo 'NEXT ROW'
NEXT ROW
IFS=,
read v1 v2 v3 v4 other
++ echo p098650
++ tr '"/,\&' ' '
v2=p098650
NAME=(${v2})
cnt=1
[[ p098650 =~ [P|p][[:digit:]]{6}$|[J|j][[:digit:]]{5}$ ]]
++ getent passwd p098650
++ awk -F: '{print $1}'
ADresult1=p098650
echo p098650
p098650
declare -A arrmail
arrmail[$ADresult1]='CNU327B1GP, BUZZLITE_0340_COR_00'
appendvar+='; CNU327B1GP, BUZZLITE_0340_COR_00'
'[' '!' -z 'CNU327B1GP, BUZZLITE_0340_COR_00' ']'
echo CNU327B1GP, BUZZLITE_0340_COR_00
CNU327B1GP, BUZZLITE_0340_COR_00
echo 'NEXT ROW'
NEXT ROW
What's I'm expecting to see is, arrmail[$ADresult1]='CNU327B1Y3, EDGARLITE_0340_COR_00';'CNU327B1GP, BUZZLITE_0340_COR_00'
I couldn't include a screenshot of the CSV, so I'm just going to cut and paste some of it.
Sernum $V1 Username $V2 Effort $V3 Startdate $V4
CNU327B1Y3 p098650 EDGARLITE_0340_COR_00 4-Mar-14
CNU3199N31 Bell SDLCONTENTPORTER_2013_COR_00 10-Mar-14
CNU327B1GP p098650 BUZZLITE_0340_COR_00 4-Mar-14
CNU2479GLB Mike XENDESKAGENTX64_0640_COR_00 28-Feb-14
CNU327B1PB Mike Bell BONDONEUOBRA_2001_COR_00 6-Mar-14
2UA24705YY Bell Mike SASENTGUIDEX64_0610_COR_00 25-Nov-13
CNU2479K7Z Bell, Mike Software Testing 12-Dec-13
2UA24705ZZ Bell Mike TESTGUIDEX64_0610_COR_00 25-Nov-13
I know this is a lot. I've been working on this for a long time and I'm trying to gain a greater understanding of how to use and manipulate arrays. Also if you have an recommendations on books covering the subject, I'll be happy to review those also. Thank you in advance for all the help.

Related

Issue using diff with array and value quoted SHELL [duplicate]

This question already has answers here:
How can I store the "find" command results as an array in Bash
(8 answers)
Closed 2 months ago.
Hi guys i'm having an issue while using diff.
In my script i'm trying to compare all files in 1 dir to all files in 2 other dir
Using diff to compare is files are the same.
Here is my script :
`
#!/bin/bash
files1=()
files2=()
# Directories to compare. Adding quotes at the begining and at the end of each files found in content1 & content3
content2=$(find /data/logs -name "*.log" -type f)
content1=$(find /data/other/logs1 -type f | sed 's/^/"/g' | sed 's/$/"/g')
content3=$(find /data/other/logs2 -type f | sed 's/^/"/g' | sed 's/$/"/g')
# ADDING CONTENT INTO FILES1 & FILES2 ARRAY
while read -r line; do
files1+=("$line")
done <<< "$content1"
# content1 and content3 goes into the same array
while read -r line3;do
files1+=("$line3")
done <<< "$content3"
while read -r line2; do
files2+=("$line2")
done <<< "$content2"
# Here i'm trying to compare 1 by 1 the files in files2 to all files1
for ((i=0; i<${#files2[#]}; i++))
do
for ((j=0; j<${#files1[#]}; j++))
do
if [[ -n ${files2[$i]} ]];then
diff -s "${files2[$i]}" "${files1[$j]}" > /dev/null
if [[ $? == 0 ]]; then
echo ${files1[$j]} "est identique a" ${files2[$i]}
unset 'files2[$i]'
break
fi
fi
done
done
#SHOW THE FILES WHO DIDN'T MATCHED
echo ${files2[#]}
`
I'm having the folling issue when i'm trying to diff :
diff: "/data/content3/other/log2/perso log/somelog.log": No such file or directory
But when i'm doing
ll "/data/content3/other/log2/perso log/somelog.log" -rw-rw-r-- 2 lopom lopom 551M 30 oct. 18:53 '/data/content3/other/logs2/perso log/somelog.log'
So the file exist.
i need those quotes because sometimes there are space in the path
Does some1 know how to fix that ?
Thanks.
I already tried to change the quotes by single quotes, but it didn't fixed it
First, don't do this -
content2=$(find /data/logs -name "*.log" -type f)
content1=$(find /data/other/logs1 -type f | sed 's/^/"/g' | sed 's/$/"/g')
content3=$(find /data/other/logs2 -type f | sed 's/^/"/g' | sed 's/$/"/g')
don't stack all these into single vars. This is asking for ten kinds of obscure trouble. More importantly, those sed calls are embedding the quotation marks into the data as part of the filenames, which is probably what's causing diff to crash, because there are no actual files with the quotes in the name.
Also, if you are throwing away the output and just using diff to check the files are identical, try cmp instead. The -s is silent, and it's a lot faster since it exits at the first differing byte without reading the rest of both files and generating a report. If there ae a lot of files, this will add up.
If the logs are the only things in the directories, and you don't have to scan subdirectoies, and the filename can't appear in both /data/other/logs1 AND /data/other/logs2, but you're pretty sure it will be in at least one of them... then simplify:
for f in /data/logs/*.log # I'll assume these are all files...
do t=/data/other/logs[12]/"${f#/data/logs/}" # always just one?
if cmp -s "$f" "$t" # cmp -s *has* no output
then echo "$t est identique a $f" # files are same
elif [[ -e "$t" ]] # check t exists
then echo "$t diffère de $f" # maybe ls -l "$f" "$t" ?
else echo "$t n'existe pas" # report it does not
fi
done
This needs no arrays, no find, no sed calls, etc.
If you do need to read subdirectories, use shopt to handle it with globs so that you don't have to worry about parsing odd characters with read. (c.f. https://mywiki.wooledge.org/ParsingLs for some reasons.)
shopt -s globstar
for f in /data/logs/**/*.log # globstar makes ** match at arbitrary depth
do for t in /data/other/logs[12]/**/"${f#/data/logs/}" # if >1 possible hit
do if cmp -s "$f" "$t"
then echo "$t est identique a $f"
elif [[ -e "$t" ]]
then echo "$t diffère de $f"
else echo "$t n'existe pas" # $t will be the glob, one iteration
fi
done
done

Create an array with device logical name based on list of serial numbers

I'm tying to identify disks based on a list of UUID I get out of an api call.
The list of devices on the linux VM is following:
NAME="sda" SERIAL="NUTANIX_NFS_5_0_70509_a07b6add_60c9_4758_8b27_0afe77820dbd"
NAME="sdb" SERIAL="NUTANIX_NFS_18_0_625_ae748138_9bc0_4499_8068_d00a84a800b6"
NAME="sdc" SERIAL="NUTANIX_NFS_18_0_626_8a082956_a2be_42ca_a14b_7d21ed3d5a2d"
NAME="sdd" SERIAL="NUTANIX_NFS_18_0_627_6185353f_5c13_47ab_af58_b72d4d07106b"
NAME="sde" SERIAL="NUTANIX_NFS_18_0_628_fbeed366_6956_4de1_a217_487d75fd003c"
NAME="sdf" SERIAL="NUTANIX_NFS_18_0_629_019dcefb_0e1b_4086_96a9_982ae52fd88a"
NAME="sdg" SERIAL="NUTANIX_NFS_18_0_630_5d0b55e8_d0bd_4ee4_a2be_0bef88d94732"
NAME="sdh" SERIAL="NUTANIX_NFS_18_0_631_823beaed_1c2e_4203_ab5f_be294382d0c5"
NAME="sdi" SERIAL="NUTANIX_NFS_18_0_632_e4ee8620_5b53_4187_8f8c_7123ee8d3486"
NAME="sdj" SERIAL="NUTANIX_NFS_18_0_633_41551a1b_7dd9_4fd4_95bc_1fcdebb9dc85"
NAME="sdk" SERIAL="NUTANIX_NFS_18_0_634_12d517fc_e726_4512_b176_5d4075d0ecfe"
NAME="sdl" SERIAL="NUTANIX_NFS_18_0_635_d682eee4_cec0_4809_9efe_97aec7bf15d0"
NAME="sdm" SERIAL="NUTANIX_NFS_18_0_636_59ea679e_cbd3_4cf5_a974_67409b719391"
NAME="sdn" SERIAL="NUTANIX_NFS_18_0_637_5523c88e_310a_4fb6_b76f_48f624d1ce1f"
NAME="sdo" SERIAL="NUTANIX_NFS_18_0_638_0e189f7a_785d_46f1_bb56_8f422d421ecb"
NAME="sdp" SERIAL="NUTANIX_NFS_18_0_639_9143ba87_d742_4130_bbe0_2084e8ebad3f"
NAME="sdq" SERIAL="NUTANIX_NFS_18_0_640_3b823817_c2b6_49af_9a55_fc4706216501"
The list of drives I want to identify is in this format:
'ae748138_9bc0_4499_8068_d00a84a800b6'
'8a082956_a2be_42ca_a14b_7d21ed3d5a2d'
'6185353f_5c13_47ab_af58_b72d4d07106b'
'fbeed366_6956_4de1_a217_487d75fd003c'
'019dcefb_0e1b_4086_96a9_982ae52fd88a'
'5d0b55e8_d0bd_4ee4_a2be_0bef88d94732'
'823beaed_1c2e_4203_ab5f_be294382d0c5'
'e4ee8620_5b53_4187_8f8c_7123ee8d3486'
'41551a1b_7dd9_4fd4_95bc_1fcdebb9dc85'
'12d517fc_e726_4512_b176_5d4075d0ecfe'
'd682eee4_cec0_4809_9efe_97aec7bf15d0'
'59ea679e_cbd3_4cf5_a974_67409b719391'
'5523c88e_310a_4fb6_b76f_48f624d1ce1f'
'0e189f7a_785d_46f1_bb56_8f422d421ecb'
'9143ba87_d742_4130_bbe0_2084e8ebad3f'
'3b823817_c2b6_49af_9a55_fc4706216501'
I need to identify the logical name for only the UUIDs on that list in order to create a LVM volume group out of them. But I'm don't understand how to proceed from there. Anyone can propose an example ?
After removing the 's from you list of UUIDs (can be done with tr) you could use simply grep to extract the corresponding lines from your list of devices. To extract only the values of NAME="..." from these lines use cut.
grep -Ff <(tr -d \' < fileListOfUUIDs) fileListOfDevices | cut -d\" -f2
If the inputs you showed us are not stored in files but generated by commands you can use
commandListOfDevices | grep -Ff <(commandListOfUUIDs | tr -d \') | cut -d\" -f2
You are trying to extract field values from match results generated by a set of tokens. sed or awk can do the trick.
To extract the value of a field called NAME from one of the match results simply do this:
josh#linux ~ $ result='NAME="sda" FSTYPE="ext4" SERIAL="NUTANIX_...dbd"'
josh#linux ~ $ echo $result | sed -e 's/.*NAME="//; s/".*//;'
sda
To extract the value of a field called FSTYPE from the same match result:
josh#linux ~ $ echo $result | sed -e 's/.*FSTYPE="//; s/".*//;'
ext4
What if the order of the field changes? ..well the same code works.
josh#linux ~ $ result='FSTYPE="ext4" NAME="sda" SERIAL="NUTANIX_...dbd"'
josh#linux ~ $ echo $result | sed -e 's/.*NAME="//; s/".*//;'
sda
With this in mind, extraction of field values from all match results can then be processed in a loop. I have created a small script to automate the process and I am posting it here in the hope that it would be useful.
#!/bin/bash
#
# usage: doodle -F <file-containing-pattern(s)>
# doodle -P <pattern(s)>
#
if [ $# -ne 2 ]; then
echo 'invalid argument...'; exit;
else
pattern=
fi
if [ "$1" == "-F" ]; then
if [ ! -r $2 ]; then
echo "unable to read - $2"; exit
else
pattern=$(cat $2)
fi
elif [ "$1" == "-P" ]; then
pattern=$2
else
echo 'invalid argument...'; exit;
fi
if [ -z "$pattern" ]; then
echo 'invalid argument...'; exit;
fi
declare -a buffer
readarray buffer
if [ ${#buffer[*]} -gt 0 ]; then
for token in $pattern; do
for i in ${!buffer[*]}; do
match=$(echo ${buffer[i]} | sed -n "/$token/=");
if [ -n "$match" ]; then
##### Option 1 ##############################################################
# Uncomment the next line to print the same result that 'grep' would print.
#echo -e $(echo ${buffer[i]} | sed "s/$token/\\\\e[91m$token\\\\e[0m/");
##### Option 2 ##############################################################
# Uncomment the next line to print characters before the first [:space:]
#echo ${buffer[i]} | awk -F' ' '{ print $1 }';
##### Option 3 ##############################################################
# Uncomment the next line to email device-id to NSA
#TODO...
##### Option 4 ##############################################################
# Uncomment the next line to extract value of 'NAME' field
echo ${buffer[i]} | sed -e 's/.*NAME="//; s/".*//;'
##### Additional Option #####################################################
# Uncomment the next line (break command) to process one match per token
break;
fi
done
done
fi
unset buffer
To use this script, simply:
Copy the text in the above snippet into a file called doodle.
Open terminal and type:
josh#linux ~ $ chmod a+x doodle
To process tokens saved in a file called device-id.txt:
josh#linux ~ $ lsblk -nodeps -no name,serial -PS /dev/sd* | ./doodle -F device-ids.txt
To process tokens supplied as a single string:
josh#linux ~ $ lsblk -nodeps -no name,serial -PS /dev/sd* | ./doodle -P 'ae748138_9bc0_4499_8068_d00a84a800b6
8a082956_a2be_42ca_a14b_7d21ed3d5a2d
6185353f_5c13_47ab_af58_b72d4d07106b
...
3b823817_c2b6_49af_9a55_fc4706216501'
The script is very generic and you an can easily adapt it for use in related problems. You will find lots of tutorials on bash, sed and awk on the internet. There is also a comprehensive bash reference at gnu.org.

Bash array not passed out of loop

Am having this little bash script to build a custom ROM (see below)
Now the issue I encounter is the fact that I can't seem to be able to read the content of the array after the loop is done
While calling it before the new loop pass, all is good, however when the loop is done, I can not seem to be able to read it (seems to be empty)
I need to say that I am no bash expert and I usually do my code from out pal Google
What am I doing wrong here?
#!/bin/bash
#Build crDroid and optionally upload to FTP for all devices in devices.txt
# << Start configuration >>
#crDroid_path is the location of build environment root path aka {ANDROID_BUILD_TOP}
crDroid_path=~/crDroid
#set how much RAM is used by JACK if building with
RAM=12
#CCache size
ccachesize=30G
#set if you want to save changelog file to script_path (from where the script runs) at end of build (useful to add changelog info to forums and so on... easy to find)
copy_changelog=true
#FTP config
upload_build=true
FTP_hostname="ftp://domain.tld"
FTP_username=username
FTP_password=password
# << End configuration >>
# Specify colors utilized in the terminal
red=$(tput setaf 1) # red
grn=$(tput setaf 2) # green
ylw=$(tput setaf 3) # yellow
blu=$(tput setaf 4) # blue
cya=$(tput rev)$(tput bold)$(tput setaf 6) # bold cyan reversed
ylr=$(tput rev)$(tput bold)$(tput setaf 3) # bold yellow reversed
grr=$(tput rev)$(tput bold)$(tput setaf 2) # bold green reversed
txtrst=$(tput sgr0) # Reset
echo "==========================================="
echo "${cya}Initiate build script - v2.9 by Gabriel Lup (gwolfu#xda)"${txtrst}
echo "==========================================="
#detect path where the script is running
script_path="`dirname \"$0\"`" # relative
script_path="`( cd \"$script_path\" && pwd )`" # absolutized and normalized
if [ -z "$script_path" ] ; then
# error; for some reason, the path is not accessible
echo "${red}Can not read run path"
echo "Build can not continue"${txtrst}
exit 1 # fail
fi
#check if devices.txt exists
devices=$script_path/devices.txt
if [ ! -f $devices ]; then
echo "${red}devices.txt missing"
echo "Build can not continue"${txtrst}
exit 1 # fail
fi
#cleanup old changelog.txt
changelog=$script_path/changelog.txt
if [ -e $changelog ]; then
rm -f $changelog
fi
#check if already synced
crdroid_synced=$crDroid_path/vendor/lineage/config/crdroid.mk
if [ ! -f $crdroid_synced ]; then
echo "${ylw}Detected missing first sync... atempting first time sync..."${txtrst}
cd $crDroid_path
repo sync -f --force-sync --no-clone-bundle
fi
if [ ! -f $crdroid_synced ]; then
read -p "${red}Something went wrong :( - Maybe misconfigured script!?"
exit
fi
cd $crDroid_path
echo "${blu}Run sync?${txtrst}"
select yn in "Yes" "No"; do
case $yn in
Yes ) repo sync -f --force-sync --no-clone-bundle; break;;
No ) break;;
esac
done
echo "${blu}Make clean build?${txtrst}"
select yn in "Yes" "No"; do
case $yn in
Yes ) . build/envsetup.sh && make clean; break;;
No ) break;;
esac
done
#check if repopick.txt exists and execute commands from it
repopick_file=$script_path/repopick.txt
if [ -f $repopick_file ]; then
. build/envsetup.sh
cat $script_path/repopick.txt | while read line
do
$line
done
fi
INFO=()
#Set CCache size
$crDroid_path/prebuilts/misc/linux-x86/ccache/ccache -M $ccachesize
#detect android version based on crdroid.mk
rl2="`sed -n '2p' $crDroid_path/vendor/lineage/config/crdroid.mk`"
set -- "$rl2"
IFS=" "; declare -a android_major=($*)
rl3="`sed -n '3p' $crDroid_path/vendor/lineage/config/crdroid.mk`"
set -- "$rl3"
IFS=" "; declare -a android_minor=($*)
android=${android_major[2]}"."${android_minor[2]}
#detect crDroid version based on crdroid.mk
rl6="`sed -n '6p' $crDroid_path/vendor/lineage/config/crdroid.mk`"
set -- "$rl6"
IFS=" "; declare -a crDroid_version=($*)
crDroid=${crDroid_version[2]}
echo "==========================================="
echo "${ylr}Setting build environment"${txtrst}
echo "-------------------------------------------"
echo "Script path set to: "${grn}$script_path${txtrst}
echo "crDroid path set to: "${grn}$crDroid_path${txtrst}
echo "Jack RAM usage set to: "${grn}$RAM"GB RAM"${txtrst}
echo "Copy changelog to script path at end of build?: "${grn}$copy_changelog${txtrst}
echo "Upload complete build to FTP?: "${grn}$upload_build${txtrst}
echo "Trying to compile crDroid "${grn}$crDroid${txtrst}" based on Android "${grn}$android${txtrst}
echo "==========================================="
echo ""
echo "${ylw}Initiate build for all devices...${txtrst}" #described in devices.txt
cat $script_path/devices.txt | while read line
do
IFS=', ' read -r -a device_uploadpath <<< "$line"
#set device name
device=${device_uploadpath[0]}
if [[ $device == *"#"* ]]; then
device="${device//#}"
echo "${red}Found comment (#): Skipping build for ${ylw}$device ${red}${txtrst}"
INFO+=('1 '$device' (Reason: instruction to skip found in devices.txt)')
else
echo "${grn}Now building "${ylw}$device${txtrst}
#set BuildID - aka name of the zip file from OUT folder at the end of the build
BuildID="crDroidAndroid-"$android"-"$(date -d "$D" '+%Y')$(date -d "$D" '+%m')$(date -d "$D" '+%d')"-"$device"-v"$crDroid"-BETA.zip"
#Jack settings
echo "Adding "$RAM" RAM to JACK"
export JACK_SERVER_VM_ARGUMENTS="-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx"$RAM"g"
./prebuilts/sdk/tools/jack-admin kill-server
./prebuilts/sdk/tools/jack-admin start-server
#initiate build script and start actual build
. build/envsetup.sh
brunch $device
#define upload file path
uploadfile=$crDroid_path/out/target/product/$device/$BuildID
echo "Checking build result status..."
if [ -e "$uploadfile" ]; then
echo "${grn}Seems compilation is ready!${txtrst}"
if [ "$upload_build" = true ] ; then
echo "${grn}Uploading new "$device" build to FTP server${txtrst}"
echo "${ylw}Creating folder over FTP if not exist${txtrst}"
curl $FTP_hostname/${device_uploadpath[1]}/ --user $FTP_username:$FTP_password --ftp-create-dirs
echo "${ylw}Uploading...${txtrst}"
curl -T $uploadfile $FTP_hostname/${device_uploadpath[1]}/ --user $FTP_username:$FTP_password
res=$?
if test "$res" != "0"; then
echo "${red}Upload of build for device $device failed with code $res${txtrst}"
INFO+=('1 '$device' (Reason: CURL error no.'$res' ,however compilation appears to be made)')
else
INFO+=('2 '$device)
fi
fi
if [ "$copy_changelog" = true ] ; then
echo "Copy changelog file to "$script_path
cp $crDroid_path/out/target/product/$device/system/etc/Changelog.txt $script_path/changelog.txt
fi
else
echo "${red}Device "$device "did not produce a proper build${txtrst}"
INFO+=('0 '$device' (Reason: unknown - better run a new build manually)')
fi
fi
done
echo ${grr}"Script finished with following results"${txtrst}
for i in "${INFO[#]}"
do
#echo $i
if [[ $i == *"0"* ]]; then
codename=${i//0}
echo "${red}Compilation error for device" $codename${txtrst}
elif [[ $i == *"1"* ]]; then
codename=${i//1}
echo "${ylw}Warning for device" $codename${txtrst}
else
codename=${i//2}
echo "${grn}Device" $codename "compiled and uploaded successfully :)${txtrst}"
fi
done
This is the part that I am lost
echo ${grr}"Script finished with following results"${txtrst}
for i in "${INFO[#]}"
do
#echo $i
if [[ $i == *"0"* ]]; then
codename=${i//0}
echo "${red}Compilation error for device" $codename${txtrst}
elif [[ $i == *"1"* ]]; then
codename=${i//1}
echo "${ylw}Warning for device" $codename${txtrst}
else
codename=${i//2}
echo "${grn}Device" $codename "compiled and uploaded successfully :)${txtrst}"
fi
done

shell script array won't populate from for loop

Can anyone tell me why this array creation: cccr[$string_1]=$string_2 #doesn't work?
#!/bin/bash
firstline='[Event "Marchand Open"][Site "Rochester NY"][Date "2005.03.19"][Round "1"][White "Smith, Igor"][Black "Jones, Matt"][Result "1-0"][ECO "C01"][WhiteElo "2409"][BlackElo "1911"]'
unset cccr
declare -A cccr
(IFS='['; for word in $firstline; do
string_1=$(echo $word | cut -f1 -d'"' | tr -d ' ')
string_2=$( echo $word | cut -f2 -d'"' )
if [ ! -z $string_1 ]; then # If $string_1 is not empty
cccr[$string_1]=$string_2 # why doesn't this line work?
fi
done)
echo ${cccr[Event]} # echos null string
It happens because the value of string_1 is empty at the first iteration.
Example :
#!/bin/bash
firstline='[Event "Marchand Open"][Site "Rochester NY"][Date "2005.03.19"][Round "1"][White "Smith, Igor"][Black "Jones, Matt"][Result "1-0"][ECO "C01"][WhiteElo "2409"][BlackElo "1911"]'
unset cccr
declare -A cccr
(IFS='['; for word in $firstline; do
string_1=$( echo $word | cut -f1 -d'"' )
string_2=$( echo $word | cut -f2 -d'"' )
echo "$string_1 - $string_2"
#cccr[$string_1]=$string_2
done)
Output :
- # Problem !
Event - Marchand Open
Site - Rochester NY
...
You have to modify your script to prevent the value of being empty.
A very simple workaround is to check the value of string_1 before using it.
Example :
# ...
string_1=$( echo $word | cut -f1 -d'"' )
string_2=$( echo $word | cut -f2 -d'"' )
if [ ! -z $string_1 ]; then # If $string_1 is not empty
echo "$string_1 - $string_2"
cccr[$string_1]=$string_2
fi
# ...
From the man page of [
-z STRING
the length of STRING is zero
Output :
Event - Marchand Open
Site - Rochester NY
# ... No problem
EDIT
BTW, if look at the value of string_1, you will see that the value is Event' ' and not Event (there's a whitespace at the end of Event)
So cccr[Event] does not exist, but cccr[Event ] exists.
To fix that, you can delete the whitespaces in string_1 :
string_1=$(echo $word | cut -f1 -d'"' | tr -d ' ') # tr -d ' ' deletes all the whitespaces
EDIT 2
I forgot to tell you that it's normal if it does not work. Indeed, the loop is executed in a subshell environment. So the array is filled in the subshell, but not in the current shell.
From the man page of bash :
(list) list is executed in a subshell environment (see COMMAND EXECUTION ENVIRONMENT below). Variable
assignments and builtin commands that affect the shell's environment do not remain in effect
after the command completes. The return status is the exit status of list.
So there are 2 solutions :
1. Don't run the loop in a subshell (remove the parentheses).
# ...
OLDIFS=$IFS
IFS='['
for word in $firstline; do
string_1=$(echo $word | cut -f1 -d'"' | tr -d ' ')
string_2=$(echo $word | cut -f2 -d'"')
if [ ! -z $string_1 ]; then
cccr[$string_1]=$string_2
fi
done
IFS=$OLDIFS
echo "Event = ${cccr[Event]}"
echo "Site = ${cccr[Site]}"
Output :
Event = Marchand Open
Site = Rochester NY
2. Use your array in the subshell.
# ...
(IFS='['
for word in $firstline; do
string_1=$(echo $word | cut -f1 -d'"' | tr -d ' ')
string_2=$(echo $word | cut -f2 -d'"')
if [ ! -z $string_1 ]; then # If $string_1 is not empty
cccr[$string_1]=$string_2
fi
done
echo "Event = ${cccr[Event]}"
echo "Site = ${cccr[Site]}"
)
Output :
Event = Marchand Open
Site = Rochester NY

grep result into array

I want to count the number of IP's in a given file using this function, the IP's should go into an array so that I can use them later, but I get 'declare: not found' and 'cnt+=1: not found', why is this?
#!/bin/bash
searchString=$1
file=$2
countLines()
{
declare -a ipCount
cnt=0
while read line ; do
((cnt+=1))
ipaddr=$( echo "$line" | grep -o -E '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)' )
((ipCount[$ipaddr]+=1))
done
for ip in ${ipCount[*]}
do
printf "%-15s %s\n" "$ip" "${ipCount[$ip]}"
done
echo "total count=$cnt"
}
if [ $# -lt 2 ]; then
echo "missing searchString and file"
else
grep "$searchString" $file | countLines
fi
This is a piece of the test file I am trying on
Apr 25 11:33:21 Admin CRON[2792]: pam_unix(cron:session): session opened for user 192.168.1.2 by (uid=0)
Apr 25 12:39:01 Admin CRON[2792]: pam_unix(cron:session): session closed for user 192.168.1.2
Apr 27 07:42:07 John CRON[2792]: pam_unix(cron:session): session opened for user 192.168.2.22 by (uid=0)
The desired output would be just the IP's inside an array, then also a 'count' on how many IP's there were.
I know I can get the ip's with a grep command but I would like to do more with it later, and it's important that it's in an array.
Your two main issues were that you were using declare -a but to declare an associative array, you need declare -A. Then, to iterate over the keys of an associative array, you need to use for foo in ${!ArrayName[#]}. I also added some quotes to your variables to be on the safe side:
#!/bin/bash
searchString="$1"
file="$2"
countLines()
{
## You need -A for associative arrays
declare -A ipCount
cnt=0
while read line ; do
(( cnt+=1 ))
ipaddr=$( echo "$line" | grep -o -E '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)' )
(( ipCount["$ipaddr"]+=1 ))
done
## To iterate over the keys of an associative
## array, you need ${!ArrayName[#]}
for ip in "${!ipCount[#]}"
do
printf "%-15s %s\n" "$ip" "${ipCount[$ip]}"
done
echo "total count=$cnt"
}
if [ $# -lt 2 ]; then
echo "missing searchString and file"
else
grep "$searchString" "$file" | countLines
fi
This is the output of the above on your example file:
$ bash a.sh 27 a
192.168.2.22 1
192.168.1.2 2
total count=3

Resources