Bash array not passed out of loop - arrays

Am having this little bash script to build a custom ROM (see below)
Now the issue I encounter is the fact that I can't seem to be able to read the content of the array after the loop is done
While calling it before the new loop pass, all is good, however when the loop is done, I can not seem to be able to read it (seems to be empty)
I need to say that I am no bash expert and I usually do my code from out pal Google
What am I doing wrong here?
#!/bin/bash
#Build crDroid and optionally upload to FTP for all devices in devices.txt
# << Start configuration >>
#crDroid_path is the location of build environment root path aka {ANDROID_BUILD_TOP}
crDroid_path=~/crDroid
#set how much RAM is used by JACK if building with
RAM=12
#CCache size
ccachesize=30G
#set if you want to save changelog file to script_path (from where the script runs) at end of build (useful to add changelog info to forums and so on... easy to find)
copy_changelog=true
#FTP config
upload_build=true
FTP_hostname="ftp://domain.tld"
FTP_username=username
FTP_password=password
# << End configuration >>
# Specify colors utilized in the terminal
red=$(tput setaf 1) # red
grn=$(tput setaf 2) # green
ylw=$(tput setaf 3) # yellow
blu=$(tput setaf 4) # blue
cya=$(tput rev)$(tput bold)$(tput setaf 6) # bold cyan reversed
ylr=$(tput rev)$(tput bold)$(tput setaf 3) # bold yellow reversed
grr=$(tput rev)$(tput bold)$(tput setaf 2) # bold green reversed
txtrst=$(tput sgr0) # Reset
echo "==========================================="
echo "${cya}Initiate build script - v2.9 by Gabriel Lup (gwolfu#xda)"${txtrst}
echo "==========================================="
#detect path where the script is running
script_path="`dirname \"$0\"`" # relative
script_path="`( cd \"$script_path\" && pwd )`" # absolutized and normalized
if [ -z "$script_path" ] ; then
# error; for some reason, the path is not accessible
echo "${red}Can not read run path"
echo "Build can not continue"${txtrst}
exit 1 # fail
fi
#check if devices.txt exists
devices=$script_path/devices.txt
if [ ! -f $devices ]; then
echo "${red}devices.txt missing"
echo "Build can not continue"${txtrst}
exit 1 # fail
fi
#cleanup old changelog.txt
changelog=$script_path/changelog.txt
if [ -e $changelog ]; then
rm -f $changelog
fi
#check if already synced
crdroid_synced=$crDroid_path/vendor/lineage/config/crdroid.mk
if [ ! -f $crdroid_synced ]; then
echo "${ylw}Detected missing first sync... atempting first time sync..."${txtrst}
cd $crDroid_path
repo sync -f --force-sync --no-clone-bundle
fi
if [ ! -f $crdroid_synced ]; then
read -p "${red}Something went wrong :( - Maybe misconfigured script!?"
exit
fi
cd $crDroid_path
echo "${blu}Run sync?${txtrst}"
select yn in "Yes" "No"; do
case $yn in
Yes ) repo sync -f --force-sync --no-clone-bundle; break;;
No ) break;;
esac
done
echo "${blu}Make clean build?${txtrst}"
select yn in "Yes" "No"; do
case $yn in
Yes ) . build/envsetup.sh && make clean; break;;
No ) break;;
esac
done
#check if repopick.txt exists and execute commands from it
repopick_file=$script_path/repopick.txt
if [ -f $repopick_file ]; then
. build/envsetup.sh
cat $script_path/repopick.txt | while read line
do
$line
done
fi
INFO=()
#Set CCache size
$crDroid_path/prebuilts/misc/linux-x86/ccache/ccache -M $ccachesize
#detect android version based on crdroid.mk
rl2="`sed -n '2p' $crDroid_path/vendor/lineage/config/crdroid.mk`"
set -- "$rl2"
IFS=" "; declare -a android_major=($*)
rl3="`sed -n '3p' $crDroid_path/vendor/lineage/config/crdroid.mk`"
set -- "$rl3"
IFS=" "; declare -a android_minor=($*)
android=${android_major[2]}"."${android_minor[2]}
#detect crDroid version based on crdroid.mk
rl6="`sed -n '6p' $crDroid_path/vendor/lineage/config/crdroid.mk`"
set -- "$rl6"
IFS=" "; declare -a crDroid_version=($*)
crDroid=${crDroid_version[2]}
echo "==========================================="
echo "${ylr}Setting build environment"${txtrst}
echo "-------------------------------------------"
echo "Script path set to: "${grn}$script_path${txtrst}
echo "crDroid path set to: "${grn}$crDroid_path${txtrst}
echo "Jack RAM usage set to: "${grn}$RAM"GB RAM"${txtrst}
echo "Copy changelog to script path at end of build?: "${grn}$copy_changelog${txtrst}
echo "Upload complete build to FTP?: "${grn}$upload_build${txtrst}
echo "Trying to compile crDroid "${grn}$crDroid${txtrst}" based on Android "${grn}$android${txtrst}
echo "==========================================="
echo ""
echo "${ylw}Initiate build for all devices...${txtrst}" #described in devices.txt
cat $script_path/devices.txt | while read line
do
IFS=', ' read -r -a device_uploadpath <<< "$line"
#set device name
device=${device_uploadpath[0]}
if [[ $device == *"#"* ]]; then
device="${device//#}"
echo "${red}Found comment (#): Skipping build for ${ylw}$device ${red}${txtrst}"
INFO+=('1 '$device' (Reason: instruction to skip found in devices.txt)')
else
echo "${grn}Now building "${ylw}$device${txtrst}
#set BuildID - aka name of the zip file from OUT folder at the end of the build
BuildID="crDroidAndroid-"$android"-"$(date -d "$D" '+%Y')$(date -d "$D" '+%m')$(date -d "$D" '+%d')"-"$device"-v"$crDroid"-BETA.zip"
#Jack settings
echo "Adding "$RAM" RAM to JACK"
export JACK_SERVER_VM_ARGUMENTS="-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx"$RAM"g"
./prebuilts/sdk/tools/jack-admin kill-server
./prebuilts/sdk/tools/jack-admin start-server
#initiate build script and start actual build
. build/envsetup.sh
brunch $device
#define upload file path
uploadfile=$crDroid_path/out/target/product/$device/$BuildID
echo "Checking build result status..."
if [ -e "$uploadfile" ]; then
echo "${grn}Seems compilation is ready!${txtrst}"
if [ "$upload_build" = true ] ; then
echo "${grn}Uploading new "$device" build to FTP server${txtrst}"
echo "${ylw}Creating folder over FTP if not exist${txtrst}"
curl $FTP_hostname/${device_uploadpath[1]}/ --user $FTP_username:$FTP_password --ftp-create-dirs
echo "${ylw}Uploading...${txtrst}"
curl -T $uploadfile $FTP_hostname/${device_uploadpath[1]}/ --user $FTP_username:$FTP_password
res=$?
if test "$res" != "0"; then
echo "${red}Upload of build for device $device failed with code $res${txtrst}"
INFO+=('1 '$device' (Reason: CURL error no.'$res' ,however compilation appears to be made)')
else
INFO+=('2 '$device)
fi
fi
if [ "$copy_changelog" = true ] ; then
echo "Copy changelog file to "$script_path
cp $crDroid_path/out/target/product/$device/system/etc/Changelog.txt $script_path/changelog.txt
fi
else
echo "${red}Device "$device "did not produce a proper build${txtrst}"
INFO+=('0 '$device' (Reason: unknown - better run a new build manually)')
fi
fi
done
echo ${grr}"Script finished with following results"${txtrst}
for i in "${INFO[#]}"
do
#echo $i
if [[ $i == *"0"* ]]; then
codename=${i//0}
echo "${red}Compilation error for device" $codename${txtrst}
elif [[ $i == *"1"* ]]; then
codename=${i//1}
echo "${ylw}Warning for device" $codename${txtrst}
else
codename=${i//2}
echo "${grn}Device" $codename "compiled and uploaded successfully :)${txtrst}"
fi
done
This is the part that I am lost
echo ${grr}"Script finished with following results"${txtrst}
for i in "${INFO[#]}"
do
#echo $i
if [[ $i == *"0"* ]]; then
codename=${i//0}
echo "${red}Compilation error for device" $codename${txtrst}
elif [[ $i == *"1"* ]]; then
codename=${i//1}
echo "${ylw}Warning for device" $codename${txtrst}
else
codename=${i//2}
echo "${grn}Device" $codename "compiled and uploaded successfully :)${txtrst}"
fi
done

Related

Appending to an array in bash, why this isn't working?

I'm trying from a list of files with a pattern correctly matched by regex to check whether this value is in my array, if not, append it.
Unfortunately, this code that I build up inspired by some stack overflow post doesn't work (nothing is happened, the =~ doesn't seem to find the bash_rematch, and also it doesn't output anything?
sample_array=() #creating the array
for context_files in data/*.txt.gz # checking all the different samples id we have
do
[[ $context_files =~ SL[0-9]{6} ]]
echo 'context file:' "$context_files"
echo 'rematch:' "${BASH_REMATCH[0]}"
if ! [[ " ${sample_array[*]} " =~ (^|[[:space:]])"${BASH_REMATCH[0]}"($|[[:space:]]) ]]; then
echo 'condition matched'
echo 'rematch:' "${BASH_REMATCH[0]}"
sample_array+=(" ${BASH_REMATCH[0]} ")
fi
done
echo "${sample_array[*]}"
replacing this code by
sample_array=() #creating the array
for context_files in data/*.txt.gz # checking all the different samples id we have
do
[[ $context_files =~ SL[0-9]{6} ]]
echo 'context file:' "$context_files"
echo 'rematch:' "${BASH_REMATCH[0]}"
if ! [[ " ${sample_array[*]} " == "${BASH_REMATCH[0]}" ]]; then
echo 'condition matched'
echo 'rematch:' "${BASH_REMATCH[0]}"
sample_array+=(" ${BASH_REMATCH[0]} ")
fi
done
echo "${sample_array[*]}"
will this time add all the variable
output :
A B A B A B
I probably don't get something in how the if is managed and/or how the regex lookup in a bash array is to be made but I'd gladly get some help!
The second match is negated, so in order to enter the then part, the match needs to fail. A failed match resets $BASH_REMATCH.
#! /bin/bash
sample_array=()
for context_files in data/SL{111111,222222,333333,111111,222222}.txt.gz ; do
[[ $context_files =~ SL[0-9]{6} ]]
match=${BASH_REMATCH[0]}
echo 'context file:' "$context_files"
echo 'rematch:' "$match"
if ! [[ " ${sample_array[*]} " =~ (^|[[:space:]])"$match"($|[[:space:]]) ]]; then
echo 'condition matched'
echo 'rematch:' "$match"
sample_array+=(" $match ")
fi
done
echo "${sample_array[*]}"
Here is a completely alternative solution in bash-style like John Kugelman suggested:
printf %s\\n data/*.txt.gz | grep -Eo 'SL[0-9]{6}' | sort -u
If you need the results in an array, use mapfile:
mapfile -t array <(printf %s\\n data/*.txt.gz | grep -Eo 'SL[0-9]{6}' | sort -u)

Using mapfile to save output to associative arrays

In practicing bash, I tried writing a script that searches the home directory for duplicate files in the home directory and deletes them. Here's what my script looks like now.
#!/bin/bash
# create-list: create a list of regular files in a directory
declare -A arr1 sumray origray
if [[ -d "$HOME/$1" && -n "$1" ]]; then
echo "$1 is a directory"
else
echo "Usage: create-list Directory | options" >&2
exit 1
fi
for i in $HOME/$1/*; do
[[ -f $i ]] || continue
arr1[$i]="$i"
done
for i in "${arr1[#]}"; do
Name=$(sed 's/[][?*]/\\&/g' <<< "$i")
dupe=$(find ~ -name "${Name##*/}" ! -wholename "$Name")
if [[ $(find ~ -name "${Name##*/}" ! -wholename "$Name") ]]; then
mapfile -t sumray["$i"] < <(find ~ -name "${Name##*/}" ! -wholename "$Name")
origray[$i]=$(md5sum "$i" | cut -c 1-32)
fi
done
for i in "${!sumray[#]}"; do
poten=$(md5sum "$i" | cut -c 1-32)
for i in "${!origray[#]}"; do
if [[ "$poten" = "${origray[$i]}" ]]; then
echo "${sumray[$i]} is a duplicate of $i"
fi
done
done
Originally, where mapfile -t sumray["$i"] < <(find ~ -name "${Name##*/}" ! -wholename "$Name") is now, my line was the following:
sumray["$i"]=$(find ~ -name "${Name##*/}" ! -wholename "$Name")
This saved the output of find to the array. But I had an issue. If a single file had multiple duplicates, then all locations found by find would be saved to a single value. I figured I could use the mapfile command to fix this, but now it's not saving anything to my array at all. Does it have to do with the fact that I'm using an associative array? Or did I just mess up elsewhere?
I'm not sure if I'm allowed to answer my own question, but I figured that I should post how I solved my problem.
As it turns out, the mapfile command does not work on associative arrays at all. So my fix was to save the output of find to a text file and then store that information in an indexed array. I tested this a few times and I haven't seemed to encounter any errors yet.
Here's my finished script.
#!/bin/bash
# create-list: create a list of regular files in a directory
declare -A arr1 origray
declare indexray
#Verify that Parameter is a directory.
if [[ -d "$HOME/$1/" && -n "$1" ]]; then
echo "Searching for duplicates of files in $1"
else
echo "Usage: create-list Directory | options" >&2
exit 1
fi
#create list of files in specified directory
for i in $HOME/${1%/}/*; do
[[ -f $i ]] || continue
arr1[$i]="$i"
done
#search for all duplicate files in the home directory
#by name
#find checksum of files in specified directory
for i in "${arr1[#]}"; do
Name=$(sed 's/[][?*]/\\&/g' <<< "$i")
if [[ $(find ~ -name "${Name##*/}" ! -wholename "$Name") ]]; then
find ~ -name "${Name##*/}" ! -wholename "$Name" >> temp.txt
origray[$i]=$(md5sum "$i" | cut -c 1-32)
fi
done
#create list of duplicate file locations.
if [[ -f temp.txt ]]; then
mapfile -t indexray < temp.txt
else
echo "No duplicates were found."
exit 0
fi
#compare similarly named files by checksum and delete duplicates
count=0
for i in "${!indexray[#]}"; do
poten=$(md5sum "${indexray[$i]}" | cut -c 1-32)
for i in "${!origray[#]}"; do
if [[ "$poten" = "${origray[$i]}" ]]; then
echo "${indexray[$count]} is a duplicate of a file in $1."
fi
done
count=$((count+1))
done
rm temp.txt
This is kind of sloppy but it does what it's supposed to do. md5sum may not be the optimal way to check for file duplicates but it works. All I have to do is replace echo "${indexray[$count]} is a duplicate of a file in $1." with rm -i ${indexray[$count]} and it's good to go.
So my next question would have to be...why doesn't mapfile work with associative arrays?

getting error while looping through array in shell

I have written a script to ftp a set of files from my directory , but i am getting an error while trying to loop through an array.
#!/usr/bin/ksh
HOST='xxx.xxx.xxx.xxx'
USER='avio'
PASSWD='jun'
FILES[0]=D141203.T024413
FILES[1]=D150101.T012755
FILES[2]=D141203.T024418
echo 'no of files: ' ${#FILES[#]}
ftp -n -v $HOST << EOS
ascii
user $USER $PASSWD
for i in "${FILES[#]}"
do
get $i
done
bye
EOS
here is my o/p :
no of files: 3
ftp.sh[10]: i: 0403-009 The specified number is not valid for this command.
The for loop is not executed as a bash command; rather, it is passed as a string to the ftp command. Instead, use a pipe to feed the output of the command to ftp:
#!/usr/bin/ksh
HOST='xxx.xxx.xxx.xxx'
USER='avio'
PASSWD='jun'
FILES[0]=D141203.T024413
FILES[1]=D150101.T012755
FILES[2]=D141203.T024418
echo 'no of files: ' ${#FILES[#]}
{ echo "ascii"
echo "user $USER $PASSWD"
for i in "${FILES[#]}"; do
echo "get $i"
done
echo "bye"
} | ftp -n -v $HOST

Appending to elements in a Bash associative array

My name is Mike and I am a total noob at Bash programming. With that being said, please don't chew me to a pulp. Here's my scenario. I'm importing a csv and processing it line by line. It looks up information in AD and pipes that result to a variable $ADresult1.
I then take that output and send it into the array.
declare -A arrmail
arrmail[$ADresult1]="$v1, $v3"
appendvar+="; ${arrmail[$ADresult1]}"
if [ ! -z "${arrmail[$ADresult1]}" ]; then
echo ${arrmail[#]}
else
echo ${arrmail[#]}="$appendvar"
fi
Now in my CSV file, the var $ADresult1 has more than one entry that is the same, however each element should be different.
For example: The first line might contain arrmail[p100]=Y2K, Y2J
The second line might arrmail[p100]=GGG, GG1
My question would be how to combine the elements from the first line and second line into one line of output and/or if this is possible.
For instance arrmail[p100]=Y2K, Y2J; GGG, GG1
If needs be, I can post the whole entire script for a bigger picture.
Here's the script.
#!/bin/bash
###--------------------------------------------------------------------------
# Author : Volha Zhdanava (p146819)
# Date : 11-FEB-2013
# Purpose: To automate the process of sending out notification emails to developers when
# Assigned DEV Machine is on the Expired Devices List.
#----------------------------------------------------------------------------
# ----MODIFIED----
#----------------------------------------------------------------------------------------------------
# Author Revsn Date Revsn ID Revsn Purpose
# Mike Bell 02-28-14 Rev.03 Add ability to email j or p number
# Mike Bell 01-02-15 Rev.04 Add verbiage about 90 day licensing
###----------------------------------------------------------------------------------------------------
Purple='\e[1;35m'
ColorOFF='\e[0m'
TO_TEAM="dev-tools#edwardjones.com"
#-----------------------------------------
#Ask for input CSV file
#-----------------------------------------
FILEDIR="/export/share/is/dev_env/expdev"
FILENAME="expired_devices.csv"
#if the directory is not empty, find the csv file
if [[ $(ls -A $FILEDIR) ]]
then
#check if the csv file was stored with default name <expired_devices.csv>
if [ -f $FILEDIR/$FILENAME ]
then
#absolute path to the downloaded file
CSVFILE=`echo $FILEDIR/$FILENAME`
#if the csv file was stored under a user-defined name,
#re-define the absolute path to that file
else
#count how many files are saved in the expdev directory
NAME=( `ls -A $FILEDIR` )
CNT=${#NAME[#]}
echo -e "There is/are ${Purple}${CNT}${ColorOFF} file(s) in the ${Purple}$FILEDIR${ColorOFF} directory:"
echo ${NAME[#]}
#if there is only one file saved within the expdev directory
if [ ${CNT} -eq 1 ]
then
FILENAME=${NAME[0]}
else
#it will prompt to enter the name of the file
REPLY="n"
while [ "$REPLY" != "y" ]
do
echo -e "Please enter the full name of the downloaded csv file: \c"
read FILENAME
echo -e "You entered ${Purple}$FILENAME${ColorOFF}. Is this correct? y/n: \c\n"
read ANSWER
REPLY=`echo $ANSWER | tr [:upper:] [:lower:]`
done
fi
#verify the specified file exists in <expdev> directory
echo `ls -Al $FILEDIR/$FILENAME`
echo -e "\nThe script will run against the listed above ${Purple}$FILENAME${ColorOFF} file.\n${Purple}NOTE${ColorOFF}: Do NOT proceed if it says ${Purple}No such file or directory${ColorOFF}.\nWould you like to proceed? y/n -->:\c "
read PROCEED
PROCEED=`echo $PROCEED | tr [:upper:] [:lower:]`
if [ "$PROCEED" == "y" ]
#user agrees
then
CSVFILE=`echo $FILEDIR/$FILENAME`
else
echo -e "Make sure you saved your CSV file in ${Purple}$FILEDIR${ColorOFF} directory and run the script again."
exit 1
fi
fi
#----------------------------------------------
#Remove header from CSVFILE
#and create a temp csvfile.txt
#for "WHILE READ" loop
#----------------------------------------------
Nrows=`cat ${CSVFILE}| wc -l`
N=`expr $Nrows - 1`
tail -$N ${CSVFILE} > csvfile.txt
#----------------------------------------------
#Determine the Extension Date
#----------------------------------------------
TODAYsDATE=`date +"%a"`
case $TODAYsDATE in
Mon ) ExtDate=`date --date='4 day' +"%A, %d-%b-%Y"`
;;
Tue ) ExtDate=`date --date='6 day' +"%A, %d-%b-%Y"`
;;
Wed ) ExtDate=`date --date='6 day' +"%A, %d-%b-%Y"`
;;
Thu ) ExtDate=`date --date='6 day' +"%A, %d-%b-%Y"`
;;
Fri ) ExtDate=`date --date='6 day' +"%A, %d-%b-%Y"`
;;
* ) echo "Cannot determine the Extention Date."
REPLY="n"
while [ "$REPLY" != "y" ]
do
echo -e "Please enter Extended Date: \c"
read ExtDate
echo -e "You entered ${Purple}$ExtDate${ColorOFF}. Is this correct? y/n: \c "
read ANSWER
REPLY=`echo $ANSWER | tr [:upper:] [:lower:]`
done
esac
echo -e "The Extended Date for the Application Owners will be ${Purple}$ExtDate${ColorOFF}.\nPlease make a note for End Date in the APEX Workstation Tracking Tool."
#create a temp confirmation file that will be sent to dev-tools
echo -e "\nThe Extended Date is $ExtDate.\nSent emails to: " > confirmation.txt
#----------------------------------------------
#Read the csvfile.txt
#----------------------------------------------
while IFS=',' read v1 v2 v3 v4 other
do
#----------------------------------------------
#Determine email address
#----------------------------------------------
v2=` echo $v2 | tr '"/,\&' ' ' `
NAME=( ${v2} )
cnt=${#NAME[#]}
#-----------------------------------------------
# Filter j or p<number> userids from table V2
#--------------------------------------------------
if [[ "$NAME" =~ [P|p][[:digit:]]{6}$|[J|j][[:digit:]]{5}$ ]]
then
ADresult1=`getent passwd "$NAME" | awk -F":" '{print $1}'`
echo ${ADresult1}
#------------------------------------------------------------------------------------------------------
# Processing Array
#-----------------------------------------------------------------------------------------------------
declare -A arrmail
arrmail[$ADresult1]="$v1, $v3"
appendvar+="; ${arrmail[$ADresult1]}"
if [ ! -z "${arrmail[$ADresult1]}" ]
then
echo ${arrmail[#]}
else
echo ${arrmail[#]}="$appendvar"
fi
#--------------------------------------------------------------------------------------
# If query matches j or p#, then email user
#------------------------------------------------------------------------------------------
#
#
# v1=` echo $v1 | tr '"' ' ' `
# SUBJECT=" ${v1}, ${v3} - Your Development Machine Status Update"
#
# TO="${ADresult1}#edwardjones.com"
#
# v4=` echo $v4 | tr '"' ' ' `
#
# devmailmessage="\nGreetings ${NAME[0]}, \n\nI am in the process of cleaning up the machines in our lab.\n\nAssigned device: ${v1} \n\nEffort: ${v3}\n\nThe above machine have been assigned to you since ${v4}.\n\nDo you still need this machine or can I reclaim it?\nIf this machine is still required, please let us know how much longer you will need it.\nPlease keep in mind that Dev. workstations or laptops can only be assigned for 90 days due to licensing restrictions.\n\nIf this machine is no longer needed, please let us know also.\n\nThis machine will be rebuilt on or after $ExtDate.\n\nPlease reply to dev-tools#edwardjones.com by $ExtDate.\n\n\nThank you,\n\nDevelopment Environment Support team\ndev-tools#edwardjones.com "
#
#echo -e ${devmailmessage}|mail -s "$SUBJECT" $TO -- -f $TO_TEAM
#--------------------------------------------------------------------------------------------
# If query does not match j or p number, then lookup username via first and last name
#----------------------------------------------------------------------------------------------
elif [ ${cnt} -eq 2 ]
then
ADresult=`getent passwd | grep -i "${NAME[0]}" | grep -i "${NAME[1]}"`
echo ${ADresult}
#count search results from Active Directory; only email if there is one unique result
ADsearch1=`echo ${ADresult} | grep -i "${NAME[0]}" | wc -l`
ADsearch2=`echo ${ADresult} | grep -i "${NAME[1]}" | wc -l`
if [ ${ADsearch1} -eq 1 ] && [ ${ADsearch2} -eq 1 ]
then
pNumber=`echo ${ADresult} | awk -F":" '{print $1}'`
#------------------------------------------------------------------------------------------------------
# Processing Array
#-----------------------------------------------------------------------------------------------------
#declare -A arrmail
#arrmail[$pNumber]="$v1, $v3"
#appendvar+="; ${arrmail[$pNumber]}"
#if [ ! -z "${arrmail[$pNumber]}" ]
# then
# {arrmail[$pNumber]}="${arrmail[$ADresult1]}"
#else
# {arrmail[$ADresult1]}="$appendvar"
#fi
#------------------------------------------------------------------------------------------------------------
# v1=` echo $v1 | tr '"' ' ' `
# SUBJECT=" ${v1}, ${v3} - Your Development Machine Status Update"
# TO="${pNumber}#edwardjones.com"
# v4=` echo $v4 | tr '"' ' ' `
# devmailmessage="\nGreetings ${NAME[0]}, \n\nI am in the process of cleaning up the machines in our lab.\n\nAssigned device: ${v1} \n\nEffort: ${v3}\n\nThe above machine have been assigned to you since ${v4}.\n\nDo you still need this machine or can I reclaim it?\nIf this machine is still required, please let us know how much longer you will need it.\nPlease keep in mind that Dev. workstations or laptops can only be assigned for 90 days due to licensing restrictions.\n\nIf this machine is no longer needed, please let us know also.\n\nThis machine will be rebuilt on or after $ExtDate.\n\nPlease reply to dev-tools#edwardjones.com by $ExtDate.\n\n\nThank you,\n\nDevelopment Environment Support team\ndev-tools#edwardjones.com "
# echo -e ${devmailmessage}| mail -s "$SUBJECT" $TO -- -f $TO_TEAM
# else
# TO=`echo "ACTION REQUIRED: Email address cannot be determined for the following user: ${v2}, with assigned device: ${v1}. Please email this user yourself."`
fi
# else
# TO=`echo "ACTION REQUIRED: Email address cannot be determined for the following user: ${v2}, with assigned device: ${v1}. Please email this user yourself."`
fi
#echo -e "\n$TO ${v2}, ${v1}" >> confirmation.txt
echo "NEXT ROW"
done < csvfile.txt
#----------------------------------------------
#sending out confirmation email to dev-tools
#----------------------------------------------
#subject="WW - Notification Emails to Developers - Status"
#cat confirmation.txt | mail -s "$subject" $TO_TEAM -- -f $TO_TEAM
#echo -e "Result: Processing is COMPLETED. Please check the ${Purple}$subject${ColorOFF} email. Note: ${Purple}$FILENAME${ColorOFF} file was deleted from ${Purple}$FILEDIR${ColorOFF} directory to avoid duplicates in the future."
rm ${CSVFILE}
rm csvfile.txt
rm confirmation.txt
else
#----------------------------------------------------
#if the directory is empty
#------------------------------------------------------
echo -e "Please make sure the CSV file was downloaded to ${Purple}$FILEDIR${ColorOFF} directory and run the script again."
fi
###
#--------THE END---------------------------------
###
Here's some of the output of the script.
IFS=,
read v1 v2 v3 v4 other
++ echo p098650
++ tr '"/,\&' ' '
v2=p098650
NAME=(${v2})
cnt=1
[[ p098650 =~ [P|p][[:digit:]]{6}$|[J|j][[:digit:]]{5}$ ]]
++ getent passwd p098650
++ awk -F: '{print $1}'
ADresult1=p098650
echo p098650
p098650
declare -A arrmail
arrmail[$ADresult1]='CNU327B1Y3, EDGARLITE_0340_COR_00'
appendvar+='; CNU327B1Y3, EDGARLITE_0340_COR_00'
'[' '!' -z 'CNU327B1Y3, EDGARLITE_0340_COR_00' ']'
echo CNU327B1Y3, EDGARLITE_0340_COR_00
CNU327B1Y3, EDGARLITE_0340_COR_00
echo 'NEXT ROW'
NEXT ROW
IFS=,
read v1 v2 v3 v4 other
++ echo p098650
++ tr '"/,\&' ' '
v2=p098650
NAME=(${v2})
cnt=1
[[ p098650 =~ [P|p][[:digit:]]{6}$|[J|j][[:digit:]]{5}$ ]]
++ getent passwd p098650
++ awk -F: '{print $1}'
ADresult1=p098650
echo p098650
p098650
declare -A arrmail
arrmail[$ADresult1]='CNU327B1GP, BUZZLITE_0340_COR_00'
appendvar+='; CNU327B1GP, BUZZLITE_0340_COR_00'
'[' '!' -z 'CNU327B1GP, BUZZLITE_0340_COR_00' ']'
echo CNU327B1GP, BUZZLITE_0340_COR_00
CNU327B1GP, BUZZLITE_0340_COR_00
echo 'NEXT ROW'
NEXT ROW
What's I'm expecting to see is, arrmail[$ADresult1]='CNU327B1Y3, EDGARLITE_0340_COR_00';'CNU327B1GP, BUZZLITE_0340_COR_00'
I couldn't include a screenshot of the CSV, so I'm just going to cut and paste some of it.
Sernum $V1 Username $V2 Effort $V3 Startdate $V4
CNU327B1Y3 p098650 EDGARLITE_0340_COR_00 4-Mar-14
CNU3199N31 Bell SDLCONTENTPORTER_2013_COR_00 10-Mar-14
CNU327B1GP p098650 BUZZLITE_0340_COR_00 4-Mar-14
CNU2479GLB Mike XENDESKAGENTX64_0640_COR_00 28-Feb-14
CNU327B1PB Mike Bell BONDONEUOBRA_2001_COR_00 6-Mar-14
2UA24705YY Bell Mike SASENTGUIDEX64_0610_COR_00 25-Nov-13
CNU2479K7Z Bell, Mike Software Testing 12-Dec-13
2UA24705ZZ Bell Mike TESTGUIDEX64_0610_COR_00 25-Nov-13
I know this is a lot. I've been working on this for a long time and I'm trying to gain a greater understanding of how to use and manipulate arrays. Also if you have an recommendations on books covering the subject, I'll be happy to review those also. Thank you in advance for all the help.

how to split files from error.log file of apache server while reading file line by line continuously?

I have done like this but I am having trouble with shellscript I have written. I am confused with tail command functionality and also when I see output of error.log on terminal it shows lines with 'e' deleted from words.
I have written like this please guide me how can I get my problem solved. I want to read this error.log file line by line and during reading lines I want to split fixed number of lines to small files with suffix i.e log-aa,log-ab,... I did this using split command. After splitting I want to filter lines with GET or POST word in them using regex and store this filtered lines into new file. After this store gets completed I need to delete all these log-* files.
I have written like this:
enter code here
processLine(){
line="$#"
echo $line
$ tail -f $FILE
}
FILE="/var/log/apache2/error.log"
if [ "$1" == "/var/log/apache2/error.log" ]; then
FILE="/dev/stdin"
else
FILE="$1"
if [ ! -f $FILE ]; then
echo "$FILE : does not exists"
exit 1
elif [ ! -r $FILE ]; then
echo "$FILE: can not read"
exit 2
fi
fi
#BAKIFS=$IFS
#IFS=$(echo -en "\n\b")
exec 3<&0
exec 0<"$FILE"
#sed -e 's/\[debug\].*\(data\-HEAP\)\:\/-->/g' error.log > /var/log/apache2/error.log.1
while read -r line
do
processLine $line
done
exec 0<&3
IFS=$BAKIFS
logfile="/var/log/apache2/error.log"
pattern="bytes"
# read each new line as it gets written
# to the log file
#tail -1 $logfile
tail -fn0 $logfile | while read line ; do
# check each line against our pattern
echo "$line" | grep -i "$pattern"
#sed -e 's/\[debug\].*\(data\-HEAP\)\:/-->/g' error.log >/var/log/apache2/error.log
split -l 1000 error.log log-
FILE2="/var/log/apache2/log-*"
if [ "$1" == "/var/log/apache2/log-*" ]; then
FILE2="/dev/stdin"
else
FILE2="$1"
if [ ! -f $FILE2 ]; then
echo "$FILE : does not exists"
exit 1
elif [ ! -r $FILE2 ]; then
echo "$FILE: can not read"
exit 2
fi
fi
BAKIFS=$IFS
IFS=$(echo -en "\n\b")
exec 3<&0
exec 0<"$FILE2"
while read -r line
do
processLine $line
echo $line >>/var/log/apache2/url.txt
done
exec 0<&3
IFS=$BAKIFS
find . -name "var/log/apache2/logs/log-*.*" -delete
done
exit 0
The below code deletes files after reading and splitting error.log but when I put tail -f $FILE it stops deleting files I want to delete log-* files after it reaches last line of error.log file:
enter code here
processLine(){
line="$#"
echo $line
}
FILE=""
if [ "$1" == "" ]; then
FILE="/dev/stdin"
else
FILE="$1"
# make sure file exist and readable
if [ ! -f $FILE ]; then
echo "$FILE : does not exists"
exit 1
elif [ ! -r $FILE ]; then
echo "$FILE: can not read"
exit 2
fi
fi
#BAKIFS=$IFS
#IFS=$(echo -en "\n\b")
exec 3<&0
exec 0<"$FILE"
while read -r line
do
processLine $line
split -l 1000 error.log log-
cat log-?? | grep "GET\|POST" > storefile
#tail -f $FILE
done
rm log-??
exec 0<&3
#IFS=$BAKIFS
exit 0
Your code seems unnecessarily long and complex, and the logic is unclear. It should not have been allowed to grow this big without working correctly.
Consider this:
split -l 1000 error.log log-
cat log-?? | grep "GET\|POST" > storefile
rm log-??
Experiment with this. If these three commands do what you expect, you can add more functionality (e.g. using paths, checking for the existence of error.log), but don't add code until you have this part working.

Resources