After using bash -x to debug this script, it appears that a problem occurs near $InvalidText, where grep treats the output of InvalidText as separate file names.
The only solution I can think of is to direct the output of InvalidText to a file, then grep that file.. are there any other alternatives?
# find IP address of Invalid user and give the fake name they tried. Loop for all IP addresses
InvalidText=$(grep "Invalid user" syslog.log)
IPInvalidUser=$(grep "Invalid user" syslog.log | awk '{print $NF}' | uniq)
InvalidUserarray=( $IPInvalidUser )
for FAKEHOST in "${InvalidUserarray[#]}"
do
fakeinfo=$(grep ${FAKEHOST} $InvalidText | cut -d: -f5 | awk '{print $3}')
echo -e "Invalid user attempts:\n"
echo -e "$FAKEHOST $fakeinfo"
done
Related
I have a problem with “basename” command as follow:
In my host directory I have two samples’ fastq.gz files, named as:
A29_WES_S3_R1_001.fastq.gz
A29_WES_S3_R2_001.fastq.gz
A30_WES_S1_R1_001.fastq.gz
A30_WES_S1_R2_001.fastq.gz
Now I need to have their basename without suffix like:
A29_WES_S3_R1_001
A29_WES_S3_R2_001
A30_WES_S1_R1_001
A30_WES_S1_R2_001
I used the bash pipeline as follow:
#!/bin/bash
FILES1=(*R1_001.fastq.gz)
FILES2=(*R2_001.fastq.gz)
read1="${FILES1[#]}"
read2="${FILES2[#]}"
Ffile=$read1
Ffileprevix=$(basename "$Ffile" .fastq.gz)
Mfile=$read2
Mfileprevix=$(basename "$Mfile" .fastq.gz)
echo $Ffileprevix
echo $Mfileprevix
exit;
But every time I just get this output:
A29_WES_S3_R1_001.fastq.gz A30_WES_S1_R1_001
A29_WES_S3_R2_001.fastq.gz A30_WES_S1_R2_001
Only the last file (A30) would be included in the command!
I checked my pipeline in this way:
echo $read1
echo $read2
The result:
A29_WES_S3_R1_001.fastq.gz A30_WES_S1_R1_001.fastq.gz
A29_WES_S3_R2_001.fastq.gz A30_WES_S1_R2_001.fastq.gz
Then I did:
echo $Ffile
echo $Mfile
The result:
A29_WES_S3_R1_001.fastq.gz A30_WES_S1_R1_001.fastq.gz
A29_WES_S3_R2_001.fastq.gz A30_WES_S1_R2_001.fastq.gz
So $read1, $read2, $Ffile, and $Mfile work well.
Then I put “-a” in my basename command as it will take multiple files:
Ffileprevix=$(basename -a "$Ffile" .fastq.gz)
Mfileprevix=$(basename -a "$Mfile" .fastq.gz)
But it got worse! The result was like:
A29_WES_S3_R1_001.fastq.gz A30_WES_S1_R1_001.fastq.gz .fastq.gz
A29_WES_S3_R2_001.fastq.gz A30_WES_S1_R2_001.fastq.gz .fastq.gz
Finally, I tried “for ..... do ....” command to make a loop for basename command. Again, nothing changed!!
Is there anybody can help me to obtain what I want:
A29_WES_S3_R1_001
A29_WES_S3_R2_001
A30_WES_S1_R1_001
A30_WES_S1_R2_001
I'd leave basename out of this entirely, but that's entirely personal preference. You could do something more like:
FILES_PATTERN_1=".*R1_001.fastq.gz"
FILES_PATTERN_2=".*R2_001.fastq.gz"
# Get FILE PATTERN 1
echo "Pattern 1:"
for FILE in $(find . | grep "${FILES_PATTERN_1}" | cut -d. -f2 | tr -d /); do
echo $FILE
done
# Get FILE PATTERN 2
echo "Pattern 2:"
for FILE in $(find . | grep "${FILES_PATTERN_2}" | cut -d. -f2 | tr -d /); do
echo $FILE
done
Output should be:
Pattern 1:
A30_WES_S1_R1_001
A29_WES_S3_R1_001
Pattern 2:
A29_WES_S3_R2_001
A30_WES_S1_R2_001
You could also play with awk to parse things instead:
# Get FILE PATTERN 1
echo "Pattern 1:"
for FILE in $(find . | grep "${FILES_PATTERN_1}" | awk -F '[/.]' '{print $3}'); do
echo $FILE
done
There are a number of ways to approach this. If you had a lot more patterns to test you could make more use of functions here to reduce code duplication.
Also note, I'm doing this from a shell on Mac OSX, so if you're doing this from a Linux box some of these commands may need to be tweaked due to differences in output for some commands, like find. (ex: print $1 instead of print $3)
I am trying to write a script that parses the output of a URL. Everything works great if I hard code a URL into the curl command, however when I try to add the URL from an array I am getting a syntax error.
Here is the script
#!/usr/bin/bash
hosts=(
room-f3185-dsp
room-c5180-dsp
room-f3187-dsp
room-f3183-dsp
)
for i in "${hosts[#]}"
url1=$(curl -d "action=login&userPassword=PASSWORD" -H "Content-Type: application/x-www-form-urlencoded" -X POST http://$i)
url2=$(echo $url | sed 's/,/\n/g')
url3=$(echo $url2 | sed 's/:/ /g' | grep macAddress ./mactest | awk '{print $2,$3,$4,$5,$6,$7,$8}')
url4=$(echo $url3 | sed 's/ /:/g' | tr -d '"')
mac=$(echo $url4 | sed 's/.$//' | sed 's/.$//' |sed 's/.$//')
echo "------------------------------------------------" >> dspmac
echo $i >> dspmac
echo "Mac Address = "$mac >> dspmac
done
Here is the error I am getting,
line 10: syntax error near unexpected token `url1=$(curl -d "action=login&userPassword=PASSWORD" -H "Content-Type: application/x-www-form-urlencoded" -X POST $i)'
Also in the url1 line when I put in HTTP:// it is highlighted as a link, but the $i is just showing as standard text
Thank in advance
If I replace the $i in the script with the full URL (http://room-f3185-dsp) it works great. I have also tried putting the full url as the host in the array, but still get the same error.
I also tried various quotes in different places.
How about adding do:
for i in "${hosts[#]}"
do
...
done
I'm trying to improve my scripting skills.
I know there's online converters that are doing it but i'd like to make a script that compare in a certain folder (/opt/Citrix/ICAClient/keystore/cacerts/), if the .crt are already in the .pem format and if not convert them.
So i know this clearly isn't the best way to do so... But i've started like this:
`
#!/bin/bash
#remove both files if they already exists
rm crt.txt
rm pem.txt
#certificate selection in .crt format
Certificate_crt=$(sudo ls /opt/Citrix/ICAClient/keystore/cacerts/ | grep .crt | sed 's/\(.*\)\..*/\1/')
#certificate selection in .pem format
Certificate_pem=$(sudo ls /opt/Citrix/ICAClient/keystore/cacerts/ | grep .pem | sed 's/\(.*\)\..*/\1/')
#sending results in text files
echo "$Certificate_crt" >> crt.txt
echo "$Certificate_pem" >> pem.txt
#recovery of certificate names in .crt not having a .pem equivalent
Certificate_crt_WO_pem=$(cat crt.txt | grep -v "$Certificate_pem" | tr " " "\n")
#echo "$Certificate_crt_WO_pem"
#initialisation
i=0
#Print the split string
for i in "${Certificate_crt_WO_pem[#]}"
do
name_certificate=$(echo $i | tr " " "\n")
echo "${name_certificate[#]}"
echo "$i"
i=i+1
done
`
The thing is that when my "for" is launched, it stores all the result of "Certificate_crt_WO_pem" in the array $name_certificate[0] and then stop it self.
What i want is to store, line by line, the result of "cat crt.txt | grep -v "$Certificate_pem" | tr " " "\n"" into the array name_certificate.
This array will be use to launch something like this " openssl -in $name_certificate[$i].crt -out $name_certificate[$i].pem PEM" (in a for loop in the will to convert every namefile.crt in every namefile.pem).
If someone can help me i'll be more than gratefull... (And yes i've already tried to search on the net, had followed some online courses but none of them was saying the same thing about the bash's arrays, so i'm a bit lost...)
What i want is to store, line by line, the result of
Then do that.
var=$(something)
# ^^ - normal variable assignment
var=( $(something) )
# ^^ ^ - array assignment
# the unquoted result of expansions is split on IFS
# default on tabs, newlines and spaces
so I guess you want:
Certificate_crt_WO_pem=($(grep -v "$Certificate_pem" crt.txt))
Doing cat file | grep is a useless use of cat. Use grep .. file or < file grep ....
Remember do not parse ls output. Don't ls | something. Prefer globulation or find instead.
Read how to read a stream line by line in bashfaq. Read how to use arrays in bashfaq.
Note that grep parses a regex, so grep .pem matches any character followed by pem. I guess you wanted grep '\.pem'. I do not think grep -v "$Certificate_pem" does what you think it does - I guess you meant to use comm or join to filter elements from one newline separated separated that are present in the other list.
This array will be use to launch something like this " openssl -in $name_certificate[$i].crt -out $name_certificate[$i].pem PEM"
The best would be not do it, rather parse the data as they come, not store them in variables.
# get all .crt fiels
find /opt/Citrix/ICAClient/keystore/cacerts/ -maxdepth 1 -mindepth 1 -type f -name '*.crt' |
# remove extension
sed 's/\.crt$//' |
# filter out files where .pem does exists already
while IFS= read -r file; do
if [[ -r "$file".pem ]]; then continue; fi
printf "%s\n" "$file"
done |
# execute openssl for the rest
xargs -d'\n' -i{} openssl -in {}.crt -out {}.pem PEM
but if you want, rather use mapfile to store a string with a newline separated list into an array (be sure to understand how to store variables in a pipeline).
mapfile -t Certificate_crt_WO_pem < <(something)
I'm sysadmin for couple of webservers deployed with cpanel hosting panel. I'm trying to finish up with a backup script. There are two commands bundled with Cpanel, that will be used in this script. These commands are;
1. whmapi1 modifyacct user=USERNAME BACKUP=[01]
This Command has booleans to set, what it does is either enable or
disable backup for a specific user.
2. /usr/local/cpanel/bin/backup --force
Once backup is enabled for a user/users, then this command starts the
backup process on the server.
So here is my script logic & the script.
#!/bin/bash
Arrays
L=($( comm -23 <(du -h --max-depth=1 /home 2>/dev/null | grep G | awk -F"/" '{print $NF}' | sort | egrep -vw '(home|virtfs)') <(ls -al /var/cpanel/suspended/ | grep -v 'lock' | sort) ))
Above Array contains all the account whose home directories have
exceeded 1GB limit.
S=($(comm -23 <(du -h --max-depth=1 /home 2>/dev/null | egrep -v '(!G|.cp|cP|clamav)' | awk -F"/" '{print $NF}' | sort | egrep -vw '(home|virtfs)') <(ls -al /var/cpanel/suspended/ | grep -v 'lock' | sort) ))
Above Array contains all the account whose home directories are less
than 1GB limit.
whmapi1 modifyacct user=${L[#]} BACKUP=0 && whmapi1 modifyacct user=${S[#]} BACKUP=0
Above command disables backup for all users for start, to start from
scracth.
whmapi1 modifyacct user=${S[#]} BACKUP=1
T
his command enables backup for all accounts whose home dirs are less
than 1 GB
/usr/local/cpanel/bin/backup --force
This command starts backup process for all enabled users.
The logic is, that I want to create backup of small accounts first, and then when it's finished, I'll run it for larger accounts.
PROBLEM: all commands execute successfully when run directly in terminal, but it doesn't when run via a script. Problem occurs at account enabling & disabling.
It either disables all or enables all, and not the partial accounts, as intended by the logic of the script.
Can anyone point out, where & what I'm missing?
Thanks in advance !!
${l[#]} exands to user1 user2 user3 ..., so user=${L[#]} expands to user=user1 user2 user3 ..., if you want to fun foreach user, you need to loop over users.
du_buff=$(du -h --max-depth=1 /home 2>/dev/null)
lock_buff=$(ls -al /var/cpanel/suspended/ | grep -v 'lock' | sort)
L=($(comm -23 <(echo "$du_buff" | grep G | awk -F"/" '{print $NF}' | sort | egrep -vw '(home|virtfs)') <(echo "$lock_buff") ))
S=($(comm -23 <(echo "$du_buff" | egrep -v '(!G|.cp|cP|clamav)' | awk -F"/" '{print $NF}' | sort | egrep -vw '(home|virtfs)') <(echo "$lock_buff") ))
# for every user in L and S
for user in "${L[#]}" "${S[#]}"; do
whmapi1 modifyacct user=$user BACKUP=0
done
# for every user in S
for user in "${S[#]}"; do
whmapi1 modifyacct user=$user BACKUP=1
done
/usr/local/cpanel/bin/backup --force
I am struggling with passing several grep patterns that are contained within a variable. This is the code I have:
#!/bin/bash
GREP="$(which grep)"
GREP_MY_OPTIONS="-c"
for i in {-2..2}
do
GREP_MY_OPTIONS+=" -e "$(date --date="$i day" +'%Y-%m-%d')
done
echo $GREP_MY_OPTIONS
IFS=$'\n'
MYARRAY=( $(${GREP} ${GREP_MY_OPTIONS} "/home/user/this path has spaces in it/"*"/abc.xyz" | ${GREP} -v :0$ ) )
This is what I wanted it to do:
determine/define where grep is
assign a variable (GREP_MY_OPTIONS) holding parameters I will pass to grep
assign several patterns to GREP_MY_OPTIONS
using grep and the patterns I have stored in $GREP_MY_OPTIONS search several files within a path that contains spaces and hold them in an array
When I use "echo $GREP_MY_OPTIONS" it is generating what I expected but when I run the script it fails with an error of:
/bin/grep: invalid option -- ' '
What am I doing wrong? If the path does not have spaces in it everything seems to work fine so I think it is something to do with the IFS but I'm not sure.
If you want to grep some content in a set of paths, you can do the following:
find <directory> -type f -print0 |
grep "/home/user/this path has spaces in it/\"*\"/abc.xyz" |
xargs -I {} grep <your_options> -f <patterns> {}
So that <patterns> is a file containing the patterns you want to search for in each file from directory.
Considering your answer, this shall do what you want:
find "/path\ with\ spaces/" -type f | xargs -I {} grep -H -c -e 2013-01-17 {}
From man grep:
-H, --with-filename
Print the file name for each match. This is the default when
there is more than one file to search.
Since you want to insert the elements into an array, you can do the following:
IFS=$'\n'; array=( $(find "/path\ with\ spaces/" -type f -print0 |
xargs -I {} grep -H -c -e 2013-01-17 "{}") )
And then use the values as:
echo ${array[0]}
echo ${array[1]}
echo ${array[...]}
When using variables to pass the parameters, use eval to evaluate the entire line. Do the following:
parameters="-H -c"
eval "grep ${parameters} file"
If you build the GREP_MY_OPTIONS as an array instead of as a simple string, you can get the original outline script to work sensibly:
#!/bin/bash
path="/home/user/this path has spaces in it"
GREP="$(which grep)"
GREP_MY_OPTIONS=("-c")
j=1
for i in {-2..2}
do
GREP_MY_OPTIONS[$((j++))]="-e"
GREP_MY_OPTIONS[$((j++))]=$(date --date="$i day" +'%Y-%m-%d')
done
IFS=$'\n'
MYARRAY=( $(${GREP} "${GREP_MY_OPTIONS[#]}" "$path/"*"/abc.xyz" | ${GREP} -v :0$ ) )
I'm not clear why you use GREP="$(which grep)" since you will execute the same grep as if you wrote grep directly — unless, I suppose, you have some alias for grep (which is then the problem; don't alias grep).
You can do one thing without making things complex:
First do a change directory in your script like following:
cd /home/user/this\ path\ has\ spaces\ in\ it/
$ pwd
/home/user/this path has spaces in it
or
$ cd "/home/user/this path has spaces in it/"
$ pwd
/home/user/this path has spaces in it
Then do what ever your want in your script.
$(${GREP} ${GREP_MY_OPTIONS} */abc.xyz)
EDIT :
[sgeorge#sgeorge-ld stack1]$ ls -l
total 4
drwxr-xr-x 2 sgeorge eng 4096 Jan 19 06:05 test tesd
[sgeorge#sgeorge-ld stack1]$ cat test\ tesd/file
SUKU
[sgeorge#sgeorge-ld stack1]$ grep SUKU */file
SUKU
EDIT :
[sgeorge#sgeorge-ld stack1]$ find */* -print | xargs -I {} grep SUKU {}
SUKU