How to substract two stdout lists in linux bash - arrays

heed help.
I have one list "A" from
netstat -ntlp | grep -oP ":[:1]?[:1]?(.*)+" | grep -oP "\d\d+"
it looks like
80
443
8080
22
25
I have another list "B" from
ufw status numbered | grep -oP "\] \d+" | grep -oP "\d+"
it looks as
80
443
22
So i want to know, which ports are listening, but not open with ufw, i.e. substract ["A"]-["B"]
and going to see
8080
25
with some command like
netstat -ntlp | grep -oP ":[:1]?[:1]?(.*)+" | grep -oP "\d\d+" | SELECT ALL NOT IN `ufw status numbered | grep -oP "\] \d+" | grep -oP "\d+"`
How to do this?

Typically it's comm job:
netstat -ntlp | grep -oP ":[:1]?[:1]?(.*)+" | grep -oP "\d\d+" |
sort | comm -23 - <(ufw status numbered | grep -oP "\] \d+" | grep -oP "\d+" | sort)

You may use grep:
grep -vxFf <(cmd2) <(cmd1)
Here replace cmd1 with netstat ... command and replace cmd2 with ufw ... command.

This solution requires pre-sorting of the outputs:
$ netstat -ntlp | grep -oP ":[:1]?[:1]?(.*)+" | grep -oP "\d\d+" | sort > A
^^^^^^
$ ufw status numbered | grep -oP "\] \d+" | grep -oP "\d+" | sort > B
^^^^^^
Items unique to A:
$ comm -23 A B
25
8080
$
... but also, in case you require, items unique to B:
$ comm -13 A B
$
... and items common to A and B:
$ comm -12 A B
22
443
80
$
See man comm for details.

You can check the uniq -u command:
http://man7.org/linux/man-pages/man1/uniq.1.html
You pass a group of lines to uniq -d and redirect to an output. It will print only the duplicated ones.
So you just need to aggregate both results from list A and list B into a text:
List A:
netstat -ntlp | grep -oP ":[:1]?[:1]?(.*)+" | grep -oP "\d\d+" >> output.txt
List B:
ufw status numbered | grep -oP "\] \d+" | grep -oP "\d+" > output.txt >> output.txt`
(NOTE: You use '>>' over '>' to append the content to end of the file. So make sure to clean it on each iteration!)
Then:
uniq -u output.txt
You can redirect the uniq -u output too, if needed:
uniq -u output.txt > gotuniques.txt
Edit: formatting
Edit2: I was confused by -d when the answer requires -u.

Related

In side the loop if the id is 09 (ex: f132a09) it will skip that student but process all other students

I know i need to use continue but just don't know where to start. Any help would be appreciated.
Not looking for an answer, just a clearer understanding
Here's my code so far
prefix=$1
for id in $(grep "^$prefix" /etc/passwd | cut -d: -f1 | sort)
do
echo -e "$(grep "^${id}:" /etc/passwd | cut -d: -f5 | sed "s/^\(.*\), \(.*\)$/\2 \1 /g") \c"
echo -e " has the cisweb id ${id}"
if who | grep "^${id} " > /dev/null
then
echo -e "$(grep -w "^$prefix" /etc/passwd | cut -d: -f5 | uniq | sed 's/^\(.*\), \(.*\)$/\2 \1 /g' )is currently logged on"
fi
done
I've tried several Mods but o luck.

Concatenate outputs of the commands

Can someone please help me to concatenate the outputs of the two commands?
finger | awk '{print $2,$3}' | uniq | sed '1d'
system_profiler SPHardwareDataType | awk '/Serial/{print $NF}'
The output should be firstnamelastname.Serialnumber.local
you can affect result of two commands in variables to be able to concatenate them in a result one
first=$(finger | awk '{print $2,$3}' | uniq | sed '1d')
second=$(system_profiler SPHardwareDataType | awk '/Serial/{print $NF}')
result="$first.$second"
echo $result;

bash scripting , loop not looping

I wrote a while loop to search inside files and append the output to a text file , but it seems like it's reading only the first line of that text file . How do I fix it ?
while read line
do
x=`echo $line`
y=`grep $x: /etc/group | cut -d ":" -f 3`
grep $y /etc/passwd | cut -d ":" -f 1 >> users
grep $y /etc/group | cut -d ":" -f 4 | tr "," "\n" >> users
done < filename
Perhaps you need to wrap $x and $y in quotes, as otherwise grep may interpret anything after the first space as the name of the file to be searched:
#!/bin/bash
while read line
do
x=`echo $line`
y=`grep "$x:" /etc/group | cut -d ":" -f 3`
grep "$y" /etc/passwd | cut -d ":" -f 1 >> users
grep "$y" /etc/group | cut -d ":" -f 4 | tr "," "\n" >> users
done < filename
This might be a bit safer as some of the grep statements may pick up the wrong fields (i.e. it does not check for the correct field):
while read GROUP
do
GROUP_ID=`grep ^$GROUP: /etc/group | cut -d ":" -f 3`
USER_ENT=`grep -e '\(.*:\)\{3\}'$GROUP_ID':' /etc/passwd`
[ $? -eq 0 ] && cut -d ":" -f 1 <<<$USER_ENT
GROUP_ENT=`grep -e '\(.*:\)\{2\}'$GROUP_ID':' /etc/group`
[ $? -eq 0 ] && cut -d ":" -f 4 <<<$GROUP_ENT | tr "," "\n" | grep -v ^$
done < $FILE_NAME | sort | uniq >users

Define array for walking directories with du

I wrote a basic script to test a recursive du output against a directory or filesystem that selects the largest directory and repeats, then outputs the results neatly. Is there a way I can combine an array and some if/then statements to make this more elegant and continue to recurse until no more directories are matched, then printing the outputs from an array?
#!/bin/bash
dir1=$1
du1=$(du -x --max-depth=1 $dir1 | sort -nr | awk '{ print $2 }' | \
xargs du -hx --max-depth=0 | egrep -v "sys|proc|boot|lost|media|mnt|selinux" | head -10 | tail -n +2)
dir2=$(echo "$du1"|head -1|awk '{print $2}')
du2=$(du -x --max-depth=1 $dir2 | sort -nr | awk '{ print $2 }' | \
xargs du -hx --max-depth=0 | egrep -v "sys|proc|boot|lost|media|mnt|selinux" | head -10 | tail -n +2)
dir3=$(echo "$du2"|head -1|awk '{print $2}')
du3=$(du -x --max-depth=1 $dir3 | sort -nr | awk '{ print $2 }' | \
xargs du -hx --max-depth=0 | egrep -v "sys|proc|boot|lost|media|mnt|selinux" | head -10 | tail -n +2)
dir4=$(echo "$du3"|head -1|awk '{print $2}')
du4=$(du -x --max-depth=1 $dir4 | sort -nr | awk '{ print $2 }' | \
xargs du -hx --max-depth=0 | egrep -v "sys|proc|boot|lost|media|mnt|selinux" | head -10 | tail -n +2)
dir5=$(echo "$du4"|head -1|awk '{print $2}')
du5=$(du -x --max-depth=1 $dir5 | sort -nr | awk '{ print $2 }' | \
xargs du -hx --max-depth=0 | egrep -v "sys|proc|boot|lost|media|mnt|selinux" | head -10 | tail -n +2)
dir6=$(echo "$du5"|head -1|awk '{print $2}')
du6=$(du -x --max-depth=1 $dir6 | sort -nr | awk '{ print $2 }' | \
xargs du -hx --max-depth=0 | egrep -v "sys|proc|boot|lost|media|mnt|selinux" | head -10 | tail -n +2)
echo -e "##LEVEL1##"
paste -d ' ' <(echo "$du1") <(echo "$(file $(echo "$du1" | \
awk '{print $2}')|cut -d' ' -f2- | sed -e 's/[a-zA-Z0-9]/[&/' -e 's/$/]/')")
echo -e "##LEVEL2##"
paste -d ' ' <(echo "$du2") <(echo "$(file $(echo "$du2" | \
awk '{print $2}')|cut -d' ' -f2- | sed -e 's/[a-zA-Z0-9]/[&/' -e 's/$/]/')")
echo -e "##LEVEL3##"
paste -d ' ' <(echo "$du3") <(echo "$(file $(echo "$du3" | \
awk '{print $2}')|cut -d' ' -f2- | sed -e 's/[a-zA-Z0-9]/[&/' -e 's/$/]/')")
echo -e "##LEVEL4##"
paste -d ' ' <(echo "$du4") <(echo "$(file $(echo "$du4" | \
awk '{print $2}')|cut -d' ' -f2- | sed -e 's/[a-zA-Z0-9]/[&/' -e 's/$/]/')")
echo -e "##LEVEL5##"
paste -d ' ' <(echo "$du5") <(echo "$(file $(echo "$du5" | \
awk '{print $2}')|cut -d' ' -f2- | sed -e 's/[a-zA-Z0-9]/[&/' -e 's/$/]/')")
echo -e "##LEVEL6##"
paste -d ' ' <(echo "$du6") <(echo "$(file $(echo "$du6" | \
awk '{print $2}')|cut -d' ' -f2- | sed -e 's/[a-zA-Z0-9]/[&/' -e 's/$/]/')")
Here is an example output:
#./rdu.sh / 2>/dev/null
##LEVEL1##
12G /opt [directory]
1.9G /usr [directory]
452M /var [directory]
352M /root [directory]
179M /home [directory]
116M /lib [directory]
46M /tmp [sticky directory]
28M /sbin [directory]
21M /etc [directory]
##LEVEL2##
8.5G /opt/zenoss [directory]
2.9G /opt/zends [directory]
##LEVEL3##
6.6G /opt/zenoss/perf [directory]
510M /opt/zenoss/ZenPacks [directory]
486M /opt/zenoss/var [directory]
461M /opt/zenoss/lib [directory]
250M /opt/zenoss/log [directory]
85M /opt/zenoss/Products [directory]
49M /opt/zenoss/packs [directory]
31M /opt/zenoss/share [directory]
26M /opt/zenoss/webapps [directory]
##LEVEL4##
6.5G /opt/zenoss/perf/Devices [directory]
59M /opt/zenoss/perf/Daemons [directory]
##LEVEL5##
289M /opt/zenoss/perf/Devices/10.0.4.218 [directory]
288M /opt/zenoss/perf/Devices/10.215.68.9 [directory]
287M /opt/zenoss/perf/Devices/10.0.4.18 [directory]
161M /opt/zenoss/perf/Devices/<removed> [directory]
145M /opt/zenoss/perf/Devices/10.219.68.12 [directory]
143M /opt/zenoss/perf/Devices/VMs-- [directory]
143M /opt/zenoss/perf/Devices/10.0.4.219 [directory]
143M /opt/zenoss/perf/Devices/10.0.4.19 [directory]
136M /opt/zenoss/perf/Devices/10.215.68.8 [directory]
##LEVEL6##
279M /opt/zenoss/perf/Devices/10.0.4.218/ltmvirtualservers [directory]
7.1M /opt/zenoss/perf/Devices/10.0.4.218/os [directory]
888K /opt/zenoss/perf/Devices/10.0.4.218/hw [directory]
840K /opt/zenoss/perf/Devices/10.0.4.218/loadbalancerports [directory]
You code doesn't work on my system, so I cannot test it. But you can do something like this:
function durec {
dir1=$1
level=$2
du1=$(du -x --max-depth=1 $dir1 | sort -nr | awk '{ print $2 }' | \
xargs du -hx --max-depth=0 | egrep -v "sys|proc|boot|lost|media|mnt|selinux" | head -10 | tail -n +2)
echo -e "##LEVEL$level##"
paste -d ' ' <(echo "$du1") <(echo "$(file $(echo "$du1" | \
awk '{print $2}')|cut -d' ' -f2- | sed -e 's/[a-zA-Z0-9]/[&/' -e 's/$/]/')")
let level++
dir2=$(echo "$du1"|head -1|awk '{print $2}')
if [ ! -z "$dir" ]; then
durec $dir2 $level
fi
}
# call the function
durec / 1
Your 1st du counts the whole filesystem. After you counts again and again for each subdir.
This for me seems as a couple of unnecessary counting (read nonsense), because you can save the output from the 1st du and works only on it...
Something like:
root="${1:-.}"
count=${2:-10}
tmp1=/tmp/durec_du.$$
tmp2=/tmp/durec_tmp.$$
trap "rm -f $tmp1 $tmp2;exit" 0 1 2 3 15
#human readable format - need GNU sort
#du -h "$root" | gsort -hr > $tmp1
#KB format
du -k "$root" | sort -nr > $tmp1
cp /dev/null $tmp2
level=0
durec() {
dir=$1
biggest=$(grep " ${dir}/[^/][^/]*$" $tmp1 | tee $tmp2 | head -1 | sed 's/^[0-9BKMGTP][0-9BKMGTP]* //')
# ^^^ ----------------------- one <TAB> character ---------------------------------- ^^^^
# if you have GNU version of sed, and grep replace the <TAB> with \t
[[ -n "$biggest" ]] || return
let level++
echo "##LEVEL$level##"
head -$count $tmp2
durec "$biggest"
}
durec "$root"
The gsort command is the GNU sort. If your standard sort is GNU, replace the gsort with a simple sort. (Need for the -h - sorting the result of du -h.

Find duplicate lines in a file and count how many time each line was duplicated?

Suppose I have a file similar to the following:
123
123
234
234
123
345
I would like to find how many times '123' was duplicated, how many times '234' was duplicated, etc.
So ideally, the output would be like:
123 3
234 2
345 1
Assuming there is one number per line:
sort <file> | uniq -c
You can use the more verbose --count flag too with the GNU version, e.g., on Linux:
sort <file> | uniq --count
This will print duplicate lines only, with counts:
sort FILE | uniq -cd
or, with GNU long options (on Linux):
sort FILE | uniq --count --repeated
on BSD and OSX you have to use grep to filter out unique lines:
sort FILE | uniq -c | grep -v '^ *1 '
For the given example, the result would be:
3 123
2 234
If you want to print counts for all lines including those that appear only once:
sort FILE | uniq -c
or, with GNU long options (on Linux):
sort FILE | uniq --count
For the given input, the output is:
3 123
2 234
1 345
In order to sort the output with the most frequent lines on top, you can do the following (to get all results):
sort FILE | uniq -c | sort -nr
or, to get only duplicate lines, most frequent first:
sort FILE | uniq -cd | sort -nr
on OSX and BSD the final one becomes:
sort FILE | uniq -c | grep -v '^ *1 ' | sort -nr
To find and count duplicate lines in multiple files, you can try the following command:
sort <files> | uniq -c | sort -nr
or:
cat <files> | sort | uniq -c | sort -nr
Via awk:
awk '{dups[$1]++} END{for (num in dups) {print num,dups[num]}}' data
In awk 'dups[$1]++' command, the variable $1 holds the entire contents of column1 and square brackets are array access. So, for each 1st column of line in data file, the node of the array named dups is incremented.
And at the end, we are looping over dups array with num as variable and print the saved numbers first then their number of duplicated value by dups[num].
Note that your input file has spaces on end of some lines, if you clear up those, you can use $0 in place of $1 in command above :)
In Windows, using "Windows PowerShell", I used the command mentioned below to achieve this
Get-Content .\file.txt | Group-Object | Select Name, Count
Also, we can use the where-object Cmdlet to filter the result
Get-Content .\file.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select Name, Count
To find duplicate counts, use this command:
sort filename | uniq -c | awk '{print $2, $1}'
Assuming you've got access to a standard Unix shell and/or cygwin environment:
tr -s ' ' '\n' < yourfile | sort | uniq -d -c
^--space char
Basically: convert all space characters to linebreaks, then sort the tranlsated output and feed that to uniq and count duplicate lines.

Resources