How to merge duplicate entries that produced by for loop - arrays

Following my previous question which got closed— basically I have a script that check availability of packages on target server, the target server and the packages have been stored to an array.
declare -a prog=("gdebi" "firefox" "chromium-browser" "thunar")
declare -a snap=("beer2" "beer3")
# checkvar=$(
for f in "${prog[#]}"; do
for connect in "${snap[#]}"; do
ssh lg#"$connect" /bin/bash <<- EOF
if dpkg --get-selections | grep -qE "(^|\s)"$f"(\$|\s)"; then
status="[INSTALLED] [$connect]"
else
status=""
fi
printf '%s %s\n' "$f" "\$status"
EOF
done
done
With the help of fellow member here, I've made several fix to original script, script ran pretty well— except there's one problem, the output contain duplicate entries.
gdebi [INSTALLED] [beer2]
gdebi
firefox [INSTALLED] [beer2]
firefox [INSTALLED] [beer3]
chromium-browser [INSTALLED] [beer2]
chromium-browser [INSTALLED] [beer3]
thunar
thunar
I know it this is normal behavior, as for pass multiple server from snap array, making ssh travel to all the two server.
Considering that the script checks same package for two server, I want the output to be merged.
If beer2 have firefox packages, but beer3 doesn't.
firefox [INSTALLED] [beer2]
If beer3 have firefox packages, but beer2 doesn't.
firefox [INSTALLED] [beer3]
If both beer2 and beer3 have the packages.
firefox [INSTALLED] [beer2, beer3]
or
firefox [INSTALLED] [beer2] [beer3]
If both beer2 and beer3 doesn't have the package, it will return without extra parameter.
firefox
Sound like an easy task, but for the love of god I can't find how to achieve this, here's list of things I have tried.
Try to manipulate the for loops.
Try putting return value after one successful loops (exit code).
Try nested if.
All of the above doesn't seem to work, I haven't tried changing/manipulate the return string as I'm not really experienced with some text processing such as: awk, sed, tr and many others.
Can anyone shows how It's done ? Would really mean the world to me.

Pure Bash 4+ solution using associative array to store hosts the program is installed on:
#!/usr/bin/env bash
declare -A hosts_with_package=(["gdebi"]="" ["firefox"]="" ["chromium-browser"]="" ["thunar"]="")
declare -a hosts=("beer2" "beer3")
# Collect installed status
# Iterate all hosts
for host in "${hosts[#]}"; do
# Read the output of dpkg --get-selections with searched packages
while IFS=$' \t' read -r package status; do
# Test weather package is installed on host
if [ "$status" = "install" ]; then
# If no host listed for package, create first entry
if [ -z "${hosts_with_package[$package]}" ]; then
# Record the first host having the package installed
hosts_with_package["$package"]="$host"
else
# Additional hosts are concatenated as CSV
hosts_with_package["$package"]="${hosts_with_package[$package]}, $host"
fi
fi
# Feed the whole loop with the output of the dpkg --get-selections for packages names
# Packages names are the index of the hosts_with_package array
done < <(ssh "lg#$host" dpkg --get-selections "${!hosts_with_package[#]}")
done
# Output results
# Iterate the package name keys
for package in "${!hosts_with_package[#]}"; do
# Print package name without newline
printf '%s' "$package"
# If package is installed on some hosts
if [ -n "${hosts_with_package[$package]}" ]; then
# Continue the line with installed hosts
printf ' [INSTALLED] [%s]' "${hosts_with_package[$package]}"
fi
# End with a newline
echo
done

Instead of making several ssh connections in nested loops consider this change
prog=( mysql-server apache2 php ufw )
snap=( localhost )
for connect in ${snap[#]}; do
ssh $connect "
progs=( ${prog[#]} )
for prog in \${progs[#]}; do
dpkg -l | grep -q \$prog && echo \"\$prog [INSTALLED]\" || echo \"\$prog\"
done
"
done

Based on #Ivan answer
#!/bin/bash
prog=( "gdebi" "firefox" "chromium-browser" "thunar" )
snap=( "beer2" "beer3" )
# First, retrieve the list on installed program for each host
for connect in ${snap[#]}; do
ssh lg#"$connect" /bin/bash >/tmp/installed.${connect} <<- EOF
progs=( "${prog[#]}" )
for prog in \${progs[#]}; do
dpkg --get-selections | awk -v pkg=\$prog '\$1 == pkg && \$NF ~ /install/ {print \$1}'
done
EOF
done
# Filter the previous results to format the output as you need
awk '{
f = FILENAME;
gsub(/.*\./,"",f);
a[$1] = a[$1] "," f
}
END {
for (i in a)
print i ":[" substr(a[i],2) "]"
}' /tmp/installed.*
rm /tmp/installed.*
Example of output :
# With prog=( bash cat sed tail something firefox-esr )
firefox-esr:[localhost]
bash:[localhost,localhost2]
sed:[localhost,localhost2]

Related

How can I cycle through an array in bash while also passing an argument to the script?

I have the following bash script I want to use as my "standard browser" with xdg-open.
It should prompt dmenu for me to choose the browser to open the url in.
Now xdg-open passes the url as an argument to the program (I suppose) and as I'm cycling through an array of browsers using the # symbol, it confuses this one with the argument (url) and errors on the dmenu command.
Is there a workaround to this problem or am I doing something completely wrong? --This problem was solved
#!/usr/bin/env bash
set -euo pipefail
_path="$(cd "$(dirname "${BASH_SOURCE[0]}")" && cd "$(dirname "$(readlink "${BASH_SOURCE[0]}" || echo ".")")" && pwd)"
if [[ -f "${_path}/_dm-helper.sh" ]]; then
# shellcheck disable=SC1090,SC1091
source "${_path}/_dm-helper.sh"
else
# shellcheck disable=SC1090
echo "No helper-script found"
fi
# script will not hit this if there is no config-file to load
# shellcheck disable=SC1090
source "$(get_config)"
main() {
if [ -t 0 ]
then
_url=$1
else
read _url
fi
_browser=$(printf '%s\n' "${!browsers[#]}" | sort | ${DMENU} 'Select browser: ') # "$#") ## Thx to #jhnc
_command=${browsers[${_browser}]}
if [[ -n ${_url} ]];then
$_command $_url
fi
}
[[ "${BASH_SOURCE[0]}" == "${0}" ]] && main "$#"
(get config) loads the dmenu command:
DMENU=dmenu -i -l 20 -p
as well as the array of browsers:
declare -A browsers
browsers[brave]="brave-browser"
browsers[firefox]="firefox"
browsers[opera]="opera"
browsers[badwolf]="badwolf"
from my config file.
Originally if i ran xdg-open "https://" or if I clicked on a url in some other program, brave was opened with on that site.
Now after xdg-settings set default-web-browser dmenu-script.desktop with the following .desktop file:
[Desktop Entry]
Version=1.0
Name=Dmenu Browser Script
GenericName=Web Browser
# Gnome and KDE 3 uses Comment.
Comment=Access the Internet
Exec=$HOME/.local/bin/dmenu-browser %U
StartupNotify=true
Terminal=false
Icon=brave-browser
Type=Application
Categories=Network;WebBrowser;
MimeType=application/pdf;application/rdf+xml;application/rss+xml;application/xhtml+xml;application/xhtml_xml;application/xml;image/gif;image/jpeg;image/png;image/webp;text/html;text/xml;x-scheme-handler/http;x-scheme-handler/https;x-scheme-handler/ipfs;x-scheme-handler/ipns;
Actions=new-window;new-private-window;
[Desktop Action new-window]
Name=New Window
Exec=$HOME/.local/bin/dmenu-browser
[Desktop Action new-private-window]
Name=New Incognito Window
Exec=$HOME/.local/bin/dmenu-browser --incognito
It only works if I execute xdg-open from my command line. (I modified the .desktop file of brave-browser, because I had no clue how to write one.)

exclude stdout based on an array of words

Question:
Is it possible to exclude some of a commands output its output based on an array of words?**
Why?
On Ubuntu/Debian there are two ways to list all available pkgs:
apt list (show all available pkgs, installed pkgs too)
apt-cache search . (show all available pkgs, installed pkgs too)
Difference is, the first command, you can exclude all installed pkgs using grep -v, problem is unlike the first, the second command you cannot exclude these as the word "installed" isnt present. Problem with the first command is it doesnt show the pkg description, so I want to use apt-cache search . but excluding all installed pkgs.
# List all of my installed pkgs,
# Get just the pkg's name,
# Swap newlines for spaces,
# Save this list as an array for exclusion.
INSTALLED=("$(apt list --installed 2> /dev/null | awk -F/ '{print $1}' | tr '\r\n' ' ')")
I then tried:
apt-cache search . | grep -v "${INSTALLED[#]}"
Unfortunately this doesnt work as I still see my installed pkgs too so I'm guessing its excluding the first pkg in the array and not the rest.
Again thank you in advance!
Would you please try the following:
#!/bin/bash
declare -A installed # an associative array to memorize the "installed" command names
while IFS=/ read -r cmd others; do # split the line on "/" and assign the variable "cmd" to the 1st field
(( installed[$cmd]++ )) # increment the array value associated with the "$cmd"
done < <(apt list --installed 2> /dev/null) # excecute the `apt` command and feed the output to the `while` loop
while IFS= read -r line; do # read the whole line "as is" because it includes a package description
read -r cmd others <<< "$line" # split the line on the whitespace and assign the variable "cmd" to the 1st field
[[ -z ${installed[$cmd]} ]] && echo "$line" # if the array value is not defined, the cmd is not installed, then print the line
done < <(apt-cache search .) # excecute the `apt-cache` command to feed the output to the `while` loop
The associative array installed is used to check if the command is
installed.
The 1st while loop scans over the installed list of the command and
store the command names in the associative array installed.
The 2nd while loop scans over the available command list and if the
command is not found in the associative array, then print it.
BTW your trial code starts with #!/bin/sh which is run with sh, not bash.
Please make sure it looks like #!/bin/bash.
Sorry if I'm misunderstanding, but you just want to get the list of packages which are not installed, right?
If so, you can just do this -
apt list --installed=false | grep -v '\[installed'

Trouble with AWK'd command output and bash array

I am attempting to get a list of running VirtualBox VMs (the UUIDs) and put them into an array. The command below produces the output below:
$ VBoxManage list runningvms | awk -F '[{}]' '{print $(NF-1)}'
f93c17ca-ab1b-4ba2-95e5-a1b0c8d70d2a
46b285c3-cabd-4fbb-92fe-c7940e0c6a3f
83f4789a-b55b-4a50-a52f-dbd929bdfe12
4d1589ba-9153-489a-947a-df3cf4f81c69
I would like to take those UUIDs and put them into an array (possibly even an associative array for later use, but a simple array for now is sufficient)
If I do the following:
array1="( $(VBoxManage list runningvms | awk -F '[{}]' '{print $(NF-1)}') )"
The commands
array1_len=${#array1[#]}
echo $array1_len
Outputs "1" as in there's only 1 element. If I print out the elements:
echo ${array1[*]}
I get a single line of all the UUIDs
( f93c17ca-ab1b-4ba2-95e5-a1b0c8d70d2a 46b285c3-cabd-4fbb-92fe-c7940e0c6a3f 83f4789a-b55b-4a50-a52f-dbd929bdfe12 4d1589ba-9153-489a-947a-df3cf4f81c69 )
I did some research (Bash Guide/Arrays on how to tackle this and found this with command substitution and redirection, but it produces an empty array
while read -r -d '\0'; do
array2+=("$REPLY")
done < <(VBoxManage list runningvms | awk -F '[{}]' '{print $(NF-1)}')
I'm obviously missing something. I've looked at several simiar questions on this site such as:
Reading output of command into array in Bash
AWK output to bash Array
Creating an Array in Bash with Quoted Entries from Command Output
Unfortunately, none have helped. I would apprecaite any assistance in figuring out how to take the output and assign it to an array.
I am running this on macOS 10.11.6 (El Captain) and BASH version 3.2.57
Since you're on a Mac:
brew install bash
Then with this bash as your shell, pipe the output to:
readarray -t array1
Of the -t option, the man page says:
-t Remove a trailing delim (default newline) from each line read.
If the bash4 solution is admissible, then the advice given
e.g. by gniourf_gniourf at reading-output-of-command-into-array-in-bash
is still sound.

BASH indexed arrays in Cygwin

--As requested
GNU Bash: version 4.4.12(3)-release (x86_64-unknown-cygwin)
Cygwin Version: 2.8.2 (i think, or whatever is the most current)
Windows Server 2012R2
EDIT (requested to update with an actual example that can be verified)
testdir=$(mktemp -t -d testdir.XXXXXX)
cd "$testdir/" && touch file{1..99}
read -r -a ARRAY <<< $(ls -alh "$testdir" | grep -Eo "file[0-9]{2}")
# grep above grep should ignore 0-9 (single digits)
In proper Linux shell: echo "${ARRAY[#]}" returns file10 file11 file12 file13 file14 file15 ..... file99
In a proper Linux shell: echo "${ARRAY[5]}" returns: file15
In Cygwin shell: echo "${ARRAY[#]}" returns file10
In Cygwin shell: echo "${ARRAY[5]}" returns nothing (as nothing indexed)
for i in "${!ARRAY[#]}"; do echo "Key: $i"; echo "Value: ${ARRAY[$i]}"; done
In a proper Linux Shell:
Key: 0
Value: file10
...
Key: 89
Value: file99
In Cygwin:
Key: 0
Value: file10
file11
file12
file13
file14
...etc.
ORIGINAL QUESTION BELOW:
I'm using Cygwin for the first time and I'm having a hard time understanding why my indexed array does populate properly:
This has worked fine in a proper Linux shell, which I've double checked to make sure i'm not losing my mind:
Which gets a long list of results, all properly indexed with unique keys ... whether wrong or right, this is how I learned to do it, and its always worked for me.
When I do this exact same thing in Cygwin, I only get one result.
If I do declare -a ARRAY and then ARRAY=$(stuff to do, results populate the array) I get many results, but all stored under a single key.
I feel I must be missing something basic, because I don't think it should be this hard.

How to get the size of the installed package via MacPort?

port installed displays all the installed packages on the local machine, but is there any way to list the size of each one? Thanks!
port space --units MB installed
Nice and easy, has been available since version 2.0 all the way back in 2011!
I don't believe there's a build in command from Macports to list the size of your installs, but you can do this:
Try this command in the terminal:
du -sh /opt/local/var/macports/software/*
This will give you the size of every package in /opt/local/var/macports/software/*, which I believe is the default install location.
Obviously, if you install your ports somewhere else you can use
du -sh [directory]
Without a built in Macports command, this is the probably the best you can do.
One alternative that comes to mind is creating a script that would take the output of
port installed
and echo the size of each install.
edit:
I was mistaken. /opt/local/var/macports/software/* contains the tarbells that the installations were extracted from, so the sizes will be smaller.
If you do du -sh /opt/local, it should list the size of everything, but there may be a few non-macports packages in the list.
The command port contents installed will show you the directory of everything macports has installed.
Here's a small bash function that will take any valid macports query
function port_size {
size=0
pkg_size=0
for pkg in $(port $# | tail -n +2 | awk '{ print $1 }')
do
pkg_size=$(port contents $pkg \
| sed -r 's/^[[:space:]]*(.*)[[:space:]]*$/\1/g;s/ /\\ /g' \
| tail -n +2 | xargs du | cut -f1 | paste -sd '+' | bc)
size=$(( $size + $pkg_size ))
printf "%10d %s\n" $pkg_size $pkg
done
printf "%10d %s\n" $size "Total Size (KB)"
}
I've only tested it with GNU versions of sed and awk but it should work regardless.
port_size installed # will print out all installed packages and their size
port_size installed gcc* # will print out all packages matching gcc* wildcard
Again, any valid macports query will work (including installed inactive or outdated.

Resources