Question:
Is it possible to exclude some of a commands output its output based on an array of words?**
Why?
On Ubuntu/Debian there are two ways to list all available pkgs:
apt list (show all available pkgs, installed pkgs too)
apt-cache search . (show all available pkgs, installed pkgs too)
Difference is, the first command, you can exclude all installed pkgs using grep -v, problem is unlike the first, the second command you cannot exclude these as the word "installed" isnt present. Problem with the first command is it doesnt show the pkg description, so I want to use apt-cache search . but excluding all installed pkgs.
# List all of my installed pkgs,
# Get just the pkg's name,
# Swap newlines for spaces,
# Save this list as an array for exclusion.
INSTALLED=("$(apt list --installed 2> /dev/null | awk -F/ '{print $1}' | tr '\r\n' ' ')")
I then tried:
apt-cache search . | grep -v "${INSTALLED[#]}"
Unfortunately this doesnt work as I still see my installed pkgs too so I'm guessing its excluding the first pkg in the array and not the rest.
Again thank you in advance!
Would you please try the following:
#!/bin/bash
declare -A installed # an associative array to memorize the "installed" command names
while IFS=/ read -r cmd others; do # split the line on "/" and assign the variable "cmd" to the 1st field
(( installed[$cmd]++ )) # increment the array value associated with the "$cmd"
done < <(apt list --installed 2> /dev/null) # excecute the `apt` command and feed the output to the `while` loop
while IFS= read -r line; do # read the whole line "as is" because it includes a package description
read -r cmd others <<< "$line" # split the line on the whitespace and assign the variable "cmd" to the 1st field
[[ -z ${installed[$cmd]} ]] && echo "$line" # if the array value is not defined, the cmd is not installed, then print the line
done < <(apt-cache search .) # excecute the `apt-cache` command to feed the output to the `while` loop
The associative array installed is used to check if the command is
installed.
The 1st while loop scans over the installed list of the command and
store the command names in the associative array installed.
The 2nd while loop scans over the available command list and if the
command is not found in the associative array, then print it.
BTW your trial code starts with #!/bin/sh which is run with sh, not bash.
Please make sure it looks like #!/bin/bash.
Sorry if I'm misunderstanding, but you just want to get the list of packages which are not installed, right?
If so, you can just do this -
apt list --installed=false | grep -v '\[installed'
Related
Following my previous question which got closed— basically I have a script that check availability of packages on target server, the target server and the packages have been stored to an array.
declare -a prog=("gdebi" "firefox" "chromium-browser" "thunar")
declare -a snap=("beer2" "beer3")
# checkvar=$(
for f in "${prog[#]}"; do
for connect in "${snap[#]}"; do
ssh lg#"$connect" /bin/bash <<- EOF
if dpkg --get-selections | grep -qE "(^|\s)"$f"(\$|\s)"; then
status="[INSTALLED] [$connect]"
else
status=""
fi
printf '%s %s\n' "$f" "\$status"
EOF
done
done
With the help of fellow member here, I've made several fix to original script, script ran pretty well— except there's one problem, the output contain duplicate entries.
gdebi [INSTALLED] [beer2]
gdebi
firefox [INSTALLED] [beer2]
firefox [INSTALLED] [beer3]
chromium-browser [INSTALLED] [beer2]
chromium-browser [INSTALLED] [beer3]
thunar
thunar
I know it this is normal behavior, as for pass multiple server from snap array, making ssh travel to all the two server.
Considering that the script checks same package for two server, I want the output to be merged.
If beer2 have firefox packages, but beer3 doesn't.
firefox [INSTALLED] [beer2]
If beer3 have firefox packages, but beer2 doesn't.
firefox [INSTALLED] [beer3]
If both beer2 and beer3 have the packages.
firefox [INSTALLED] [beer2, beer3]
or
firefox [INSTALLED] [beer2] [beer3]
If both beer2 and beer3 doesn't have the package, it will return without extra parameter.
firefox
Sound like an easy task, but for the love of god I can't find how to achieve this, here's list of things I have tried.
Try to manipulate the for loops.
Try putting return value after one successful loops (exit code).
Try nested if.
All of the above doesn't seem to work, I haven't tried changing/manipulate the return string as I'm not really experienced with some text processing such as: awk, sed, tr and many others.
Can anyone shows how It's done ? Would really mean the world to me.
Pure Bash 4+ solution using associative array to store hosts the program is installed on:
#!/usr/bin/env bash
declare -A hosts_with_package=(["gdebi"]="" ["firefox"]="" ["chromium-browser"]="" ["thunar"]="")
declare -a hosts=("beer2" "beer3")
# Collect installed status
# Iterate all hosts
for host in "${hosts[#]}"; do
# Read the output of dpkg --get-selections with searched packages
while IFS=$' \t' read -r package status; do
# Test weather package is installed on host
if [ "$status" = "install" ]; then
# If no host listed for package, create first entry
if [ -z "${hosts_with_package[$package]}" ]; then
# Record the first host having the package installed
hosts_with_package["$package"]="$host"
else
# Additional hosts are concatenated as CSV
hosts_with_package["$package"]="${hosts_with_package[$package]}, $host"
fi
fi
# Feed the whole loop with the output of the dpkg --get-selections for packages names
# Packages names are the index of the hosts_with_package array
done < <(ssh "lg#$host" dpkg --get-selections "${!hosts_with_package[#]}")
done
# Output results
# Iterate the package name keys
for package in "${!hosts_with_package[#]}"; do
# Print package name without newline
printf '%s' "$package"
# If package is installed on some hosts
if [ -n "${hosts_with_package[$package]}" ]; then
# Continue the line with installed hosts
printf ' [INSTALLED] [%s]' "${hosts_with_package[$package]}"
fi
# End with a newline
echo
done
Instead of making several ssh connections in nested loops consider this change
prog=( mysql-server apache2 php ufw )
snap=( localhost )
for connect in ${snap[#]}; do
ssh $connect "
progs=( ${prog[#]} )
for prog in \${progs[#]}; do
dpkg -l | grep -q \$prog && echo \"\$prog [INSTALLED]\" || echo \"\$prog\"
done
"
done
Based on #Ivan answer
#!/bin/bash
prog=( "gdebi" "firefox" "chromium-browser" "thunar" )
snap=( "beer2" "beer3" )
# First, retrieve the list on installed program for each host
for connect in ${snap[#]}; do
ssh lg#"$connect" /bin/bash >/tmp/installed.${connect} <<- EOF
progs=( "${prog[#]}" )
for prog in \${progs[#]}; do
dpkg --get-selections | awk -v pkg=\$prog '\$1 == pkg && \$NF ~ /install/ {print \$1}'
done
EOF
done
# Filter the previous results to format the output as you need
awk '{
f = FILENAME;
gsub(/.*\./,"",f);
a[$1] = a[$1] "," f
}
END {
for (i in a)
print i ":[" substr(a[i],2) "]"
}' /tmp/installed.*
rm /tmp/installed.*
Example of output :
# With prog=( bash cat sed tail something firefox-esr )
firefox-esr:[localhost]
bash:[localhost,localhost2]
sed:[localhost,localhost2]
I am attempting to get a list of running VirtualBox VMs (the UUIDs) and put them into an array. The command below produces the output below:
$ VBoxManage list runningvms | awk -F '[{}]' '{print $(NF-1)}'
f93c17ca-ab1b-4ba2-95e5-a1b0c8d70d2a
46b285c3-cabd-4fbb-92fe-c7940e0c6a3f
83f4789a-b55b-4a50-a52f-dbd929bdfe12
4d1589ba-9153-489a-947a-df3cf4f81c69
I would like to take those UUIDs and put them into an array (possibly even an associative array for later use, but a simple array for now is sufficient)
If I do the following:
array1="( $(VBoxManage list runningvms | awk -F '[{}]' '{print $(NF-1)}') )"
The commands
array1_len=${#array1[#]}
echo $array1_len
Outputs "1" as in there's only 1 element. If I print out the elements:
echo ${array1[*]}
I get a single line of all the UUIDs
( f93c17ca-ab1b-4ba2-95e5-a1b0c8d70d2a 46b285c3-cabd-4fbb-92fe-c7940e0c6a3f 83f4789a-b55b-4a50-a52f-dbd929bdfe12 4d1589ba-9153-489a-947a-df3cf4f81c69 )
I did some research (Bash Guide/Arrays on how to tackle this and found this with command substitution and redirection, but it produces an empty array
while read -r -d '\0'; do
array2+=("$REPLY")
done < <(VBoxManage list runningvms | awk -F '[{}]' '{print $(NF-1)}')
I'm obviously missing something. I've looked at several simiar questions on this site such as:
Reading output of command into array in Bash
AWK output to bash Array
Creating an Array in Bash with Quoted Entries from Command Output
Unfortunately, none have helped. I would apprecaite any assistance in figuring out how to take the output and assign it to an array.
I am running this on macOS 10.11.6 (El Captain) and BASH version 3.2.57
Since you're on a Mac:
brew install bash
Then with this bash as your shell, pipe the output to:
readarray -t array1
Of the -t option, the man page says:
-t Remove a trailing delim (default newline) from each line read.
If the bash4 solution is admissible, then the advice given
e.g. by gniourf_gniourf at reading-output-of-command-into-array-in-bash
is still sound.
I have the following find command with the following output:
$ find -name '*.jpg'
./public_html/github/screencasts-gh-pages/reactiveDataVis/presentation/images/telescope.jpg
./public_html/github/screencasts-gh-pages/introToBackbone/presentation/images/telescope.jpg
./public_html/github/StarCraft-master/img/Maps/(6)Thin Ice.jpg
./public_html/github/StarCraft-master/img/Maps/Snapshot.jpg
./public_html/github/StarCraft-master/img/Maps/Map_Grass.jpg
./public_html/github/StarCraft-master/img/Maps/(8)TheHunters.jpg
./public_html/github/StarCraft-master/img/Maps/(2)Volcanis.jpg
./public_html/github/StarCraft-master/img/Maps/(3)Trench wars.jpg
./public_html/github/StarCraft-master/img/Maps/(8)BigGameHunters.jpg
./public_html/github/StarCraft-master/img/Maps/(8)Turbo.jpg
./public_html/github/StarCraft-master/img/Maps/(4)Blood Bath.jpg
./public_html/github/StarCraft-master/img/Maps/(2)Switchback.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(6)Thin Ice.jpg
./public_html/github/StarCraft-master/img/Maps/Original/Map_Grass.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(8)TheHunters.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(2)Volcanis.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(3)Trench wars.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(8)BigGameHunters.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(8)Turbo.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(4)Blood Bath.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(2)Switchback.jpg
./public_html/github/StarCraft-master/img/Maps/Original/(4)Orbital Relay.jpg
./public_html/github/StarCraft-master/img/Maps/(4)Orbital Relay.jpg
./public_html/github/StarCraft-master/img/Bg/GameLose.jpg
./public_html/github/StarCraft-master/img/Bg/GameWin.jpg
./public_html/github/StarCraft-master/img/Bg/GameStart.jpg
./public_html/github/StarCraft-master/img/Bg/GamePlay.jpg
./public_html/github/StarCraft-master/img/Demo/Demo.jpg
./public_html/github/flot/examples/image/hs-2004-27-a-large-web.jpg
./public_html/github/minicourse-ajax-project/other/GameLose.jpg
How do I store this output in an array? I want it to handle filenames with spaces
I have tried this arrayname=($(find -name '*.jpg')) but this just stores the first element. # I am doing the following which seems to be just the first element?
$ arrayname=($(find -name '*.jpg'))
$ echo "$arrayname"
./public_html/github/screencasts-gh-pages/reactiveDataVis/presentation/images/telescope.jpg
$
I have tried here but again this just stores the 1st element
Other similar Qs
How do I capture the output from the ls or find command to store all file names in an array?
How do i store the output of a bash command in a variable?
If you know with certainty that your filenames will not contain newlines, then
mapfile -t arrayname < <(find ...)
If you want to be able to handle any file
arrayname=()
while IFS= read -d '' -r filename; do
arrayname+=("$filename")
done < <(find ... -print0)
echo "$arrayname" will only show the first element of the array. It is equivalent to echo "${arrayname[0]}". To dump an array:
printf "%s\n" "${arrayname[#]}"
# ............^^^^^^^^^^^^^^^^^ must use exactly this form, with the quotes.
arrayname=($(find ...)) is still wrong. It will store the file ./file with spaces.txt as 3 separate elements in the array.
If you have a sufficiently recent version of bash, you can save yourself a lot of trouble by just using a ** glob.
shopt -s globstar
files=(**/*.jpg)
The first line enables the feature. Once enabled, ** in a glob pattern will match any number (including 0) of directories in the path.
Using the glob in the array definition makes sure that whitespace is handled correctly.
To view an array in a form which could be used to define the array, use the -p (print) option to the declare builtin:
declare -p files
A script takes a URL, parses it for the required fields, and redirects its output to be saved in a file, file.txt. The output is saved on a new line each time a field has been found.
file.txt
A Cat
A Dog
A Mouse
etc...
I want to take file.txt and create an array from it in a new script, where every line gets to be its own string variable in the array. So far I have tried:
#!/bin/bash
filename=file.txt
declare -a myArray
myArray=(`cat "$filename"`)
for (( i = 0 ; i < 9 ; i++))
do
echo "Element [$i]: ${myArray[$i]}"
done
When I run this script, whitespace results in words getting split and instead of getting
Desired output
Element [0]: A Cat
Element [1]: A Dog
etc...
I end up getting this:
Actual output
Element [0]: A
Element [1]: Cat
Element [2]: A
Element [3]: Dog
etc...
How can I adjust the loop below such that the entire string on each line will correspond one-to-one with each variable in the array?
Use the mapfile command:
mapfile -t myArray < file.txt
The error is using for -- the idiomatic way to loop over lines of a file is:
while IFS= read -r line; do echo ">>$line<<"; done < file.txt
See BashFAQ/005 for more details.
mapfile and readarray (which are synonymous) are available in Bash version 4 and above. If you have an older version of Bash, you can use a loop to read the file into an array:
arr=()
while IFS= read -r line; do
arr+=("$line")
done < file
In case the file has an incomplete (missing newline) last line, you could use this alternative:
arr=()
while IFS= read -r line || [[ "$line" ]]; do
arr+=("$line")
done < file
Related:
Need alternative to readarray/mapfile for script on older version of Bash
You can do this too:
oldIFS="$IFS"
IFS=$'\n' arr=($(<file))
IFS="$oldIFS"
echo "${arr[1]}" # It will print `A Dog`.
Note:
Filename expansion still occurs. For example, if there's a line with a literal * it will expand to all the files in current folder. So use it only if your file is free of this kind of scenario.
Use mapfile or read -a
Always check your code using shellcheck. It will often give you the correct answer. In this case SC2207 covers reading a file that either has space separated or newline separated values into an array.
Don't do this
array=( $(mycommand) )
Files with values separated by newlines
mapfile -t array < <(mycommand)
Files with values separated by spaces
IFS=" " read -r -a array <<< "$(mycommand)"
The shellcheck page will give you the rationale why this is considered best practice.
You can simply read each line from the file and assign it to an array.
#!/bin/bash
i=0
while read line
do
arr[$i]="$line"
i=$((i+1))
done < file.txt
This answer says to use
mapfile -t myArray < file.txt
I made a shim for mapfile if you want to use mapfile on bash < 4.x for whatever reason. It uses the existing mapfile command if you are on bash >= 4.x
Currently, only options -d and -t work. But that should be enough for that command above. I've only tested on macOS. On macOS Sierra 10.12.6, the system bash is 3.2.57(1)-release. So the shim can come in handy. You can also just update your bash with homebrew, build bash yourself, etc.
It uses this technique to set variables up one call stack.
Make sure set the Internal File Separator (IFS)
variable to $'\n' so that it does not put each word
into a new array entry.
#!/bin/bash
# move all 2020 - 2022 movies to /backup/movies
# put list into file 1 line per dir
# dirs are "movie name (year)/"
ls | egrep 202[0-2] > 2020_movies.txt
OLDIFS=${IFS}
IFS=$'\n' #fix separator
declare -a MOVIES # array for dir names
MOVIES=( $( cat "${1}" ) ) // load into array
for M in ${MOVIES[#]} ; do
echo "[${M}]"
if [ -d "${M}" ] ; then # if dir name
mv -v "$M" /backup/movies/
fi
done
IFS=${OLDIFS} # restore standard separators
# not essential as IFS reverts when script ends
#END
I am trying to save the result from find as arrays.
Here is my code:
#!/bin/bash
echo "input : "
read input
echo "searching file with this pattern '${input}' under present directory"
array=`find . -name ${input}`
len=${#array[*]}
echo "found : ${len}"
i=0
while [ $i -lt $len ]
do
echo ${array[$i]}
let i++
done
I get 2 .txt files under current directory.
So I expect '2' as result of ${len}. However, it prints 1.
The reason is that it takes all result of find as one elements.
How can I fix this?
P.S
I found several solutions on StackOverFlow about a similar problem. However, they are a little bit different so I can't apply in my case. I need to store the results in a variable before the loop. Thanks again.
Update 2020 for Linux Users:
If you have an up-to-date version of bash (4.4-alpha or better), as you probably do if you are on Linux, then you should be using Benjamin W.'s answer.
If you are on Mac OS, which —last I checked— still used bash 3.2, or are otherwise using an older bash, then continue on to the next section.
Answer for bash 4.3 or earlier
Here is one solution for getting the output of find into a bash array:
array=()
while IFS= read -r -d $'\0'; do
array+=("$REPLY")
done < <(find . -name "${input}" -print0)
This is tricky because, in general, file names can have spaces, new lines, and other script-hostile characters. The only way to use find and have the file names safely separated from each other is to use -print0 which prints the file names separated with a null character. This would not be much of an inconvenience if bash's readarray/mapfile functions supported null-separated strings but they don't. Bash's read does and that leads us to the loop above.
[This answer was originally written in 2014. If you have a recent version of bash, please see the update below.]
How it works
The first line creates an empty array: array=()
Every time that the read statement is executed, a null-separated file name is read from standard input. The -r option tells read to leave backslash characters alone. The -d $'\0' tells read that the input will be null-separated. Since we omit the name to read, the shell puts the input into the default name: REPLY.
The array+=("$REPLY") statement appends the new file name to the array array.
The final line combines redirection and command substitution to provide the output of find to the standard input of the while loop.
Why use process substitution?
If we didn't use process substitution, the loop could be written as:
array=()
find . -name "${input}" -print0 >tmpfile
while IFS= read -r -d $'\0'; do
array+=("$REPLY")
done <tmpfile
rm -f tmpfile
In the above the output of find is stored in a temporary file and that file is used as standard input to the while loop. The idea of process substitution is to make such temporary files unnecessary. So, instead of having the while loop get its stdin from tmpfile, we can have it get its stdin from <(find . -name ${input} -print0).
Process substitution is widely useful. In many places where a command wants to read from a file, you can specify process substitution, <(...), instead of a file name. There is an analogous form, >(...), that can be used in place of a file name where the command wants to write to the file.
Like arrays, process substitution is a feature of bash and other advanced shells. It is not part of the POSIX standard.
Alternative: lastpipe
If desired, lastpipe can be used instead of process substitution (hat tip: Caesar):
set +m
shopt -s lastpipe
array=()
find . -name "${input}" -print0 | while IFS= read -r -d $'\0'; do array+=("$REPLY"); done; declare -p array
shopt -s lastpipe tells bash to run the last command in the pipeline in the current shell (not the background). This way, the array remains in existence after the pipeline completes. Because lastpipe only takes effect if job control is turned off, we run set +m. (In a script, as opposed to the command line, job control is off by default.)
Additional notes
The following command creates a shell variable, not a shell array:
array=`find . -name "${input}"`
If you wanted to create an array, you would need to put parens around the output of find. So, naively, one could:
array=(`find . -name "${input}"`) # don't do this
The problem is that the shell performs word splitting on the results of find so that the elements of the array are not guaranteed to be what you want.
Update 2019
Starting with version 4.4-alpha, bash now supports a -d option so that the above loop is no longer necessary. Instead, one can use:
mapfile -d $'\0' array < <(find . -name "${input}" -print0)
For more information on this, please see (and upvote) Benjamin W.'s answer.
Bash 4.4 introduced a -d option to readarray/mapfile, so this can now be solved with
readarray -d '' array < <(find . -name "$input" -print0)
for a method that works with arbitrary filenames including blanks, newlines, and globbing characters. This requires that your find supports -print0, as for example GNU find does.
From the manual (omitting other options):
mapfile [-d delim] [array]
-d
The first character of delim is used to terminate each input line, rather than newline. If delim is the empty string, mapfile will terminate a line when it reads a NUL character.
And readarray is just a synonym of mapfile.
The following appears to work for both Bash and Z Shell on macOS.
#! /bin/sh
IFS=$'\n'
paths=($(find . -name "foo"))
unset IFS
printf "%s\n" "${paths[#]}"
If you are using bash 4 or later, you can replace your use of find with
shopt -s globstar nullglob
array=( **/*"$input"* )
The ** pattern enabled by globstar matches 0 or more directories, allowing the pattern to match to an arbitrary depth in the current directory. Without the nullglob option, the pattern (after parameter expansion) is treated literally, so with no matches you would have an array with a single string rather than an empty array.
Add the dotglob option to the first line as well if you want to traverse hidden directories (like .ssh) and match hidden files (like .bashrc) as well.
you can try something like
array=(`find . -type f | sort -r | head -2`) , and in order to print the array values , you can try something like echo "${array[*]}"
None of these solutions suited me because I didn't feel like learning readarray and mapfile. Here is what I came up with.
#!/bin/bash
echo "input : "
read input
echo "searching file with this pattern '${input}' under present directory"
# The only change is here. Append to array for each non-empty line.
array=()
while read line; do
[[ ! -z "$line" ]] && array+=("$line")
done; <<< $(find . -name ${input} -print)
len=${#array[#]}
echo "found : ${len}"
i=0
while [ $i -lt $len ]
do
echo ${array[$i]}
let i++
done
You could do like this:
#!/bin/bash
echo "input : "
read input
echo "searching file with this pattern '${input}' under present directory"
array=(`find . -name '*'${input}'*'`)
for i in "${array[#]}"
do :
echo $i
done
In bash, $(<any_shell_cmd>) helps to run a command and capture the output. Passing this to IFS with \n as delimiter helps to convert that to an array.
IFS='\n' read -r -a txt_files <<< $(find /path/to/dir -name "*.txt")