I'm working on a project where I have to process the contents of a directory passed in as an argument, and I need to include invisible files (ones that start with .) as well. This is how I'm approaching it
#!/bin/bash
cd $1
for file in `dir -a -d * `;
do
#more code blah blah
even though I use the -a tag on the dir command, it still ignores invisible files. Any ideas why?
Just do:
#!/bin/bash
shopt -s dotglob
cd "$1"
for file in *; do
# more code blah blah
done
From the bash manpage
When a pattern is used for filename expansion, the character ‘.’ at
the start of a filename or immediately following a slash must be
matched explicitly, unless the shell option dotglob is set.
Related
I am working with the ensemble of the mol2 filles located in the same directory.
structure36S.mol2 structure30S.mol2 structure21.mol2
structure36R.mol2 structure30R.mol2 Structure20R.mol2
structure35S.mol2 structure29R.mol2 Structure19R.mol2
structure35R.mol2 structure28R.mol2 Structure13R.mol2
structure34S.mol2 structure27R.mol2
structure34R.mol2 structure26.mol2 jacks18.mol2
structure33S.mol2 structure25.mol2 5p9.mol2
structure33R.mol2 structure24.mol2 Y6J.mol2
structure32R.mol2 structure23.mol2 06I.mol2
structure31R.mol2 structure22.mol2
From this data I need to make an associative array with the names of the filles (without extension (mol2)) as well as some value (7LMF) shared between all elements:
dataset=( [structure36S]=7LMF [structure36R]=7LMF [structure35S]=7LMF ...[06I]=7LMF [Y6J]=7LMF )
We may start from the following script:
for file in ./*.mol2; do
file_name=$(basename "$file" .mol2)
#some command to add the file into the array
done
How this script could be completed for the creating of the array?
I would recommend turning on nullglob otherwise the pattern will evaluate as a string when there is no match.
Use parameter expansion to remove the file extension.
If the leading './' is included, it will need to be stripped with another expansion.
#!/usr/bin/env bash
shopt -s nullglob
declare -A dataset
for file in *.mol2; do
dataset+=([${file%.*}]=7LMF)
done
for key in "${!dataset[#]}"; do echo "dataset[$key]: ${dataset[$key]}"; done
Try this Shellcheck-clean code:
#! /bin/bash -p
shopt -s nullglob
declare -A dataset
for file in *.mol2; do
file_name=${file%.mol2}
dataset[$file_name]=7LMF
done
# Show the contents of 'dataset'
declare -p dataset
In Bash scripts, I frequently find this pattern useful, where I first print the command I'm about to execute, then I execute the command:
echo 'Running this cmd: ls -1 "$HOME/temp/some folder with spaces'
ls -1 "$HOME/temp/some folder with spaces"
echo 'Running this cmd: df -h'
df -h
# etc.
Notice the single quotes in the echo command to prevent variable expansion there! The idea is that I want to print the cmd I'm running, exactly as I will type and run the command, then run it!
How do I wrap this up into a function?
Wrapping the command up into a standard bash array, and then printing and calling it, like this, sort-of works:
# Print and run the passed-in command
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run_cmd cmd_array
# See:
# 1. My answer on how to pass regular "indexed" and associative arrays by reference:
# https://stackoverflow.com/a/71060036/4561887 and
# 1. My answer on how to pass associative arrays: https://stackoverflow.com/a/71060913/4561887
print_and_run_cmd() {
local -n array_reference="$1"
echo "Running cmd: ${cmd_array[#]}"
# run the command by calling all elements of the command array at once
${cmd_array[#]}
}
For simple commands like this it works fine:
Usage:
cmd_array=(ls -a -l -F /)
print_and_run_cmd cmd_array
Output:
Running cmd: ls -a -l -F /
(all output of that cmd is here)
But for more-complicated commands it is broken!:
Usage:
cmd_array=(ls -1 "$HOME/temp/some folder with spaces")
print_and_run_cmd cmd_array
Desired output:
Running cmd: ls -1 "$HOME/temp/some folder with spaces"
(all output of that command should be here)
Actual Output:
Running cmd: ls -1 /home/gabriel/temp/some folder with spaces
ls: cannot access '/home/gabriel/temp/some': No such file or directory
ls: cannot access 'folder': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces': No such file or directory
The first problem, as you can see, is that $HOME got expanded in the Running cmd: line, when it shouldn't have, and the double quotes around that path argument were removed, and the 2nd problem is that the command doesn't actually run.
How do I fix these 2 problems?
References:
my bash demo program where I have this print_and_run_cmd function: https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world/blob/master/bash/argument_parsing__3_advanced__gen_prog_template.sh
where I first documented how to pass bash arrays by reference, as I do in that function:
Passing arrays as parameters in bash
How to pass an associative array as argument to a function in Bash?
Follow-up question:
Bash: how to print and run a cmd array which has the pipe operator, |, in it
If you've got Bash version 4.4 or later, this function may do what you want:
function print_and_run_cmd
{
local PS4='Running cmd: '
local -
set -o xtrace
"$#"
}
For example, running
print_and_run_cmd echo 'Hello World!'
outputs
Running cmd: echo 'Hello World!'
Hello World!
local PS4='Running cmd: ' sets a prefix for commands printed by the shell when the xtrace option is on. The default is + . Localizing it means that the previous value of PS4 is automatically restored when the function returns.
local - causes any changes to shell options to be reverted automatically when the function returns. In particular, it causes the set -o xtrace on the next line to be automatically undone when the function returns. Support for local - was added in Bash 4.4.
From man bash, under the local [option] [name[=value] ... | - ] section (emphasis added):
If name is -, the set of shell options is made local to the function in which local is invoked: shell options changed using the set builtin inside the function are restored to their original values when the function returns.
set -o xtrace (which is equivalent to set -x) causes the shell to print commands, preceded by the expanded value of PS4, before running them.
See help set.
Check your scripts with shellcheck:
Line 2:
local -n array_reference="$1"
^-- SC2034 (warning): array_reference appears unused. Verify use (or export if used externally).
Line 3:
echo "Running cmd: ${cmd_array[#]}"
^-- SC2145 (error): Argument mixes string and array. Use * or separate argument.
^-- SC2154 (warning): cmd_array is referenced but not assigned.
Line 5:
${cmd_array[#]}
^-- SC2068 (error): Double quote array expansions to avoid re-splitting elements.
You might want to research https://github.com/koalaman/shellcheck/wiki/SC2068 . We fix all errors and we get:
print_and_run_cmd() {
local -n array_reference="$1"
echo "Running cmd: ${array_reference[*]}"
# run the command by calling all elements of the command array at once
"${array_reference[#]}"
}
For me it's odd to pass an array by reference in this case. I would pass the actual values. I often do:
prun() {
# in the style of set -x
# print to stderr, so output can be captured
echo "+ $*" >&2
# or echo "+ ${*#Q}" >&2
# or echo "+$(printf " %q" "$#")" >&2
# or echo "+$(/bin/printf " %q" "$#")" >&2
"$#"
}
prun "${cmd_array[#]}"
How do I fix these 2 problems?
Incorporate into your workflow linters, formatters and static analysis tools, like shellcheck, and check the problems they point out.
And quote variable expansion. It's "${array[#]}".
You can achieve what you want with DEBUG trap :
#!/bin/bash
set -T
trap 'test "$FUNCNAME" = print_and_run_cmd || trap_saved_command="${BASH_COMMAND}"' DEBUG
print_and_run_cmd(){
echo "Running this cmd: ${trap_saved_command#* }"
"$#"
}
outer(){
print_and_run_cmd ls -1 "$HOME/temp/some folder with spaces"
}
outer
# output ->
# Running this cmd: ls -1 "$HOME/temp/some folder with spaces"
# ...
I really like #pjh's answer, so I've marked it as correct. It doesn't fully answer my original question though, so if another answer comes along that does, I may have to change that. Anyway, see #pjh's answer or a full explanation of how the below code works, and what all those lines mean. I've helped edit that answer with some of the sources from man bash and help set.
I'd like to change the formatting and provide some more examples, however, to show that variable expansion does take place within the command. I'd also like to provide one version which passes by reference, and one which does not, so you can choose the call style which you like best.
Here are my examples, showing both call styles (print_and_run1 cmd_array and print_and_run2 "${cmd_array[#]}"):
#!/usr/bin/env bash
# Print and run the passed-in command, which is passed in as an
# array **by reference**.
# See here for a full explanation: https://stackoverflow.com/a/71151669/4561887
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run1 cmd_array
print_and_run1() {
local -n array_reference="$1"
local PS4='Running cmd: '
local -
set -o xtrace
# Call the cmd
"${array_reference[#]}"
}
# Print and run the passed-in command, which is passed in as members
# of an array **by value**.
# See here for a full explanation: https://stackoverflow.com/a/71151669/4561887
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run2 "${cmd_array[#]}"
print_and_run2() {
local PS4='Running cmd: '
local -
set -o xtrace
# Call the cmd
"$#"
}
cmd_array=(ls -1 "$HOME/temp/some folder with spaces")
print_and_run1 cmd_array
echo ""
print_and_run2 "${cmd_array[#]}"
echo ""
Sample run and output:
eRCaGuy_hello_world/bash$ ./print_and_run.sh
Running cmd: ls -1 '/home/gabriel/temp/some folder with spaces'
file1.txt
file2.txt
Running cmd: ls -1 '/home/gabriel/temp/some folder with spaces'
file1.txt
file2.txt
This seems to work too:
print_and_run_cmd() {
echo "Running cmd: $1"
eval "$cmd"
}
cmd='ls -1 "$HOME/temp/some folder with spaces"'
print_and_run_cmd "$cmd"
Output:
Running cmd: ls -1 "$HOME/temp/some folder with spaces"
(result of running the cmd is here)
But now the problem is, if I want to print an expanded version of the cmd too, to verify that part worked properly, I can't, or at least, don't know how.
Question Details
Suppose we have a directory with three files in it: file_1, file_2, and the very inconveniently named file 3. If my understanding of filename expansion is correct, the way bash interprets the string
echo *
is that it sees the (unquoted) *, and modifies the string so that it now reads
echo file_1 file_2 file 3
Then, since there are no more expansions to be performed, bash attempts to evaluate the string. In this case, it runs the command echo, passing to it four arguments: file, 3, file_1, and file_2. In any case, the outputs are identical:
$ echo *
> file 3 file_1 file_2
$ echo file 3 file_1 file_2
> file 3 file_1 file_2
However, in other contexts, this doesn't seem to be what happens. For instance
$ arr1=( * )
$ arr2=( file 3 file_1 file_2 )
$ echo ${#arr1}
> 3
$ echo ${#arr2}
> 4
And yet, if shell expansion works the way it's described in the bash documentation, these ought to be identical.
Something similar happens in a for loop:
$ for f in *; do echo $f; done
> file 3
> file_1
> file_2
$ for f in file 3 file_1 file_2; do echo $f; done
> file
> 3
> file_1
> file_2
What am I missing? Does globbing not happen in these cases?
Use case
I'm putting together a GitHub repo to centralize my dotfiles, following this suggestion from MIT's Hacker Tools. The script I'm writing has two usages:
./install.sh DOTFILE [DOTFILE [DOTFILE ...]]
./install.sh -a
In the first case, each of the named dotfiles in src/config is symlinked to a corresponding dotfile in my home directory; in the second, the -a flag prompts the script to run as if I had entered every dotfile as an argument.
The solution I came up with was to run ln -sih in a for loop using one of two arrays: $# and *.1 So, simply assign FILES=( $# ) or FILES=( * ), and then run for f in $FILES--except, it seems to me, * should break in this assignment if there's a filename with a space in it. Clearly bash is smarter than me, since it doesn't, but I don't understand why.
1: Obviously, you don't want the script itself to run through the loop, but that's easy enough to exclude with an if [[ "$f" != "$0" ]] clause.
From the bash documentation you linked to:
The order of expansions is: brace expansion; tilde expansion,
parameter and variable expansion, arithmetic expansion, and command
substitution (done in a left-to-right fashion); word splitting; and
filename expansion.
Filename expansion happens after word splitting, and therefore the expanded filenames are not themselves subject to further word splitting.
Let's say I have this directory structure:
DIRECTORY:
.........a
.........b
.........c
.........d
What I want to do is: I want to store elements of a directory in an array
something like : array = ls /home/user/DIRECTORY
so that array[0] contains name of first file (that is 'a')
array[1] == 'b' etc.
Thanks for help
You can't simply do array = ls /home/user/DIRECTORY, because - even with proper syntax - it wouldn't give you an array, but a string that you would have to parse, and Parsing ls is punishable by law. You can, however, use built-in Bash constructs to achieve what you want :
#!/usr/bin/env bash
readonly YOUR_DIR="/home/daniel"
if [[ ! -d $YOUR_DIR ]]; then
echo >&2 "$YOUR_DIR does not exist or is not a directory"
exit 1
fi
OLD_PWD=$PWD
cd "$YOUR_DIR"
i=0
for file in *
do
if [[ -f $file ]]; then
array[$i]=$file
i=$(($i+1))
fi
done
cd "$OLD_PWD"
exit 0
This small script saves the names of all the regular files (which means no directories, links, sockets, and such) that can be found in $YOUR_DIR to the array called array.
Hope this helps.
Option 1, a manual loop:
dirtolist=/home/user/DIRECTORY
shopt -s nullglob # In case there aren't any files
contentsarray=()
for filepath in "$dirtolist"/*; do
contentsarray+=("$(basename "$filepath")")
done
shopt -u nullglob # Optional, restore default behavior for unmatched file globs
Option 2, using bash array trickery:
dirtolist=/home/user/DIRECTORY
shopt -s nullglob
contentspaths=("$dirtolist"/*) # This makes an array of paths to the files
contentsarray=("${contentpaths[#]##*/}") # This strips off the path portions, leaving just the filenames
shopt -u nullglob # Optional, restore default behavior for unmatched file globs
array=($(ls /home/user/DIRECTORY))
Then
echo ${array[0]}
will equal to the first file in that directory.
He,
I want to read the contents of a file (which contains relative file paths) to a variable and prefix every line in the file with a path. Then copy all those files to a directory.
Like this:
$(httpd_DIR)/my.tar: $(mypath)/html.txt
rm -rf web
mkdir -p web
VV = $(addprefix $(httpd_DIR)/, $(shell cat $(mypath)/html.txt) )
cp -R $$VV $(httpd_DIR)/web
$(TAR) -C $(httpd_DIR) -cvf $(httpd_DIR)/web.tar web
The $(mypath)/html.txt file contains a list of relative file paths like this:
dir1/file1.html
dir2/file2.html
dir3/file3.html
For some reason I get the followin error:
/bin/bash: VV: command not found
I'm not trying to execute VV, so why is het giving me this error? Note that if I uncomment the cp command, I still get the same error...
I'm using GNU make on a linux PC.
You have several problems here.
VV = $(addprefix $(httpd_DIR)/, $(shell cat $(mypath)/html.txt) )
You haven't told us what shell you're using, so I'll assume bash.
If you want to assign a variable in bash, you must beware whitespace: VV=foo is legal and will do what you expect, but if you type VV = foo, the shell will interpret the first word, "VV", as a command, and balk. If you type VV=foo bar, the shell will assign foo to VV, then balk at the command bar. You can use VV="foo bar" instead.
Then you run into another problem. Each command runs in its own subshell, so variables assigned in one command don't survive to the next:
VV=foo
echo $$VV # this will echo a blank
You must combine the commands, like this:
VV=foo ; echo $$VV # this will echo foo
or this:
VV=foo ; \
echo $$VV # this will echo foo
(Note that there is only one TAB there, before the first line.)
In general, you should test these constructs with the simplest commands you can think of, before you plug in the real commands. That way it's much easier to catch these bugs.