He,
I want to read the contents of a file (which contains relative file paths) to a variable and prefix every line in the file with a path. Then copy all those files to a directory.
Like this:
$(httpd_DIR)/my.tar: $(mypath)/html.txt
rm -rf web
mkdir -p web
VV = $(addprefix $(httpd_DIR)/, $(shell cat $(mypath)/html.txt) )
cp -R $$VV $(httpd_DIR)/web
$(TAR) -C $(httpd_DIR) -cvf $(httpd_DIR)/web.tar web
The $(mypath)/html.txt file contains a list of relative file paths like this:
dir1/file1.html
dir2/file2.html
dir3/file3.html
For some reason I get the followin error:
/bin/bash: VV: command not found
I'm not trying to execute VV, so why is het giving me this error? Note that if I uncomment the cp command, I still get the same error...
I'm using GNU make on a linux PC.
You have several problems here.
VV = $(addprefix $(httpd_DIR)/, $(shell cat $(mypath)/html.txt) )
You haven't told us what shell you're using, so I'll assume bash.
If you want to assign a variable in bash, you must beware whitespace: VV=foo is legal and will do what you expect, but if you type VV = foo, the shell will interpret the first word, "VV", as a command, and balk. If you type VV=foo bar, the shell will assign foo to VV, then balk at the command bar. You can use VV="foo bar" instead.
Then you run into another problem. Each command runs in its own subshell, so variables assigned in one command don't survive to the next:
VV=foo
echo $$VV # this will echo a blank
You must combine the commands, like this:
VV=foo ; echo $$VV # this will echo foo
or this:
VV=foo ; \
echo $$VV # this will echo foo
(Note that there is only one TAB there, before the first line.)
In general, you should test these constructs with the simplest commands you can think of, before you plug in the real commands. That way it's much easier to catch these bugs.
Related
In Bash scripts, I frequently find this pattern useful, where I first print the command I'm about to execute, then I execute the command:
echo 'Running this cmd: ls -1 "$HOME/temp/some folder with spaces'
ls -1 "$HOME/temp/some folder with spaces"
echo 'Running this cmd: df -h'
df -h
# etc.
Notice the single quotes in the echo command to prevent variable expansion there! The idea is that I want to print the cmd I'm running, exactly as I will type and run the command, then run it!
How do I wrap this up into a function?
Wrapping the command up into a standard bash array, and then printing and calling it, like this, sort-of works:
# Print and run the passed-in command
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run_cmd cmd_array
# See:
# 1. My answer on how to pass regular "indexed" and associative arrays by reference:
# https://stackoverflow.com/a/71060036/4561887 and
# 1. My answer on how to pass associative arrays: https://stackoverflow.com/a/71060913/4561887
print_and_run_cmd() {
local -n array_reference="$1"
echo "Running cmd: ${cmd_array[#]}"
# run the command by calling all elements of the command array at once
${cmd_array[#]}
}
For simple commands like this it works fine:
Usage:
cmd_array=(ls -a -l -F /)
print_and_run_cmd cmd_array
Output:
Running cmd: ls -a -l -F /
(all output of that cmd is here)
But for more-complicated commands it is broken!:
Usage:
cmd_array=(ls -1 "$HOME/temp/some folder with spaces")
print_and_run_cmd cmd_array
Desired output:
Running cmd: ls -1 "$HOME/temp/some folder with spaces"
(all output of that command should be here)
Actual Output:
Running cmd: ls -1 /home/gabriel/temp/some folder with spaces
ls: cannot access '/home/gabriel/temp/some': No such file or directory
ls: cannot access 'folder': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces': No such file or directory
The first problem, as you can see, is that $HOME got expanded in the Running cmd: line, when it shouldn't have, and the double quotes around that path argument were removed, and the 2nd problem is that the command doesn't actually run.
How do I fix these 2 problems?
References:
my bash demo program where I have this print_and_run_cmd function: https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world/blob/master/bash/argument_parsing__3_advanced__gen_prog_template.sh
where I first documented how to pass bash arrays by reference, as I do in that function:
Passing arrays as parameters in bash
How to pass an associative array as argument to a function in Bash?
Follow-up question:
Bash: how to print and run a cmd array which has the pipe operator, |, in it
If you've got Bash version 4.4 or later, this function may do what you want:
function print_and_run_cmd
{
local PS4='Running cmd: '
local -
set -o xtrace
"$#"
}
For example, running
print_and_run_cmd echo 'Hello World!'
outputs
Running cmd: echo 'Hello World!'
Hello World!
local PS4='Running cmd: ' sets a prefix for commands printed by the shell when the xtrace option is on. The default is + . Localizing it means that the previous value of PS4 is automatically restored when the function returns.
local - causes any changes to shell options to be reverted automatically when the function returns. In particular, it causes the set -o xtrace on the next line to be automatically undone when the function returns. Support for local - was added in Bash 4.4.
From man bash, under the local [option] [name[=value] ... | - ] section (emphasis added):
If name is -, the set of shell options is made local to the function in which local is invoked: shell options changed using the set builtin inside the function are restored to their original values when the function returns.
set -o xtrace (which is equivalent to set -x) causes the shell to print commands, preceded by the expanded value of PS4, before running them.
See help set.
Check your scripts with shellcheck:
Line 2:
local -n array_reference="$1"
^-- SC2034 (warning): array_reference appears unused. Verify use (or export if used externally).
Line 3:
echo "Running cmd: ${cmd_array[#]}"
^-- SC2145 (error): Argument mixes string and array. Use * or separate argument.
^-- SC2154 (warning): cmd_array is referenced but not assigned.
Line 5:
${cmd_array[#]}
^-- SC2068 (error): Double quote array expansions to avoid re-splitting elements.
You might want to research https://github.com/koalaman/shellcheck/wiki/SC2068 . We fix all errors and we get:
print_and_run_cmd() {
local -n array_reference="$1"
echo "Running cmd: ${array_reference[*]}"
# run the command by calling all elements of the command array at once
"${array_reference[#]}"
}
For me it's odd to pass an array by reference in this case. I would pass the actual values. I often do:
prun() {
# in the style of set -x
# print to stderr, so output can be captured
echo "+ $*" >&2
# or echo "+ ${*#Q}" >&2
# or echo "+$(printf " %q" "$#")" >&2
# or echo "+$(/bin/printf " %q" "$#")" >&2
"$#"
}
prun "${cmd_array[#]}"
How do I fix these 2 problems?
Incorporate into your workflow linters, formatters and static analysis tools, like shellcheck, and check the problems they point out.
And quote variable expansion. It's "${array[#]}".
You can achieve what you want with DEBUG trap :
#!/bin/bash
set -T
trap 'test "$FUNCNAME" = print_and_run_cmd || trap_saved_command="${BASH_COMMAND}"' DEBUG
print_and_run_cmd(){
echo "Running this cmd: ${trap_saved_command#* }"
"$#"
}
outer(){
print_and_run_cmd ls -1 "$HOME/temp/some folder with spaces"
}
outer
# output ->
# Running this cmd: ls -1 "$HOME/temp/some folder with spaces"
# ...
I really like #pjh's answer, so I've marked it as correct. It doesn't fully answer my original question though, so if another answer comes along that does, I may have to change that. Anyway, see #pjh's answer or a full explanation of how the below code works, and what all those lines mean. I've helped edit that answer with some of the sources from man bash and help set.
I'd like to change the formatting and provide some more examples, however, to show that variable expansion does take place within the command. I'd also like to provide one version which passes by reference, and one which does not, so you can choose the call style which you like best.
Here are my examples, showing both call styles (print_and_run1 cmd_array and print_and_run2 "${cmd_array[#]}"):
#!/usr/bin/env bash
# Print and run the passed-in command, which is passed in as an
# array **by reference**.
# See here for a full explanation: https://stackoverflow.com/a/71151669/4561887
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run1 cmd_array
print_and_run1() {
local -n array_reference="$1"
local PS4='Running cmd: '
local -
set -o xtrace
# Call the cmd
"${array_reference[#]}"
}
# Print and run the passed-in command, which is passed in as members
# of an array **by value**.
# See here for a full explanation: https://stackoverflow.com/a/71151669/4561887
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run2 "${cmd_array[#]}"
print_and_run2() {
local PS4='Running cmd: '
local -
set -o xtrace
# Call the cmd
"$#"
}
cmd_array=(ls -1 "$HOME/temp/some folder with spaces")
print_and_run1 cmd_array
echo ""
print_and_run2 "${cmd_array[#]}"
echo ""
Sample run and output:
eRCaGuy_hello_world/bash$ ./print_and_run.sh
Running cmd: ls -1 '/home/gabriel/temp/some folder with spaces'
file1.txt
file2.txt
Running cmd: ls -1 '/home/gabriel/temp/some folder with spaces'
file1.txt
file2.txt
This seems to work too:
print_and_run_cmd() {
echo "Running cmd: $1"
eval "$cmd"
}
cmd='ls -1 "$HOME/temp/some folder with spaces"'
print_and_run_cmd "$cmd"
Output:
Running cmd: ls -1 "$HOME/temp/some folder with spaces"
(result of running the cmd is here)
But now the problem is, if I want to print an expanded version of the cmd too, to verify that part worked properly, I can't, or at least, don't know how.
I am trying to loop through an array of directories using a bash script so I can list directories with their timestamp, ownership etc using ls -arlt. I am reviewing bash so would like some feedback.
It works with declare -a for those indirect references but for each directory it outputs and extra directory from the /home/user.
I tried to use declare -n and declare -r for each directory and doesn't work.
#!/bin/bash
# Bash variables
acpi=/etc/acpi
apm=/etc/apm
xml=/etc/xml
array=( acpi apm xml )
# Function to display timestamp, ownership ...
displayInfo()
{
for i in "${array[#]}"; do
declare -n curArray=$i
if [[ -d ${curArray} ]]; then
declare -a _acpi=${curArray[0]} _apm=${curArray[1]} _xml=${curArray[2]}
echo "Displaying folder apci: "
cd $_acpi
ls -alrt
read -p "Press enter to continue"
echo "Displaying folder apm: "
cd $_apm
ls -alrt
read -p "Press enter to continue"
echo "Displaying folder xml: "
cd $_xml
ls -alrt
read -p "Press enter to continue"
else
echo "Displayed Failed" >&2
exit 1
fi
done
}
displayInfo
exit 0
It outputs an extra directory listing the /home/user and don't want that output.
There are a lot of complex and powerful shell features being used here, but in ways that don't fit together or make sense. I'll go over the mistakes in a minute, first let me just give how I'd do it. One thing I will use that you might not be familiar with is indirect variable references with ${!var} -- this is like using a nameref variable, but IMO it's clearer what's going on.
acpi=/etc/acpi
apm=/etc/apm
xml=/etc/xml
array=( acpi apm xml )
displayInfo()
{
for curDirectory in "${array[#]}"; do
if [[ -d ${!curDirectory} ]]; then
echo "Displaying folder $curDirectory:"
ls -alrt "${!curDirectory}"
read -p "Press enter to continue"
else
echo "Error: ${!curDirectory} does not exist or is not a directory" >&2
exit 1
fi
done
}
displayInfo
(One problem with this is that it does the "Press enter to continue" thing after each directory, rather than just between them. This can be fixed, but it's a little more work.)
Ok, now for what went wrong with the original. My main recommendation for you would be to try mentally stepping through your code to see what it's doing. It can help to put set -x before it, so the shell will print its interpretation of what it's doing as it runs, and see how it compares to what you expected. Let's do a short walkthrough of the displayInfo function:
for i in "${array[#]}"; do
This will loop over the contents of array, so on the first pass through the loop i will be set to "acpi". Good so far.
declare -n curArray=$i
This creates a nameref variable pointing to the other variable acpi -- this is similar to what I did with ${! }, and basically reasonable so far. Well, with one exception: the name suggests it's an array, but acpi is a plain variable, not an array.
if [[ -d ${curArray} ]]; then
This checks whether the contents of the acpi variable, "/etc/acpi" is the path of an existing directory (which it is). Still doing good.
declare -a _acpi=${curArray[0]} _apm=${curArray[1]} _xml=${curArray[2]}
Here's where things go completely off the rails. curArray points to the variable acpi, so ${curArray[0]} etc are equivalent to ${acpi[0]} etc. But acpi isn't an array, it's a plain variable, so ${acpi[0]} gets its value, and ${acpi[1]} and ${acpi[2]} get nothing. Furthermore, you're using declare -a (declare arrays), but you're just assigning single values to _acpi, _apm, and _xml. They're declared as arrays, but you're just using them as plain variables (basically the reverse of how you're using curArray -> acpi).
There's a deeper confusion here as well. The for loop above is iterating over "acpi", "apm", and "xml", and we're currently working on "acpi". During this pass through the loop, you should only be working on acpi, not also trying to work on apm and xml. That's the point of having a for loop there.
Ok, that's the main problem here, but let me just point out a couple of other things I'd consider bad practice:
cd $_apm
ls -alrt
Using a variable reference without double-quotes around it like this invites parsing confusion; you should almost always put double-quotes, like cd "$_apm". Also, using cd in a script is dangerous because if it fails the rest of the script will execute in the wrong place. In this case, _apm is empty, so without double-quotes it's equivalent to just cd, which moves to your home directory. This is why you're getting that result. If you used cd "$_apm" it would get an error instead... but since you don't check for that it'll go ahead and still list an irrelevant location.
It's almost always better to avoid cd and its complications entirely, and just use explicit paths, like ls -alrt "$_apm".
echo "Displayed Failed" >&2
exit 1
Do you actually want to exit the entire script if one of the directories doesn't exist? It'd make more sense to me to just return 1 (which exits just the function, not the entire script), or better yet continue (which just goes on to the next iteration of the loop -- i.e. the next directory on the list). I left the exit in my version, but I'd recommend changing it.
One more similar thing:
acpi=/etc/acpi
apm=/etc/apm
xml=/etc/xml
array=( acpi apm xml )
Is there any actual reason to use this array -> variable name -> actual directory path system (and resulting indirect expansion or nameref complications), rather than just having an array of directory paths, like this?
array=( /etc/acpi /etc/apm /etc/xml )
I left the indirection in my version above, but really if there's no reason for it I'd remove the complication.
I am writing a simple makefile named run.mk, as shown in the following code.
a = 0
b := $(shell echo `expr $(a) + 1`)
app: main.o
gcc -o app main.o
main.o: main.c
gcc -c main.c
test:
while [ $(a) -lt 10 ];\
do\
echo $(a);\
a:= $(shell echo `expr $(a) + 1`);\
echo $(a);\
done
when i run this makefile using command make -f run.mk test, error comes: a not found and loop runs infinitly i.e value of variable a is not updated at a:= $(shell echo expr $(a) + 1) within while loop. However at the beginning, value of variable b is set to 1 via the same code line $(shell echo expr $(a) + 1). Someone please tell how to update the value of variable a within the loop.
Thank you.
You're mixing up make syntax and shell syntax. Commands that are in a recipe are run by the shell (after make expands them one time). The shell is a separate process and anything that happens in the shell is completely invisible to make. All make sees is the exit code (to know if there was an error or not). It is not possible for the shell to modify the behavior of make in any way (change variable values, etc.)
So with your rule what make does is first expand the recipe for test (by the way, this is a very bad name for a target because test is actually a real program already on your system), to get this result:
while [ 0 -lt 10 ]; \
do \
echo 0; \
a:= 1; \
echo 0;\
done
After that expansion is complete it sends that text to the shell to execute, and obviously that will run forever (as well, note that a:= 1 is not a valid shell command).
Since you only told us what you tried but didn't tell us what you actually wanted to do in the first place, we can't help you do what you want to do.
If what you want to do is write a recipe that will loop 10 times printing a value, you have to do it entirely using shell syntax, NOT make syntax. Like this:
test:
a=$(a); \
while [ $$a -lt 10 ];\
do\
echo $$a;\
a=`expr $$a + 1`;\
echo $$a;\
done
The $$ escapes variables from being expanded by make, and results in this shell script being run by the shell:
a=0; \
while [ $a -lt 10 ];\
do\
echo $a;\
a=`expr $a + 1`;\
echo $a;\
done
You're mixing shell script variables and make variables.
The recipe for test is a shell script (from while to done). Make variables, such as $(a) are expanded before it starts to execute, so the while line is while [ 0 -lt 10 ] which is always true.
The assignment a:=... doesn't work, because it's makefile syntax, not shell syntax. Even if it did work, it wouldn't affect the while line.
I'm working on a project where I have to process the contents of a directory passed in as an argument, and I need to include invisible files (ones that start with .) as well. This is how I'm approaching it
#!/bin/bash
cd $1
for file in `dir -a -d * `;
do
#more code blah blah
even though I use the -a tag on the dir command, it still ignores invisible files. Any ideas why?
Just do:
#!/bin/bash
shopt -s dotglob
cd "$1"
for file in *; do
# more code blah blah
done
From the bash manpage
When a pattern is used for filename expansion, the character ‘.’ at
the start of a filename or immediately following a slash must be
matched explicitly, unless the shell option dotglob is set.
i am trying to set a variable's value to use during the compilation.
I try to do that in a separate makefile target
svnversion:
SVN_REV=$(shell svnversion -cn | sed -e 's/.*://' -e 's/\([0-9]*\).*/\1/' | grep '[0-9]')
$(info svn_rev = $(SVN_REV))
I have read that this is the way to set the value of a variable.
Yet when i run 'make' I see :
SVN_REV=613
svn_rev =
so the variable seems to be empty. Afterwards I expect that this variable will be present while the compilation takes place (in other targets). Is this the case? or should I add an 'export' command in the svnversion target? and how to I address the SVN_REV variable? $(SVN_REV) or $$(SVN_REV).
thank you
You are actually assigning the value SVN_REV in the subshell that is concluded at the end of the line,
what you probably want is:
svnversion:: SVN_REV=$(shell svnversion -cn | sed -e 's/.*://' -e 's/\([0-9]*\).*/\1/' | grep '[0-9]')
svnversion:
$(info svn_rev = $(SVN_REV))
This sets the variable when the target is set.
If this isn't what was intended, say you want to do some processing with the variable, then you need to make each line a continuation of the previous one using the horrible ; \ at the end of line semantics. If you are then referencing shell variables (like the one evaluated in your first line), then you need to use the $$ syntax before the variable name
e.g.
svnversion:
SVN_REV=$(shell svnversion -cn | sed -e 's/.*://' -e 's/\([0-9]*\).*/\1/' | grep '[0-9]'); \
echo svn_rev = $$SVN_REV
but because it's in a shell, you can't use the variable in the $(info command, as that takes place outside of the evaluation of the target.
You can't use variables between different lines in rules or even between different rules. Each line is being executed in it's own shell, so there's no way of passing informations this way. (Petesh stated that right now as I just see)
if you need to store intermediate informations, use files like this:
foo:
uname -m > current_arch
...
bar:
gcc -m $$(cat current_arch) ...
...
You may also set a macro if the command being executed is not too time-consuming and does not depend on when it is called during the build process:ant
ARCH = $$(uname -m)
bar:
gcc -m $(ARCH) ...
But this is not a variable being set but a macro substitution. The actual command passed to the shell when calling make bar would be:
gcc -m $(uname -n)
and then the uname command would be executed.