I'm writing a Terminal Match-Anything Pattern Rule, i.e. %::, that, as expected, will run only if no other target is matched. In its recipe I want to iterate over makefile's explicit targets and check if the found pattern ($*) is the beginning of any other target
By now I'm successfully getting all desired targets in a space-separated string and storing it in a variable TARGETS, however I couldn't turn it in an array to be able to iterate over each word in the string.
For instance
%::
$(eval TARGETS ::= $(shell grep -Ph "^[^\t].*::.*##" ./Makefile | cut -d : -f 1 | sort))
echo $(TARGETS)
gives me just what I was expecting:
build clean compile deploy execute init run serve
The Question
How could I iterate over each of $(TARGET) string words inside a GNU Make 4.2.1 loop?
I found a bunch of BASH solutions, but none of them worked in my tests:
Reading a delimited string into an array in Bash
How to split one string into multiple strings separated by at least >one space in bash shell?
It's generally a really bad idea to use eval and shell inside a recipe. A recipe is already a shell script so you should just use shell scripting.
It's not really clear exactly what you want to do. If you want to do this in a recipe, you can use a shell loop:
%::
TARGETS=$$(grep -Ph "^[^\t].*::.*##" ./Makefile | cut -d : -f 1 | sort); \
for t in $$TARGETS; do \
echo $$t; \
done
If you want to do it outside of a recipe you can use the GNU make foreach function.
Related
The $# variable seems to maintain quoting around its arguments so that, for example:
$ function foo { for i in "$#"; do echo $i; done }
$ foo herp "hello world" derp
herp
hello world
derp
I am also aware that bash arrays, work the same way:
$ a=(herp "hello world" derp)
$ for i in "${a[#]}"; do echo $i; done
herp
hello world
derp
What is actually going on with variables like this? Particularly when I add something to the quote like "duck ${a[#]} goose". If its not space separated what is it?
Usually, double quotation marks in Bash mean "make everything between the quotation marks one word, even if it has separators in it." But as you've noticed, $# behaves differently when it's within double quotes. This is actually a parsing hack that dates back to Bash's predecessor, the Bourne shell, and this special behavior applies only to this particular variable.
Without this hack (I use the term because it seems inconsistent from a language perspective, although it's very useful), it would be difficult for a shell script to pass along its array of arguments to some other command that wants the same arguments. Some of those arguments might have spaces in them, but how would it pass them to another command without the shell either lumping them together as one big word or reparsing the list and splitting the arguments that have whitespace?
Well, you could pass an array of arguments, and the Bourne shell really only has one array, represented by $* or $#, whose number of elements is $# and whose elements are $1, $2, etc, the so-called positional parameters.
An example. Suppose you have three files in the current directory, named aaa, bbb, and cc c (the third file has a space in the name). You can initialize the array (that is, you can set the positional parameters) to be the names of the files in the current directory like this:
set -- *
Now the array of positional parameters holds the names of the files. $#, the number of elements, is three:
$ echo $#
3
And we can iterate over the position parameters in a few different ways.
1) We can use $*:
$ for file in $*; do
> echo "$file"
> done
but that re-separates the arguments on whitespace and calls echo four times:
aaa
bbb
cc
c
2) Or we could put quotation marks around $*:
$ for file in "$*"; do
> echo "$file"
> done
but that groups the whole array into one argument and calls echo just once:
aaa bbb cc c
3) Or we could use $# which represents the same array but behaves differently in double quotes:
$ for file in "$#"; do
> echo "$file"
> done
will produce
aaa
bbb
cc c
because $1 = "aaa", $2 = "bbb", and $3 = "cc c" and "$#" leaves the elements intact. If you leave off the quotation marks around $#, the shell will flatten and re-parse the array, echo will be called four times, and you'll get the same thing you got with a bare $*.
This is especially useful in a shell script, where the positional parameters are the arguments that were passed to your script. To pass those same arguments to some other command -- without the shell resplitting them on whitespace -- use "$#".
# Truncate the files specified by the args
rm "$#"
touch "$#"
In Bourne, this behavior only applies to the positional parameters because it's really the only array supported by the language. But you can create other arrays in Bash, and you can even apply the old parsing hack to those arrays using the special "${ARRAYNAME[#]}" syntax, whose at-sign feels almost like a wink to Mr. Bourne:
$ declare -a myarray
$ myarray[0]=alpha
$ myarray[1]=bravo
$ myarray[2]="char lie"
$ for file in "${myarray[#]}"; do echo "$file"; done
alpha
bravo
char lie
Oh, and about your last example, what should the shell do with "pre $# post" where you have $# within double quotes but you have other stuff in there, too? Recent versions of Bash preserve the array, prepend the text before the $# to the first array element, and append the text after the $# to the last element:
pre aaa
bb
cc c post
How can I run a command line from a bash array containing a pipeline?
For example, I want run ls | grep x by means of:
$ declare -a pipeline
$ pipeline=(ls)
$ pipeline+=("|")
$ pipeline+=(grep x)
$ "${pipeline[#]}"
But I get this:
ls: cannot access |: No such file or directory
ls: cannot access grep: No such file or directory
ls: cannot access x: No such file or directory
Short form: You can't (without writing some code), and it's a feature, not a bug.
If you're doing things in a safe way, you're protecting your data from being parsed as code (syntax). What you explicitly want here, however, is to treat data as code, but only in a controlled way.
What you can do is iterate over elements, use printf '%q ' "$element" to get a safely quoted string if they aren't a pipeline, and leave them unsubstituted if they are.
After doing that, and ONLY after doing that, can you safely eval the output string.
eval_args() {
local outstr=''
while (( $# )); do
if [[ $1 = '|' ]]; then
outstr+="| "
else
printf -v outstr '%s%q ' "$outstr" "$1"
fi
shift
done
eval "$outstr"
}
eval_args "${pipeline[#]}"
By the way -- it's much safer NOT TO DO THIS. Think about the case where you're processing a list of files, and one of them is named |; this strategy could be used by an attacker to inject code. Using separate lists for the before and after arrays, or making only one side of the pipeline an array and hardcoding the other, is far better practice.
Close - just add eval:
$ eval ${pipeline[#]}
This works for me:
bash -c "${pipeline[*]}"
Edit: I think this has been answered successfully, but I can't check 'til later. I've reformatted it as suggested though.
The question: I have a series of files, each with a name of the form XXXXNAME, where XXXX is some number. I want to move them all to separate folders called XXXX and have them called NAME. I can do this manually, but I was hoping that by naming them XXXXNAME there'd be some way I could tell Terminal (I think that's the right name, but not really sure) to move them there. Something like
mv *NAME */NAME
but where it takes whatever * was in the first case and regurgitates it to the path.
This is on some form of Linux, with a bash shell.
In the real life case, the files are 0000GNUmakefile, with sequential numbering. I'm having to make lots of similar-but-slightly-altered versions of a program to compile and run on a cluster as part of my research. It would probably have been quicker to write a program to edit all the files and put in the right place in the first place, but I didn't.
This is probably extremely simple, and I should be able to find an answer myself, if I knew the right words. Thing is, I have no formal training in programming, so I don't know what to call things to search for them. So hopefully this will result in me getting an answer, and maybe knowing how to find out the answer for similar things myself next time. With the basic programming I've picked up, I'm sure I could write a program to do this for me, but I'm hoping there's a simple way to do it just using functionality already in Terminal. I probably shouldn't be allowed to play with these things.
Thanks for any help! I can actually program in C and Python a fair amount, but that's through trial and error largely, and I still don't know what I can do and can't do in Terminal.
SO many ways to achieve this.
I find that the old standbys sed and awk are often the most powerful.
ls | sed -rne 's:^([0-9]{4})(NAME)$:mv -iv & \1/\2:p'
If you're satisfied that the commands look right, pipe the command line through a shell:
ls | sed -rne 's:^([0-9]{4})(NAME)$:mv -iv & \1/\2:p' | sh
I put NAME in brackets and used \2 so that if it varies more than your example indicates, you can come up with a regular expression to handle your filenames better.
To do the same thing in gawk (GNU awk, the variant found in most GNU/Linux distros):
ls | gawk '/^[0-9]{4}NAME$/ {printf("mv -iv %s %s/%s\n", $1, substr($0,0,4), substr($0,5))}'
As with the first sample, this produces commands which, if they make sense to you, can be piped through a shell by appending | sh to the end of the line.
Note that with all these mv commands, I've added the -i and -v options. This is for your protection. Read the man page for mv (by typing man mv in your Linux terminal) to see if you should be comfortable leaving them out.
Also, I'm assuming with these lines that all your directories already exist. You didn't mention if they do. If they don't, here's a one-liner to create the directories.
ls | sed -rne 's:^([0-9]{4})(NAME)$:mkdir -p \1:p' | sort -u
As with the others, append | sh to run the commands.
I should mention that it is generally recommended to use constructs like for (in Tim's answer) or find instead of parsing the output of ls. That said, when your filename format is as simple as /[0-9]{4}word/, I find the quick sed one-liner to be the way to go.
Lastly, if by NAME you actually mean "any string of characters" rather than the literal string "NAME", then in all my examples above, replace NAME with .*.
The following script will do this for you. Copy the script into a file on the remote machine (we'll call it sortfiles.sh).
#!/bin/bash
# Get all files in current directory having names XXXXsomename, where X is an integer
files=$(find . -name '[0-9][0-9][0-9][0-9]*')
# Build a list of the XXXX patterns found in the list of files
dirs=
for name in ${files}; do
dirs="${dirs} $(echo ${name} | cut -c 3-6)"
done
# Remove redundant entries from the list of XXXX patterns
dirs=$(echo ${dirs} | uniq)
# Create any XXXX directories that are not already present
for name in ${dirs}; do
if [[ ! -d ${name} ]]; then
mkdir ${name}
fi
done
# Move each of the XXXXsomename files to the appropriate directory
for name in ${files}; do
mv ${name} $(echo ${name} | cut -c 3-6)
done
# Return from script with normal status
exit 0
From the command line, do chmod +x sortfiles.sh
Execute the script with ./sortfiles.sh
Just open the Terminal application, cd into the directory that contains the files you want moved/renamed, and copy and paste these commands into the command line.
for file in [0-9][0-9][0-9][0-9]*; do
dirName="${file%%*([^0-9])}"
mkdir -p "$dirName"
mv "$file" "$dirName/${file##*([0-9])}"
done
This assumes all the files that you want to rename and move are in the same directory. The file globbing also assumes that there are at least four digits at the start of the filename. If there are more than four numbers, it will still be caught, but not if there are less than four. If there are less than four, take off the appropriate number of [0-9]s from the first line.
It does not handle the case where "NAME" (i.e. the name of the new file you want) starts with a number.
See this site for more information about string manipulation in bash.
i am trying to set a variable's value to use during the compilation.
I try to do that in a separate makefile target
svnversion:
SVN_REV=$(shell svnversion -cn | sed -e 's/.*://' -e 's/\([0-9]*\).*/\1/' | grep '[0-9]')
$(info svn_rev = $(SVN_REV))
I have read that this is the way to set the value of a variable.
Yet when i run 'make' I see :
SVN_REV=613
svn_rev =
so the variable seems to be empty. Afterwards I expect that this variable will be present while the compilation takes place (in other targets). Is this the case? or should I add an 'export' command in the svnversion target? and how to I address the SVN_REV variable? $(SVN_REV) or $$(SVN_REV).
thank you
You are actually assigning the value SVN_REV in the subshell that is concluded at the end of the line,
what you probably want is:
svnversion:: SVN_REV=$(shell svnversion -cn | sed -e 's/.*://' -e 's/\([0-9]*\).*/\1/' | grep '[0-9]')
svnversion:
$(info svn_rev = $(SVN_REV))
This sets the variable when the target is set.
If this isn't what was intended, say you want to do some processing with the variable, then you need to make each line a continuation of the previous one using the horrible ; \ at the end of line semantics. If you are then referencing shell variables (like the one evaluated in your first line), then you need to use the $$ syntax before the variable name
e.g.
svnversion:
SVN_REV=$(shell svnversion -cn | sed -e 's/.*://' -e 's/\([0-9]*\).*/\1/' | grep '[0-9]'); \
echo svn_rev = $$SVN_REV
but because it's in a shell, you can't use the variable in the $(info command, as that takes place outside of the evaluation of the target.
You can't use variables between different lines in rules or even between different rules. Each line is being executed in it's own shell, so there's no way of passing informations this way. (Petesh stated that right now as I just see)
if you need to store intermediate informations, use files like this:
foo:
uname -m > current_arch
...
bar:
gcc -m $$(cat current_arch) ...
...
You may also set a macro if the command being executed is not too time-consuming and does not depend on when it is called during the build process:ant
ARCH = $$(uname -m)
bar:
gcc -m $(ARCH) ...
But this is not a variable being set but a macro substitution. The actual command passed to the shell when calling make bar would be:
gcc -m $(uname -n)
and then the uname command would be executed.
Suppose there is a C program, which stores its version in a global char* in main.c. Can the buildsystem (gnu make) somehow extract the value of this variable on build time, so that the executable built could have the exact version name as it appears in the program?
What I'd like to achieve is that given the source:
char g_version[] = "Superprogram 1.32 build 1142";
the buildsystem would generate an executable named Superprogram 1.32 build 1142.exe
The shell function allows you to call external applications such as sed that can be used to parse the source for the details required.
Define your version variable from a Macro:
char g_version[] = VERSION;
then make your makefile put a -D argument on the command line when compiling
gcc hack.c -DVERSION=\"Superprogram\ 1.99\"
And of course you should in your example use sed/grep/awk etc to generate your version string.
You can use any combination of unix text tools (grep, sed, awk, perl, tail, ...) in your Makefile in order to extract that information from source file.
Usually the version is defined as a composition of several #define values (like in the arv library for example).
So let's go for a simple and working example:
// myversion.h
#define __MY_MAJOR__ 1
#define __MY_MINOR__ 8
Then in your Makefile:
# Makefile
source_file := myversion.h
MAJOR_Identifier := __MY_MAJOR__
MINOR_Identifier := __MY_MINOR__
MAJOR := `cat $(source_file) | tr -s ' ' | grep "\#define $(MAJOR_Identifier)" | cut -d" " -f 3`
MINOR := `cat $(source_file) | tr -s ' ' | grep '\#define $(MINOR_Identifier)' | cut -d" " -f 3`
all:
#echo "From the Makefile we extract: MAJOR=$(MAJOR) MINOR=$(MINOR)"
Explanation
Here I have used several tools so it is more robust:
tr -s ' ': to remove extra space between elements,
grep: to select the unique line matching our purpose,
cut -d" " -f 3: to extract the third element of the selected line which is the target value!
Note that the define values can be anything (not only numeric).
Beware to use := (not =) see: https://stackoverflow.com/a/10081105/4716013