Make target containing subdirectory based on target dependencies - static

Say I have a list of source files and each are to be compiled to separate binaries:
SRCS = abcd.c efgh.c ijkl.c
And I want output files in separate subdirectories based on the file names like this:
build/abcd/abcd
build/efgh/efgh
build/ijkl/ijkl
I'm thinking a static pattern rule is the way to go. The pseudo-make-rule can be something like:
$(TARGETS): build/%/%: %.c
# stuff ...
I started by making a list of the subdirectories based on the filenames:
DIRS = $(SRCS:%.c=build/%)
So now we have DIRS = build/abcd build/efgh build/ijkl. I thought I can make the list of targets now with something like:
BLDS = $(DIRS:%=%/$(basename %))
But of course this doesn't work since the wildcard can not be used multiple times within a pattern. Therefore I'm now stuck at BLDS = build/abcd/% build/efgh/% build/ijkl/%.
Obviously I'm totally going about this the wrong way. How would you go about this?
For now I'm writing each rule explicitly, which is starting to get a bit tedious:
compile = # command to do stuff
BD = build
all: $(BD)/abcd/abcd $(BD)/efgh/efgh $(BD)/ijkl/ijkl
$(BD)/abcd/abcd: abcd.c
$(call compile)
$(BD)/efgh/efgh: efgh.c
$(call compile)
$(BD)/ijkl/ijkl: ijkl.c
$(call compile)
clean:
rm -rf build/*
.PHONY: all

I believe this does what you want:
SRCS:=abcd.c efgh.c ijkl.c
# We could fold NAMES into BLDS's definition if NAMES is not used elsewhere.
NAMES:=$(SRCS:%.c=%)
BLDS:=$(foreach name,$(NAMES),$(subst foo,$(name),build/foo/foo))
# We don't use DIRS below but the question had this variable.
DIRS:=$(dir $(BLDS))
TARGETS:=$(BLDS)
.PHONY: all
all: $(TARGETS)
.SECONDEXPANSION:
$(TARGETS): $$(notdir $$#).c
#echo Build $# from $^
mkdir -p $(dir $#)
touch $#
There are two important changes. The first is to reorder how the variables are created, and use subst, which allows replacing a matched string multiple times. The second is to use secondary expansion so that make builds rules for each of your targets. You initially a pattern with two %, but the docs say:
A pattern rule looks like an ordinary rule, except that its target contains the character `%' (exactly one of them).
(Emphasis added.)
I've tested the above with fake files abcd.c efgh.c and ijkl.c and get the following output:
$ make
Build build/abcd/abcd from abcd.c
mkdir -p build/abcd/
touch build/abcd/abcd
Build build/efgh/efgh from efgh.c
mkdir -p build/efgh/
touch build/efgh/efgh
Build build/ijkl/ijkl from ijkl.c
mkdir -p build/ijkl/
touch build/ijkl/ijkl

Related

make clean: Only remove files that have been generated

With the Makefile I'm working on, I convert pdf files into txt files.
I've implemented a clean target that would remove all .txt files. However, I do not wish to delete the source files, only those that have been generated.
Example:
I have following files in my folder:
pdfsource.pdf and donotharm.txt
Running my makefile would create following file:
pdfsource.txt
For now, my clean looks like this:
rm -f *.txt
Using make clean would not only delete pdfsource.txt, which is desired, but also donotharm.txt.
I think I could use: .PRECIOUS: donotharm.txt, but this is really specific. I'd like to have a general solution to this.
Thanks in advance!
You can list the generated files in a make variable and use it to clean only these:
PDF := $(wildcard *.pdf)
TEXT := $(patsubst %.pdf,%.txt,$(PDF))
...
clean:
rm -f $(TEXT)
Or, if you prefer a more compact (but a bit less readable) form:
clean:
rm -f $(patsubst %.pdf,%.txt,$(wildcard *.pdf))
Of course, this works only if there is no {foo.pdf,foo.txt} pair for which you want to preserve foo.txt from deletion by make clean.
Note: using make variables, in such a case, is usually a good idea because they can be shared among various rules. Example:
PDF := $(wildcard *.pdf)
TEXT := $(patsubst %.pdf,%.txt,$(PDF))
.PHONY: all clean
all: $(TEXT)
$(TEXT): %.txt: %.pdf
pdftotext $< $#
clean:
rm -f $(TEXT)
Another approach: "make -nps" gives you all make's metadata about dependencies. For any intermediate file, it prints
filename: ...
So you can exactly delete such files with a generic "clean" rule:
clean:; MAKEFLAGS= ${MAKE} -j1 -spinf $(word 1,${MAKEFILE_LIST}) \
| sed -n '/^# I/,$${/^[^\#\[%.][^ %]*: /s/:.*//p;}; 1s|.*|${clean}|p' | xargs rm -rf
The first line handles use of makefiles other than the defaults (makefile, GNUmakefile, Makefile)
In the "sed" command:
/^# I/,$
... selects the zone of make metadata with dependencies.
/^[^\#\[%.][^ %]*: /
... filters out comments, implicit rules, and files with no dependencies (the trailing space). It doesn't filter out phony targets; oh well.
Finally:
1s|.*|${clean}|p
adds any explicit targets for "clean" -- what you know that make does not; e.g.
clean += tmpdir/* *.gcda

using MAKEFILE to copy files before compilation and delete them after

I am trying to copy files befoe compilation (I have two source files with same name so I copy the files to a files with a different name) and delete them at the end of the MAKEFILE.
I am trying to do the folliwng but probably there is mismatch in the execution order.
How can I do it correctly?
all: copy_dup_files $(dst_dir) $(APP_TARGET_LIB) delete_dup_files
copy_dup_files:
#echo "COPYING DUP FILES"
$(shell cp /aaa/hmac.c /aaa/hmac1.c )
$(shell cp /bbb/hmac.c /bbb/hmac2.c )
delete_dup_files:
#echo "DELETING DUP FILES"
$(shell rm /aaa/hmac1.c )
$(shell rm /bbb/hmac2.c )
Thanks
The purpose of $(shell) is to produce an output which Make reads. The recipe lines should not have this construct at all.
# this is evaluated when the Makefile is read
value := $(shell echo "Use the shell to produce a value for a variable")
# this is evaluated when you say "make foo"
foo:
echo 'No $$(shell ...) stuff here'
So, all the $(shell ...) stuff in your attempt gets evaluated when the Makefile is read, but before any actual target is executed.
Your makefile is trying to say /aaa/hmac1.c depends on /aaa/hmac.c.
Thus we have:
/aaa/hmac1.c: /aaa/hmac.c
cp $< $#
/bbb/hmac2.c: /bbb/hmac.c
cp $< $#
/aaa/hmac1.o /bbb/hmac2.o: %.o: %.c
gcc $< -o $#
myprog: /aaa/hmac1.o /bbb/hmac2.o
gcc $^ -o $#
This is clean and parallel safe (a good test of any makefile).
There are innumerable style improvements you could make, like
Get rid of the absolute paths
Use symbolic links instead of copying
Automatic dependency generation (for .h files, etc.)
Don't besmirch the source tree — put all the intermediate files (the .os and the temporary .cs) in their own build folder
&c. &c.

Iterating over a directory of files using GNU Make

My project has a directory called tests/ which contains an arbitrary number of C source files, each one is a self-contained program designed to test a library. For each of these sourcefiles, I want to create an executable of the same name in my build/ directory.
E.g. tests/test_init.c would compile to an executable file build/test_init.
Currently, my Makefile snippet looks like the following:
BUILD_DIR = build
TEST_DIR = tests
test_sources:= $(TEST_DIR)/*.c
test_executables:= $(patsubst %.c, %, $(test_sources))
.PHONY: tests
tests: $(test_executables)
$(CC) $^ -o $# -g
But this fails to produce to desired result. Any help would be much appreciated.
First you need the wildcard function to find the sources:
test_sources:= $(wildcard $(TEST_DIR)/*.c)
Then the correct names for the executables:
test_executables:= $(patsubst $(TEST_DIR)/%.c, $(BUILD_DIR)/%, $(test_sources))
Then a pattern rule to build a test executable from the corresponding source:
$(BUILD_DIR)/%: $(TEST_DIR)/%.c
$(CC) $< -o $# -g
(A static pattern rule might be a little tidier, but it's a more advanced method.)
Finally a phony target to build all of the tests:
.PHONY: tests
tests: $(test_executables)
If you wanted Make to run all of these tests, you could make a phony pattern rule run_test_%, but that can wait for another day.
This Makefile will detect all files in test/*.c and provides tasks build_tests, run_tests, and clean_tests.
all: build_tests
define test_template
build_tests: test/$(1)
run_tests: build_tests run_test_$(1)
test/$(1) : test/$(1).c
$$(CC) $$^ -o $$#
.PHONY : run_test_$(1)
run_test_$(1) : test/$(1)
test/$(1)
endef
clean: clean_tests
clean_tests:
rm -fv $(foreach test, $(tests),test/$(test))
# Auto detect the tests (any .c file in the test directory),
# and store the list of tests names.
tests := $(foreach test, $(wildcard test/*.c),$(patsubst %.c,%,$(notdir $(test))))
# Add information about each test to the Makefile.
$(foreach test, $(tests), $(eval $(call test_template,$(test))))
Note: I'm not sure how to get tabs working inside a code block in markdown, so you'll need to replace spaces with a single tab on each indented line if you copy and paste this.

Expanding pattern twice for dependency

Is it possible to expand % twice for a dependency name?
I have project split into several sections, where each section is in it's own folder and file of the same name, e.g.
sections/first/first.c,sections/hello/hello.c, etc.
Now I would like to build for each section a target in another folder build; I've come up with the following
SECTIONS = $(wildcard sections/*/*.c)
TARGETS = $(addprefix build/,$(notdir $(SECTIONS)))
$(TARGETS): build/%.c: sections/%/%.c
# do something
The problem however is that % is expanded only once, so I end up with an error
make: *** No rule to make target 'sections/first/%.c', needed by 'build/first.o'.
Is there a way to expand % twice?
It's not possible to get the % to expand more than one time. If you need this you'll have to do something more fancy: you can use either secondary expansion or eval. For example, this should work:
.SECONDEXPANSION:
$(TARG) : build/%.c : source/$$*/$$*.c
#echo copy $< $#
For more in-depth discussion you can check these blog posts.
SECTIONS = $(wildcard sections/*)
.PHONY: all
all: $(SECTIONS)
$(foreach dir,$(SECTIONS), \
( cd $d && $(MAKE) -f ../makefile.bot name=$(dir) all ); )
Where the above is in a top level makefile
and a second makefile (in this case makefile.bot) performs the actual work.
a similar rule as the all rule for clean install,etc would be in the top level makefile, but the target would be changed from all to clean, etc
The makefile.bot would use $name to set the executable name, etc
Notice the use of the parens around the cd and make -f ... so those commands are run in a 'new' shell so when the make exits, execution is in the original top level directory and ready to loop to do it all again in the next directory.

Automake, GNU make, check, and (ignored) pattern rules

I have the following Makefile.am which is supposed to create foo.hdb and foo.cdb from foo.h (via the Python script):
TESTS = check_foo
check_PROGRAMS = check_foo
check_foo_SOURCES = check_foo.c $(top_builddir)/src/isti.h \
foo.cdb foo.h foo.hdb
check_foo_CFLAGS = #CHECK_CFLAGS# $(all_includes) -I../../clib/src/
check_foo_LDADD = $(top_builddir)/src/libcorm.la #CHECK_LIBS# -lsqlite3
%.hdb %.cdb: %.h
PYTHONPATH=$(top_builddir)/cgen/src python $(top_builddir)/cgen/src/isti/cgen/run.py $<
clean-local:
rm -f *.hdb *.cdb
However, although make foo.hdb and make foo.cdb work (call the Python code and generates the foo.hdb and foo.cdb files from foo.h), make clean check (or the two separately) does not (missing foo.hdb - no such file) - the pattern rule is not called to generate foo.hdb from foo.h.
In other words: the pattern rule is not being called for the files listed in check_foo_SOURCES.
How can I make this work? The rest of the autotools infrastructure is working fine. From looking at the Makefile I suspect the issue is with how autotools expands the check sources.
This is all on Linux with Gnu make. Here is the Makefile.
[Updated slightly to reflect the help from MadScientist].
Later update
The following Makefile (just make, not autotools) works fine, so the issue seems to be related to autotools and check support.
all: check_foo
CFLAGS=-I../../clib/src
LDFLAGS=-L../../clib/src/.libs
check_foo: check_foo.c foo.h corm_foo.h corm_foo.c
gcc $(CFLAGS) $(LDFLAGS) $^ -o $# -lcorm -lsqlite3
corm_%.h corm_%.c: %.h
PYTHONPATH=../../cgen/src python ../../cgen/src/isti/cgen/run.py $<
clean:
rm -f corm_*.h corm_*.c
rm -f *.o
(Note that I've switched from xxx.hdb to corm_xxx.h, etc, so that file extensions remain OK).
More Details
Since it seems to be related to the CHECK macros, this is configure.ac:
AC_INIT([corm], [0.1], [a.cooke#isti.com])
AC_CONFIG_MACRO_DIR([m4])
PKG_CHECK_MODULES([CHECK], [check >= 0.9.4])
AM_INIT_AUTOMAKE([-Wall foreign -Werror])
AC_PROG_CC_C99
AM_PROG_CC_C_O
LT_INIT
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([Makefile clib/Makefile clib/src/Makefile clib/tests/Makefile clib/docs/Makefile cgen/Makefile cgen/docs/Makefile example/Makefile example/src/Makefile])
AC_CHECK_PROGS([DOXYGEN], [doxygen], AC_MSG_WARN([Doxygen not found - continuing without Doxygen support]))
AM_CONDITIONAL([HAVE_DOXYGEN], [test -n "$DOXYGEN"])
AM_COND_IF([HAVE_DOXYGEN], [AC_CONFIG_FILES([clib/docs/Doxyfile cgen/docs/Doxyfile])])
SOLUTION
OK, so summarizing the various things below, there were two important issues (once I had fixed file extensions - see the "plain" makefile and fceller's answer), either one of which was sufficient to make things work:
(The handling of) Header files is complicated. Because of auto-dependencies, programatically generated header files break things. The solution is to use BUILT_SOURCES
But (the handling of) .c files is not complicated. So putting the corm_foo.c in front of check_foo.c would trigger the generation of that file. Since that also generates corm_foo.h, everything works (because check_foo.c now can include corm_foo.h).
Also, fceller has some good general points about tidy makefiles and explains why the "plain" makefile works.
The line
%.cdb: %.hdb
does not do anything. Check the GNU make manual and you'll see that a pattern rule without a command line is used to DELETE a previously defined rule with that same pattern. Since there's no previous rule, this is essentially a no-op.
If you have a command that creates TWO output files with ONE invocation of a rule, then you need to put both patterns in the same rule, like this:
%.cdb %.hdb: %.h
PYTHONPATH=$(top_builddir)/cgen/src python $(top_builddir)/cgen/src/isti/cgen/run.py $<
This will tell GNU make that both targets are generated from one invocation of the rule. BE AWARE! This syntax only has this behavior for pattern rules. Explicit rules with multiple targets do something entirely different (confusingly enough).
As for the make clean behavior, I'm not sure. The makefile generated by automake is too complex for me to just read it; it would require some debugging (run it and trace what happens). However, I suspect that the rules there are not designed properly to allow a clean target and a build target to both be run in the same invocation of make. Run the two commands separately and see if that works better:
make clean
make check
You need to tell automake that foo.hdb is a source file that is to be constructed. Add the following to Makefile.am:
BUILT_SOURCES = foo.hdb
First of all: you do not need to include the "*.h" in *_SOURCES. The automake will generated code to generate the dependencies. From the manual: Header files listed in a _SOURCES definition will be included in the distribution but otherwise ignored
The change you made in the plain Makefile ("Note that I've switched from xxx.hdb to corm_xxx.h, etc, so that file extensions remain OK") is essential. The automake FILTERS the *_SOURCES list using the filename extension in order to see what to call (CC, CXX, F77).
The following Makefile.am will work:
TEST = check_foo
check_PROGRAMS = check_foo
check_foo_SOURCES = check_foo.c foo.db.c
check_foo_CFLAGS =
check_foo_LDADD =
%.db.c %.db.h: %.h
echo "int i = 1;" > foo.db.c
echo "int j;" > foo.db.h
clean-local:
rm -f *.db.h *.db.c
Let me try to add an indirect answer/discussion to the existing direct ones.
I recently moved away from make for the same kind of reasons you experienced: it is extremely powerful but sometimes a bit difficult to debug when things do not go as expected.
I recently discovered pydoit which is very promising as a debuggable replacement build tool for make. Since the notion of "pattern rules" was not present in it, I proposed an independent package to do the job: fprules.
This is how you would perform the same task that you mention in your post, with doit and fprules:
from fprules import file_pattern
# all: check_foo
DOIT_CONFIG = {'default_tasks': ['check_foo']}
CFLAGS = '-I../../clib/src'
LDFLAGS = '-L../../clib/src/.libs'
# check_foo: check_foo.c foo.h corm_foo.h corm_foo.c
# gcc $(CFLAGS) $(LDFLAGS) $^ -o $# -lcorm -lsqlite3
def task_check_foo():
"""
Compiles the `check_foo` executable
"""
src_files = ('check_foo.c', 'foo.h', 'corm_foo.h', 'corm_foo.c')
dst_file = 'check_foo'
return {
'file_dep': src_files,
'actions': ["gcc %s %s %s -o %s -lcorm -lsqlite3" % (CFLAGS, LDFLAGS, ' '.join(src_files), dst_file)],
'verbosity': 2,
'targets': [dst_file],
'clean': True
}
# corm_%.h corm_%.c: %.h
# PYTHONPATH=../../cgen/src python ../../cgen/src/isti/cgen/run.py $<
def task_gen_corm():
"""
Generates files `corm_%.h` and `corm_%.c`
for each header file `%.h`.
"""
for data in file_pattern('./*.h', dict(h_file='./corm_%.h', c_file='./corm_%.c')):
yield {
'name': data.name,
'file_dep': [data.src_path],
'actions': ["PYTHONPATH=../../cgen/src python ../../cgen/src/isti/cgen/run.py %s" % data.src_path],
'verbosity': 2,
'targets': [data.h_file, data.c_file],
'clean': True
}
# clean:
# rm -f corm_*.h corm_*.c
# rm -f *.o
# No need to create tasks for this:
# with 'clean': True, `doit clean` will clean all target files
Then simply run doit in the folder.
Do not hesitate to provide feedback on the projects pages if needed: for example multiline commands are not supported, you can vote for them if you too feel that they are missing: https://github.com/pydoit/doit/issues/314

Resources