Is it possible to expand % twice for a dependency name?
I have project split into several sections, where each section is in it's own folder and file of the same name, e.g.
sections/first/first.c,sections/hello/hello.c, etc.
Now I would like to build for each section a target in another folder build; I've come up with the following
SECTIONS = $(wildcard sections/*/*.c)
TARGETS = $(addprefix build/,$(notdir $(SECTIONS)))
$(TARGETS): build/%.c: sections/%/%.c
# do something
The problem however is that % is expanded only once, so I end up with an error
make: *** No rule to make target 'sections/first/%.c', needed by 'build/first.o'.
Is there a way to expand % twice?
It's not possible to get the % to expand more than one time. If you need this you'll have to do something more fancy: you can use either secondary expansion or eval. For example, this should work:
.SECONDEXPANSION:
$(TARG) : build/%.c : source/$$*/$$*.c
#echo copy $< $#
For more in-depth discussion you can check these blog posts.
SECTIONS = $(wildcard sections/*)
.PHONY: all
all: $(SECTIONS)
$(foreach dir,$(SECTIONS), \
( cd $d && $(MAKE) -f ../makefile.bot name=$(dir) all ); )
Where the above is in a top level makefile
and a second makefile (in this case makefile.bot) performs the actual work.
a similar rule as the all rule for clean install,etc would be in the top level makefile, but the target would be changed from all to clean, etc
The makefile.bot would use $name to set the executable name, etc
Notice the use of the parens around the cd and make -f ... so those commands are run in a 'new' shell so when the make exits, execution is in the original top level directory and ready to loop to do it all again in the next directory.
Related
With the Makefile I'm working on, I convert pdf files into txt files.
I've implemented a clean target that would remove all .txt files. However, I do not wish to delete the source files, only those that have been generated.
Example:
I have following files in my folder:
pdfsource.pdf and donotharm.txt
Running my makefile would create following file:
pdfsource.txt
For now, my clean looks like this:
rm -f *.txt
Using make clean would not only delete pdfsource.txt, which is desired, but also donotharm.txt.
I think I could use: .PRECIOUS: donotharm.txt, but this is really specific. I'd like to have a general solution to this.
Thanks in advance!
You can list the generated files in a make variable and use it to clean only these:
PDF := $(wildcard *.pdf)
TEXT := $(patsubst %.pdf,%.txt,$(PDF))
...
clean:
rm -f $(TEXT)
Or, if you prefer a more compact (but a bit less readable) form:
clean:
rm -f $(patsubst %.pdf,%.txt,$(wildcard *.pdf))
Of course, this works only if there is no {foo.pdf,foo.txt} pair for which you want to preserve foo.txt from deletion by make clean.
Note: using make variables, in such a case, is usually a good idea because they can be shared among various rules. Example:
PDF := $(wildcard *.pdf)
TEXT := $(patsubst %.pdf,%.txt,$(PDF))
.PHONY: all clean
all: $(TEXT)
$(TEXT): %.txt: %.pdf
pdftotext $< $#
clean:
rm -f $(TEXT)
Another approach: "make -nps" gives you all make's metadata about dependencies. For any intermediate file, it prints
filename: ...
So you can exactly delete such files with a generic "clean" rule:
clean:; MAKEFLAGS= ${MAKE} -j1 -spinf $(word 1,${MAKEFILE_LIST}) \
| sed -n '/^# I/,$${/^[^\#\[%.][^ %]*: /s/:.*//p;}; 1s|.*|${clean}|p' | xargs rm -rf
The first line handles use of makefiles other than the defaults (makefile, GNUmakefile, Makefile)
In the "sed" command:
/^# I/,$
... selects the zone of make metadata with dependencies.
/^[^\#\[%.][^ %]*: /
... filters out comments, implicit rules, and files with no dependencies (the trailing space). It doesn't filter out phony targets; oh well.
Finally:
1s|.*|${clean}|p
adds any explicit targets for "clean" -- what you know that make does not; e.g.
clean += tmpdir/* *.gcda
Say I have a list of source files and each are to be compiled to separate binaries:
SRCS = abcd.c efgh.c ijkl.c
And I want output files in separate subdirectories based on the file names like this:
build/abcd/abcd
build/efgh/efgh
build/ijkl/ijkl
I'm thinking a static pattern rule is the way to go. The pseudo-make-rule can be something like:
$(TARGETS): build/%/%: %.c
# stuff ...
I started by making a list of the subdirectories based on the filenames:
DIRS = $(SRCS:%.c=build/%)
So now we have DIRS = build/abcd build/efgh build/ijkl. I thought I can make the list of targets now with something like:
BLDS = $(DIRS:%=%/$(basename %))
But of course this doesn't work since the wildcard can not be used multiple times within a pattern. Therefore I'm now stuck at BLDS = build/abcd/% build/efgh/% build/ijkl/%.
Obviously I'm totally going about this the wrong way. How would you go about this?
For now I'm writing each rule explicitly, which is starting to get a bit tedious:
compile = # command to do stuff
BD = build
all: $(BD)/abcd/abcd $(BD)/efgh/efgh $(BD)/ijkl/ijkl
$(BD)/abcd/abcd: abcd.c
$(call compile)
$(BD)/efgh/efgh: efgh.c
$(call compile)
$(BD)/ijkl/ijkl: ijkl.c
$(call compile)
clean:
rm -rf build/*
.PHONY: all
I believe this does what you want:
SRCS:=abcd.c efgh.c ijkl.c
# We could fold NAMES into BLDS's definition if NAMES is not used elsewhere.
NAMES:=$(SRCS:%.c=%)
BLDS:=$(foreach name,$(NAMES),$(subst foo,$(name),build/foo/foo))
# We don't use DIRS below but the question had this variable.
DIRS:=$(dir $(BLDS))
TARGETS:=$(BLDS)
.PHONY: all
all: $(TARGETS)
.SECONDEXPANSION:
$(TARGETS): $$(notdir $$#).c
#echo Build $# from $^
mkdir -p $(dir $#)
touch $#
There are two important changes. The first is to reorder how the variables are created, and use subst, which allows replacing a matched string multiple times. The second is to use secondary expansion so that make builds rules for each of your targets. You initially a pattern with two %, but the docs say:
A pattern rule looks like an ordinary rule, except that its target contains the character `%' (exactly one of them).
(Emphasis added.)
I've tested the above with fake files abcd.c efgh.c and ijkl.c and get the following output:
$ make
Build build/abcd/abcd from abcd.c
mkdir -p build/abcd/
touch build/abcd/abcd
Build build/efgh/efgh from efgh.c
mkdir -p build/efgh/
touch build/efgh/efgh
Build build/ijkl/ijkl from ijkl.c
mkdir -p build/ijkl/
touch build/ijkl/ijkl
So I've got the following folder structure
makefile
src/my_lib.c
src/myhead.h
and I'm trying to compile *my_lib.c* with the header myhead.h as a library. This is the makefile. I attempt to put the obj files in OBJFOLDER and the compiled library in the OUTPUTFOLDER
PLUGNAME=my_lib
SOURCEFOLDER=src
OUTPUTFOLDER=bin
OBJFOLDER=bin/obj
OBJS=$(PLUGNAME).o
DEPS=myhead.h
# Configuration finishes here
_OBJS = $(patsubst %,$(OBJFOLDER)/%,$(OBJS))
_DEPS = $(patsubst %,$(SOURCEFOLDER)/%,$(DEPS))
ifeq ($(OS),Windows_NT)
EXT = .dll
else
UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S),Linux)
EXT = .so
endif
endif
all : $(OUTPUTFOLDER)/$(PLUGNAME)$(EXT)
$(OUTPUTFOLDER)/$(PLUGNAME)$(EXT) : $(_OBJS)
gcc -Wl,--add-stdcall-alias -shared -o $# $(_OBJS)
$(OBJFOLDER)/%.o: $(SOURCEFOLDER)/%.c $(_DEPS)
mkdir -p $(OUTPUTFOLDER)
mkdir -p $(OBJFOLDER)
gcc $(foreach d, $(INC), -I$d) -c $< -o $#
.PHONY: clean
clean :
rm -f $(OBJFOLDER)/*.o $(OUTPUTFOLDER)/$(PLUGNAME)$(EXT) $(SOURCEFOLDER)/TSDRPlugin.h
When I do make all it fails
make: *** No rule to make target `bin/obj/my_lib.o', needed by `bin/
my_lib.dll'. Stop.
I have no idea how this could be possible since I already have defined
$(OBJFOLDER)/%.o: $(SOURCEFOLDER)/%.c $(_DEPS)
Strangely if I change the above line in the makefile, to
bin/obj/my_lib.o: $(SOURCEFOLDER)/%.c $(_DEPS)
I now get
make: *** No rule to make target `src/%.c', needed by `bin/obj/my_lib.o'. Stop.
Your second error is because by removing the % in the target you've turned this into an explicit rule, not a pattern rule. So, the % in the prerequisite is not replaced.
Your first error means that for some reason make is deciding that your pattern rule doesn't match. This means, usually, that make can't find and doesn't know how to create one of the prerequisites. I recommend you run make with the -d flag and see why make decides your rule doesn't apply.
What version of GNU make are you using? Some very old versions would not match pattern rules if the directory that the target was to be placed into didn't exist already.
The problem was that header was missing... Stupid mistake on my side.
I overlooked it, because this was a snippet from a longer makefile that was supposed to copy over the header but it didn't which means that this line was outputting the error. Stupid me...
I have a bunch of C files in different directories and I'm getting a make: nothing to be done for 'all' error with my recursive Makefile; however if I tweak the dependences I can get it to work... but I don't understand why I have to.
Here's my original Makefile:
APP_DIRS=rescoco ressys resvm
.PHONY: all
all: $(APP_DIRS)
$(APP_DIRS):
$(MAKE) --directory $#
clean:
$(RM) *~
Now if I change my line: .PHONY to .PHONY: all $(APP_DIRS) it builds fine.
Another possibility is if I change the line: $(APP_DIRS): to $(APP_DIRS): clean it builds fine.
(NOTE: removing .PHONY target doesn't change anything)
So what's going on here? Is the Makefile trying to tell me I haven't listed dependencies correctly? I was thinking make would do something like:
to build .PHONY I first have to build all
to build all I first have to build $(APP_DIRS)
$(APP_DIRS) has no prereqs so execute the command for that (which would cause the recursive makes to run).
Clearly I am wrong; but why?
FYI, if it matters my files are structured something like this:
Makefile #top level makefile as seen above
/rescoco
rescoco.c
Makefile #builds rescoco src and moves archive to ../lib directory
/ressys
ressys.c
Makefile #same as above but for ressys
/resvm
resvm.c
Makefile #same as above but for resvm
/lib
and my build command is simply make. When I run with make -n or make -n all I get no output at all:
:~/proj$ make -n all
make: Nothing to be done for 'all'.
:~/proj$
Things first you should be aware of:
If you have directories as dependencies, make is going to consider building the targets (i.e. executing the recipes for such directory targets), only if the modification timestamp of the directory gets updated.
This would happen only when you add a new file in the directory but not for file modifications in the directory. Adding files in a sub-directory does not change the timestamp of the directory.
PHONY targets are meant to be used when executing such a target does not create a file with the name of the target. In other words, you want make to execute the rule irrespective of whether the file already exists or not.
So your Makefile esentially only tells this:
To build the target all, I need to build $(APP_DIRS). Since all is a PHONY target, I will always execute the recipe for all.
$(APP_DIRS) is not a PHONY target and does not have any dependencies. So *only if $(APP_DIRS) does not exist already (i.e. the file or directory), I'm going to execute the recipe, otherwise I'm doing nothing for this target.
clean has no pre-requisite and not a PHONY, so I expect to execute this rule only when explicitly invoked by make (from the command line or another Makefile). Also clean is not a PHONY, so I expect the recipe to create a file called clean after execution (which is incorrect for your case)
Hence changing the .PHONY line to:
.PHONY: all $(APP_DIRS)
makes the Makefile go and execute the recipe for $(APP_DIRS) always.
So if you would like make to always traverse into all of the $(APP_DIRS) directories and invoke make again on them, you need to add $(APP_DIRS) to .PHONY, which makes $(APP_DIRS) a PHONY target, and executes the recipe irrespective of the file's/directory's timestamp if it exists.
For your particular use-case, I think this is the Makefile you should be using:
APP_DIRS=rescoco ressys resvm
.PHONY: all clean $(APP_DIRS)
all: $(APP_DIRS)
$(APP_DIRS):
$(MAKE) --directory $#
clean:
$(RM) *~
BONUS:
Changing $(APP_DIRS): to $(APP_DIRS): clean implies that $(APP_DIRS) depends on the clean target.
Although clean is not marked a PHONY, make does not see a file named clean in the current directory. So it goes ahead and tries to execute the recipe for clean.
Since a dependency of $(APP_DIRS) (i.e. clean) was built, this makes the Makefile execute the recipe for building $(APP_DIRS).
This brings us to an interesting observation:
- Any target that depends on a PHONY target will always get rebuilt (i.e. the recipe would be executed).
Take this simple Makefile:
all: target1
target1: target2
#echo "$#"
#touch $#
target2: target3
#echo "$#"
#touch $#
target3:
#echo "$#"
.PHONY: all target3
The first time I run make, I see this output:
target3
target2
target1
After this, files target1 and target2 are created. Even then, if I run make again, I would see the output:
target3
target2
target1
As you can see, the PHONY dependencies get propagated up and not the other way down. target2 gets rebuilt just because target3 is a PHONY, and target1 gets rebuilt just because target2 got rebuilt.
You are defining a variable named 'APP_DIRS' with a list of directories. That is fine.
You then do
$(APP_DIRS): make blah blah, which is essentially equivalent to rescoco ressys resvm: make blah blah
which obviously isnt valid.
So you need to pretend your $(APP_DIRS) is a variable, not a target name, which seems to be what you're using it as.
having said that, think why .PHONY: all $(APP_DIRS) works
This is how you can do it, just remove the wildcard if you don't want it.
make wildcard subdirectory targets
I have the following Makefile.am which is supposed to create foo.hdb and foo.cdb from foo.h (via the Python script):
TESTS = check_foo
check_PROGRAMS = check_foo
check_foo_SOURCES = check_foo.c $(top_builddir)/src/isti.h \
foo.cdb foo.h foo.hdb
check_foo_CFLAGS = #CHECK_CFLAGS# $(all_includes) -I../../clib/src/
check_foo_LDADD = $(top_builddir)/src/libcorm.la #CHECK_LIBS# -lsqlite3
%.hdb %.cdb: %.h
PYTHONPATH=$(top_builddir)/cgen/src python $(top_builddir)/cgen/src/isti/cgen/run.py $<
clean-local:
rm -f *.hdb *.cdb
However, although make foo.hdb and make foo.cdb work (call the Python code and generates the foo.hdb and foo.cdb files from foo.h), make clean check (or the two separately) does not (missing foo.hdb - no such file) - the pattern rule is not called to generate foo.hdb from foo.h.
In other words: the pattern rule is not being called for the files listed in check_foo_SOURCES.
How can I make this work? The rest of the autotools infrastructure is working fine. From looking at the Makefile I suspect the issue is with how autotools expands the check sources.
This is all on Linux with Gnu make. Here is the Makefile.
[Updated slightly to reflect the help from MadScientist].
Later update
The following Makefile (just make, not autotools) works fine, so the issue seems to be related to autotools and check support.
all: check_foo
CFLAGS=-I../../clib/src
LDFLAGS=-L../../clib/src/.libs
check_foo: check_foo.c foo.h corm_foo.h corm_foo.c
gcc $(CFLAGS) $(LDFLAGS) $^ -o $# -lcorm -lsqlite3
corm_%.h corm_%.c: %.h
PYTHONPATH=../../cgen/src python ../../cgen/src/isti/cgen/run.py $<
clean:
rm -f corm_*.h corm_*.c
rm -f *.o
(Note that I've switched from xxx.hdb to corm_xxx.h, etc, so that file extensions remain OK).
More Details
Since it seems to be related to the CHECK macros, this is configure.ac:
AC_INIT([corm], [0.1], [a.cooke#isti.com])
AC_CONFIG_MACRO_DIR([m4])
PKG_CHECK_MODULES([CHECK], [check >= 0.9.4])
AM_INIT_AUTOMAKE([-Wall foreign -Werror])
AC_PROG_CC_C99
AM_PROG_CC_C_O
LT_INIT
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([Makefile clib/Makefile clib/src/Makefile clib/tests/Makefile clib/docs/Makefile cgen/Makefile cgen/docs/Makefile example/Makefile example/src/Makefile])
AC_CHECK_PROGS([DOXYGEN], [doxygen], AC_MSG_WARN([Doxygen not found - continuing without Doxygen support]))
AM_CONDITIONAL([HAVE_DOXYGEN], [test -n "$DOXYGEN"])
AM_COND_IF([HAVE_DOXYGEN], [AC_CONFIG_FILES([clib/docs/Doxyfile cgen/docs/Doxyfile])])
SOLUTION
OK, so summarizing the various things below, there were two important issues (once I had fixed file extensions - see the "plain" makefile and fceller's answer), either one of which was sufficient to make things work:
(The handling of) Header files is complicated. Because of auto-dependencies, programatically generated header files break things. The solution is to use BUILT_SOURCES
But (the handling of) .c files is not complicated. So putting the corm_foo.c in front of check_foo.c would trigger the generation of that file. Since that also generates corm_foo.h, everything works (because check_foo.c now can include corm_foo.h).
Also, fceller has some good general points about tidy makefiles and explains why the "plain" makefile works.
The line
%.cdb: %.hdb
does not do anything. Check the GNU make manual and you'll see that a pattern rule without a command line is used to DELETE a previously defined rule with that same pattern. Since there's no previous rule, this is essentially a no-op.
If you have a command that creates TWO output files with ONE invocation of a rule, then you need to put both patterns in the same rule, like this:
%.cdb %.hdb: %.h
PYTHONPATH=$(top_builddir)/cgen/src python $(top_builddir)/cgen/src/isti/cgen/run.py $<
This will tell GNU make that both targets are generated from one invocation of the rule. BE AWARE! This syntax only has this behavior for pattern rules. Explicit rules with multiple targets do something entirely different (confusingly enough).
As for the make clean behavior, I'm not sure. The makefile generated by automake is too complex for me to just read it; it would require some debugging (run it and trace what happens). However, I suspect that the rules there are not designed properly to allow a clean target and a build target to both be run in the same invocation of make. Run the two commands separately and see if that works better:
make clean
make check
You need to tell automake that foo.hdb is a source file that is to be constructed. Add the following to Makefile.am:
BUILT_SOURCES = foo.hdb
First of all: you do not need to include the "*.h" in *_SOURCES. The automake will generated code to generate the dependencies. From the manual: Header files listed in a _SOURCES definition will be included in the distribution but otherwise ignored
The change you made in the plain Makefile ("Note that I've switched from xxx.hdb to corm_xxx.h, etc, so that file extensions remain OK") is essential. The automake FILTERS the *_SOURCES list using the filename extension in order to see what to call (CC, CXX, F77).
The following Makefile.am will work:
TEST = check_foo
check_PROGRAMS = check_foo
check_foo_SOURCES = check_foo.c foo.db.c
check_foo_CFLAGS =
check_foo_LDADD =
%.db.c %.db.h: %.h
echo "int i = 1;" > foo.db.c
echo "int j;" > foo.db.h
clean-local:
rm -f *.db.h *.db.c
Let me try to add an indirect answer/discussion to the existing direct ones.
I recently moved away from make for the same kind of reasons you experienced: it is extremely powerful but sometimes a bit difficult to debug when things do not go as expected.
I recently discovered pydoit which is very promising as a debuggable replacement build tool for make. Since the notion of "pattern rules" was not present in it, I proposed an independent package to do the job: fprules.
This is how you would perform the same task that you mention in your post, with doit and fprules:
from fprules import file_pattern
# all: check_foo
DOIT_CONFIG = {'default_tasks': ['check_foo']}
CFLAGS = '-I../../clib/src'
LDFLAGS = '-L../../clib/src/.libs'
# check_foo: check_foo.c foo.h corm_foo.h corm_foo.c
# gcc $(CFLAGS) $(LDFLAGS) $^ -o $# -lcorm -lsqlite3
def task_check_foo():
"""
Compiles the `check_foo` executable
"""
src_files = ('check_foo.c', 'foo.h', 'corm_foo.h', 'corm_foo.c')
dst_file = 'check_foo'
return {
'file_dep': src_files,
'actions': ["gcc %s %s %s -o %s -lcorm -lsqlite3" % (CFLAGS, LDFLAGS, ' '.join(src_files), dst_file)],
'verbosity': 2,
'targets': [dst_file],
'clean': True
}
# corm_%.h corm_%.c: %.h
# PYTHONPATH=../../cgen/src python ../../cgen/src/isti/cgen/run.py $<
def task_gen_corm():
"""
Generates files `corm_%.h` and `corm_%.c`
for each header file `%.h`.
"""
for data in file_pattern('./*.h', dict(h_file='./corm_%.h', c_file='./corm_%.c')):
yield {
'name': data.name,
'file_dep': [data.src_path],
'actions': ["PYTHONPATH=../../cgen/src python ../../cgen/src/isti/cgen/run.py %s" % data.src_path],
'verbosity': 2,
'targets': [data.h_file, data.c_file],
'clean': True
}
# clean:
# rm -f corm_*.h corm_*.c
# rm -f *.o
# No need to create tasks for this:
# with 'clean': True, `doit clean` will clean all target files
Then simply run doit in the folder.
Do not hesitate to provide feedback on the projects pages if needed: for example multiline commands are not supported, you can vote for them if you too feel that they are missing: https://github.com/pydoit/doit/issues/314