I have a source tree:
/bootloader
/firmware
/system
and want to manage two separate builds for firmware and bootloader, each of them using common system sources but compiles them differently (i.e. with its own set of options).
Builds must be out-of-tree.
Obvious "repository" feature of makepp is not a solution here, because it breaks this principle. Symbolic links are not solution too, because it must work on Windows.
The problem actually is in shared system sources, whose relative path structure differs from others, causing common pattern rules not work for them:
BUILD_PATH = $(relative_to $(PROJECT_PATH), .)/BUILD/$(relative_to ., $(PROJECT_PATH)) # trick to be able to extend rules for specific files at different subtree levels (if we use Makeppfile for each level)
...
$(BUILD_PATH)/%.o : %.c
...
Approach with single RootMakeppfile and include *.mk files (instead loading them) also doesn't allow me to do something like that:
$(BUILD_ROOT_PATH)/*/%.o : %.c
I've tried a lot of totally different approaches. It's not such trivial, as it seems to be at first glance. Please, help.
I finally managed how to solve problem.
Solution is to divide Rootmakeppfile project to two Rootmakeppfile subprojects (both loading ../system makeppfile in its own way), and perform build separately for each subproject (in its own build directory). Project structure with makefiles:
/bootloader/
...
BUILD/
Rootmakeppfile
/firmware/
...
BUILD/
Rootmakeppfile
/system/
...
Makeppfile
/project.mk
Common definitions .mk file should contain following lines:
BUILD_ROOT_DIR = $(relative_to $(SUBPROJECT_PATH), .)/BUILD
BUILD_OBJ_REL_PATH = $(BUILD_ROOT_DIR)/$(relative_to ., $(BUILD_OBJ_REL_DIR))
BUILD_OBJ_REL_DIR ?= $(SUBPROJECT_PATH)
Each subproject should include system as follows:
load_makefile BUILD_OBJ_REL_DIR="$(SUBPROJECT_PATH)/.." ../system
and define:
global SUBPROJECT_PATH := $(abspath .)
This approach makes possible to define pattern rules as follows:
$(BUILD_OBJ_REL_PATH)/%.o : %.c
Hope, it helps someone!
Related
CMake offers several ways to specify the source files for a target.
One is to use globbing (documentation), for example:
FILE(GLOB MY_SRCS dir/*)
Another method is to specify each file individually.
Which way is preferred? Globbing seems easy, but I heard it has some downsides.
Full disclosure: I originally preferred the globbing approach for its simplicity, but over the years I have come to recognise that explicitly listing the files is less error-prone for large, multi-developer projects.
Original answer:
The advantages to globbing are:
It's easy to add new files as they
are only listed in one place: on
disk. Not globbing creates
duplication.
Your CMakeLists.txt file will be
shorter. This is a big plus if you
have lots of files. Not globbing
causes you to lose the CMake logic
amongst huge lists of files.
The advantages of using hardcoded file lists are:
CMake will track the dependencies of a new file on disk correctly - if we use
glob then files not globbed first time round when you ran CMake will not get
picked up
You ensure that only files you want are added. Globbing may pick up stray
files that you do not want.
In order to work around the first issue, you can simply "touch" the CMakeLists.txt that does the glob, either by using the touch command or by writing the file with no changes. This will force CMake to re-run and pick up the new file.
To fix the second problem you can organize your code carefully into directories, which is what you probably do anyway. In the worst case, you can use the list(REMOVE_ITEM) command to clean up the globbed list of files:
file(GLOB to_remove file_to_remove.cpp)
list(REMOVE_ITEM list ${to_remove})
The only real situation where this can bite you is if you are using something like git-bisect to try older versions of your code in the same build directory. In that case, you may have to clean and compile more than necessary to ensure you get the right files in the list. This is such a corner case, and one where you already are on your toes, that it isn't really an issue.
The best way to specify sourcefiles in CMake is by listing them explicitly.
The creators of CMake themselves advise not to use globbing.
See: https://cmake.org/cmake/help/latest/command/file.html?highlight=glob#glob
(We do not recommend using GLOB to collect a list of source files from your source tree. If no CMakeLists.txt file changes when a source is added or removed then the generated build system cannot know when to ask CMake to regenerate.)
Of course, you might want to know what the downsides are - read on!
When Globbing Fails:
The big disadvantage to globbing is that creating/deleting files won't automatically update the build-system.
If you are the person adding the files, this may seem an acceptable trade-off, however this causes problems for other people building your code, they update the project from version-control, run build, then contact you, complaining that"the build's broken".
To make matters worse, the failure typically gives some linking error which doesn't give any hints to the cause of the problem and time is lost troubleshooting it.
In a project I worked on we started off globbing but got so many complaints when new files were added, that it was enough reason to explicitly list files instead of globbing.
This also breaks common git work-flows(git bisect and switching between feature branches).
So I couldn't recommend this, the problems it causes far outweigh the convenience, when someone can't build your software because of this, they may loose a lot of time to track down the issue or just give up.
And another note, Just remembering to touch CMakeLists.txt isn't always enough, with automated builds that use globbing, I had to run cmake before every build since files might have been added/removed since last building *.
Exceptions to the rule:
There are times where globbing is preferable:
For setting up a CMakeLists.txt files for existing projects that don't use CMake.Its a fast way to get all the source referenced (once the build system's running - replace globbing with explicit file-lists).
When CMake isn't used as the primary build-system, if for example you're using a project who aren't using CMake, and you would like to maintain your own build-system for it.
For any situation where the file list changes so often that it becomes impractical to maintain. In this case it could be useful, but then you have to accept running cmake to generate build-files every time to get a reliable/correct build (which goes against the intention of CMake - the ability to split configuration from building).
* Yes, I could have written a code to compare the tree of files on disk before and after an update, but this is not such a nice workaround and something better left up to the build-system.
In CMake 3.12, the file(GLOB ...) and file(GLOB_RECURSE ...) commands gained a CONFIGURE_DEPENDS option which reruns cmake if the glob's value changes.
As that was the primary disadvantage of globbing for source files, it is now okay to do so:
# Whenever this glob's value changes, cmake will rerun and update the build with the
# new/removed files.
file(GLOB_RECURSE sources CONFIGURE_DEPENDS "*.cpp")
add_executable(my_target ${sources})
However, some people still recommend avoiding globbing for sources. Indeed, the documentation states:
We do not recommend using GLOB to collect a list of source files from your source tree. ... The CONFIGURE_DEPENDS flag may not work reliably on all generators, or if a new generator is added in the future that cannot support it, projects using it will be stuck. Even if CONFIGURE_DEPENDS works reliably, there is still a cost to perform the check on every rebuild.
Personally, I consider the benefits of not having to manually manage the source file list to outweigh the possible drawbacks. If you do have to switch back to manually listed files, this can be easily achieved by just printing the globbed source list and pasting it back in.
You can safely glob (and probably should) at the cost of an additional file to hold the dependencies.
Add functions like these somewhere:
# Compare the new contents with the existing file, if it exists and is the
# same we don't want to trigger a make by changing its timestamp.
function(update_file path content)
set(old_content "")
if(EXISTS "${path}")
file(READ "${path}" old_content)
endif()
if(NOT old_content STREQUAL content)
file(WRITE "${path}" "${content}")
endif()
endfunction(update_file)
# Creates a file called CMakeDeps.cmake next to your CMakeLists.txt with
# the list of dependencies in it - this file should be treated as part of
# CMakeLists.txt (source controlled, etc.).
function(update_deps_file deps)
set(deps_file "CMakeDeps.cmake")
# Normalize the list so it's the same on every machine
list(REMOVE_DUPLICATES deps)
foreach(dep IN LISTS deps)
file(RELATIVE_PATH rel_dep ${CMAKE_CURRENT_SOURCE_DIR} ${dep})
list(APPEND rel_deps ${rel_dep})
endforeach(dep)
list(SORT rel_deps)
# Update the deps file
set(content "# generated by make process\nset(sources ${rel_deps})\n")
update_file(${deps_file} "${content}")
# Include the file so it's tracked as a generation dependency we don't
# need the content.
include(${deps_file})
endfunction(update_deps_file)
And then go globbing:
file(GLOB_RECURSE sources LIST_DIRECTORIES false *.h *.cpp)
update_deps_file("${sources}")
add_executable(test ${sources})
You're still carting around the explicit dependencies (and triggering all the automated builds!) like before, only it's in two files instead of one.
The only change in procedure is after you've created a new file. If you don't glob the workflow is to modify CMakeLists.txt from inside Visual Studio and rebuild, if you do glob you run cmake explicitly - or just touch CMakeLists.txt.
Specify each file individually!
I use a conventional CMakeLists.txt and a python script to update it. I run the python script manually after adding files.
See my answer here:
https://stackoverflow.com/a/48318388/3929196
I'm not a fan of globbing and never used it for my libraries. But recently I've looked a presentation by Robert Schumacher (vcpkg developer) where he recommends to treat all your library sources as separate components (for example, private sources (.cpp), public headers (.h), tests, examples - are all separate components) and use separate folders for all of them (similarly to how we use C++ namespaces for classes). In that case I think globbing makes sense, because it allows you to clearly express this components approach and stimulate other developers to follow it. For example, your library directory structure can be the following:
/include - for public headers
/src - for private headers and sources
/tests - for tests
You obviously want other developers to follow your convention (i.e., place public headers under /include and tests under /tests). file(glob) gives a hint for developers that all files from a directory have the same conceptual meaning and any files placed to this directory matching the regexp will also be treated in the same way (for example, installed during 'make install' if we speak about public headers).
I'm working on a C++ project at work where we need to develop a small piece for a larger application. We were given headers and static libraries for all of the code that we should need to reference. These are strewn throughout multiple folders and we placed all of that inside a common folder.
When writing our code, we'll need to include the headers and libraries as a part of our compilation process. Is there an elegant solution to doing this in a Makefile, or do I have to explicitly list each include folder with -I , each library folder with -L , and each library with -l?
Or is there an alternative to a Makefile that might make sense for this?
Edit: Here is an example of the folder structure:
common
folder1
subfolder1
include
libs
subfolder2
...
subfolder10
folder2
...
...
folder10
code
makefile
ourStuff
There are multiple levels of folders under common containing headers and libraries. We need to include code from there.
It was also asked why we don't just explicitly list the path in our #include statements. This code will be living in the main application once we're done, and it doesn't exactly follow the folder structure we were given.
Well, given the above structure it's simple enough to generate the things you want. For example if you want to add all -I... flags to CXXFLAGS, you can use:
INCDIRS := $(wildcard ../common/*/*/include)
CXXFLAGS += $(addprefix -I,$(INCDIRS))
Similar for -L flags:
LIBDIRS := $(wildcard ../common/*/*/libs)
LDFLAGS += $(addprefix -L,$(LIBDIRS))
Linking all the libraries is slightly more complicated. Assuming they're all static libraries you can do something like this:
LIBFILES := $(notdir $(wildcard ../common/*/*/libs/lib*.a))
LDLIBS += $(patsubst lib%.a,-l%,$(LIBFILES))
Of course this is assuming you don't have any naming conflicts / all libraries are unique.
Obviously, your question can be formulated like this: "Do I have to write a plethora of include paths or is there some managed/automatic way to do that". The question may pop up in the context of a makefile but this is mainly because make does not try to cloak the complexity of software building from the programmer. Trying to evade to another build system buys you nothing if the components you are using were not fitted into the larger build algorithm by their original programmers. If you receive pre-configured build parts (e.g. in form of a CMake project) then you save a great deal of work, needing only to tie together some abstraction level high up in the hierarchy. The downside of this is that you are locked in this build methodology now, possibly with more ramifications radiating out into parts of your project where they do as much harm as good. You may want to read this thread here: https://softwareengineering.stackexchange.com/questions/407056/the-case-against-path-expressions-in-include-directives
The cheapest way to at least partially achieve what you want to do in GNUmake is to use the function wildcard-rec (see https://github.com/markpiffer/gmtt#call-wildcard-reclist-of-globs) which has a fairly flexible input-output-relation. You can e.g. collect all paths which are of the form project/component_a/**/include/ in a whole subtree, or all header files in such a path with project/component_a/**/include/*.h.
PS: simply include gmtt.mk at the top of your makefiles.
I have project which has not been divided into libraries, but the source is organized in a directory tree. I do not know how to tell cmake to go down a directory, then add the source in that directory to project defined in the parent directory. I have attempted the following:
in project/source/CMakelists.txt:
set(SOURCE
${CMAKE_CURRENT_SOURCE_DIR}/unitTest/main.cpp
)
add_subdirectory("${PROJECT_SOURCE_DIR}/folder1")
add_executable(UnitTestRNG ${SOURCE} ${HEADERS})
then in project/source/folder1/CMakeLists.txt:
set(SOURCE
${SOURCE}
${CMAKE_CURRENT_SOURCE_DIR}/file1.cpp
${CMAKE_CURRENT_SOURCE_DIR}/file2.cpp
)
set(HEADERS
${HEADERS}
${CMAKE_CURRENT_SOURCE_DIR}/file1.hpp
${CMAKE_CURRENT_SOURCE_DIR}/file2.hpp
)
using some message() statements, I have found that the the child folder will get the contents of the SOURCE variable, but it's new assignment to that variable will not persist on returning to the parent CMakeLists.txt
Looking for examples and at the cmake tutorial has led me to the conclusion that:
- Source file structures are usually flat within a project
- If code is divided into folders, it is usually is divided into corresponding libraries.
I wonder if there is some "best practice" from which I am deviating by attempting this structure.
Since CMake 3.1 there is a new way to add source from subdirectories: target_sources
Say you have root_dir and root_dir/sub_dir and source files in both. With target_sources you can do this:
In root_dir/CMakeLists.txt define the target
add_library(some_target main.cpp)
add_subdirectory(sub_dir)
In root_dir/sub_dir/CMakeLists.txt add sources:
target_sources(some_target PRIVATE more_cool_stuff.cpp)
some_target will now contain both source files.
It is also possible to use other commands in root_dir/sub_dir/CMakeLists.txt using some_target, for example target_compile_definitions which is quite convenient to add compilation definitions.
I learned about target_sources here, check it out if you want more explanation and examples
Like the second part of arrowdodger's answer says:
in project/source/folder1/CMakeLists.txt:
set(SOURCE
${SOURCE}
${CMAKE_CURRENT_SOURCE_DIR}/file1.cpp
${CMAKE_CURRENT_SOURCE_DIR}/file2.cpp
PARENT_SCOPE
)
set(HEADERS
${HEADERS}
${CMAKE_CURRENT_SOURCE_DIR}/file1.hpp
${CMAKE_CURRENT_SOURCE_DIR}/file2.hpp
PARENT_SCOPE
)
Can't you just set all your sources in project/source/CMakelists.txt then?
Anyway, what you need is PARENT_SCOPE or CACHE option on set command.
I have a C project that has the following structure
Main/
Makefile.am
bin/
src/
Makefile.am
main.c
SomeLibrarySource/
SomeFuncs.c
SomeFuncs.h
The main.c contains the main function that uses functions defined in the SomeFuncs.{h/c} files.
I want to use autotools for this project. I read a couple of resources on autotools. But, I was only able to manage using autotools for a single level project where all source, object and other files reside in the same directory.
Then I got some links that talked about using autotools for deep projects like this one and then I got confused.
Right now I have two Makefile.am as follows
Makefile.am
SUBDIRS=src
src/Makefile.am
mainprgdir=../
mainprg_PROGRAMS=main
main_SOURCES=main.c
I am pretty sure that these files should not be as I have them now :P
How do I use autotools for the above project structure? (At least what should be there in those Makefile.am(s) and where should I place them.
EDIT:
One more thing! At the end I would like to have the object files created in the bin directory.
Thanks
mainprogdir=../ does not make a whole lot of sense (you don't know what it is relative to on installation). Probably intended:
# Main/Makefile.am
# .━━ target for `make install`
# |
# ↓ ↓━━ target for compilation
bin_PROGRAMS = bin/main
# ↓━━ based upon compilation target name
bin_main_SOURCES = src/main.c
There are two main approaches. If the functions in SomeLibrarySource are used only by main, then there's no need to build a separate library and you can simply specify the source files in src/Makefile.am
main_SOURCES = main.c SomeLibrarySource/SomeFuncs.c
However, if you actually want to use the functions in other code in your tree, you do not want to compile SomeFuncs.c multiple times but should use a convenience library.
# Assigning main_SOURCES is redundant
main_SOURCES = main.c
main_LDADD = SomeLibrarySource/libSomeFuncs.a
noinst_LIBRARIES = SomeLibrarySource/libSomeFuncs.a
AM_CPPFLAGS = -I$(srcdir)/SomeLibrarySource
(You'll need AC_PROG_RANLIB in configure.ac to use convenience libraries.)
If the source file is named SomeFuncs.c, automake will not need Makefile.am to specify SomeLibrarySource_libSomeFuncs_a_SOURCES, but if the name of the source code file does not match the name specified in noinst_LIBRARIES, SomeLibrarySource_libSomeFuncs_a_SOURCES should be set to the list of files used to build the library. Note that you do not need to specify main_SOURCES, since main.c is the default value if left unspecified (but it's not a bad idea to be explicit.) (In all of this, I am not comfortable use CamlCase names, but the system I'm using uses a case insensitive file system (biggest mistake apple ever made) and the examples I give here are working for me. YMMV)
You could of course do a recursive make, or build the library as a separate project and install it. (I like the final option. Libraries with useful features should exist on their own.)
I've got several directories with subdirectories containing c or asm files and I want them all compiled/assembled and then linked. I'm not especially picky where the object files go (e.g. a special bin folder or in the src folder) as long as a make clean removes them all.
The structure would look something like this:
/src
/dir1
/dir1_1
+file1_1.s
+file1_2.s
+file1.s
/dir2
+file2.c
I'm sure there's some easy way to create a makefile that compiles all files without me having to specify where it should look (compiling all files in one directory is doable with wildcards, but what then?).
Do a Google search for 'recursive make considered harmful'. You'll find the original article which postulates that the recursive make procedure is a bad way of doing business, and you'll find some links to other places which debate the validity of the proposition.
Basically, there are two ways to do builds in a directory hierarchy (with make).
Recursive make: each directory contains a makefile which builds in sub-directories and then builds in the current directory.
Non-recursive make: the makefile includes all the dependent makefiles, and builds up the complete dependency structure for the entire project and only builds the requisite software.
I work routinely on a product where the main build sequence is driven by a hybrid system that uses a shell script plus one makefile for each directory. One section of the product is managed by a 'RMCH' makefile; most of it is not. The build script deals with phases of the build, and sequences the directories, and runs make in each directory when it is time to do so. (The source code is in 20k+ files spread over a multitude of directories - it is a big project/product.)
I've also converted a medium-small project (about 20 directories of relevance, and about 400 source files) to work with RMCH (from a script + makefile-per-directory system). It was a bit mind-blowing at first, but works pretty neatly now it is done. Whether I did it correctly is open for debate; it was primarily a learning exercise, though I also did some work modifying the code to work with a modern curses library instead of the archaic BSD library that was used as a part of the code (archaic, as in 1982-vintage - the code was last seriously developed in about 1986) and generally upgrading to modern (standard C) standards. It was also a chance to work with git - so, all in all, quite an extensive learning experience.
If you can wrap your brain around RMCH, it is a good system. If done correctly, with complete and accurate dependency tracking, it removes the guess-work from the build sequence, and it does run fast. However, migrating even a medium size project to it is fairly hard work - it would be a daunting task to do it on the main product I work on, though the system might well benefit from it.
An alternative is to look at other alternatives to make, such as cmake, rake, scons, bras, imake, or ant or whatever else takes your fancy. Most of those are easily discoverable via a Google search; the hard one is bras, which is based on Tcl (as in Tcl/Tk), but is probably largely dead now. And imake is mentioned more for completeness than as a serious suggestion. You might also look at the GNU Autotools. Those do not abandon make; they build atop make.
If your project is small enough, you might get away with using a single hand-crafted makefile instead of a more sophisticated build system: check out the manual page on transformation functions to see what's possible.
Your example project could be compiled with the following non-recursive makefile:
targets = $(patsubst %$(1),%$(2),$(foreach dir,$(3),$(wildcard $(dir)/*$(1))))
asmdirs := src/dir1 src/dir1/dir1_1
cdirs := src/dir2
asmobjects := $(call targets,.s,.o,$(asmdirs))
cobjects := $(call targets,.c,.o,$(cdirs))
.PHONY : all clean
all : $(asmobjects) $(cobjects)
clean :
rm -f $(asmobjects) $(cobjects)
$(cobjects) : %.o : %.c
gcc -o $# -c $<
$(asmobjects) : %.o : %.s
gcc -o $# -c $<
However, because make can access the shell, you could also use standard unix tools like find instead of the somewhat limited builtin functions, eg
asmsources := $(shell find src -name '*.s')
csources := $(shell find src -name '*.c')
asmobjects := $(asmsources:.s=.o)
cobjects := $(csources:.c=.o)