Building a Shared Library, Updating Header Files to Compiler/System Directories - linker

A friend and I are using Qt Creator with Boost to build a game engine. So far we have this idea that the engine is going to be a shared library, with the idea that we can run it with a test executable which will turn into the game we eventually want to make.
The problem is header files, mainly. I'd like to find some way for Qt Creator to be able to recognize the header files as soon as the latest builds of the engine have been built or even when they're added. At first I was thinking a script in Python which executed as a build step in Qt Creator after the engine had been built, would simply copy the header files to a system directory (/usr/include, for example - if operating on a *nix system), where the IDE would then recognize the header files when linking the engine with the test executable, and we'd also have full auto completion support.
Of course, environmental variables would be used, and while I prefer developing in Linux, my friend prefers Windows, so we agreed to take care of development in regards to our respective platform preferences.
While this seems like a good solution, I think this Python script idea may be over kill. Is there a better way to do this?
Update
From to the suggested Qmake script, I end up getting this error.
cp -f "/home/amsterdam/Programming/atlas/Engine/AtlasEngine/"AtlasEngine_global.h "/"
cp: cannot create regular file `/AtlasEngine_global.h': Permission denied
make: Leaving directory `/home/amsterdam/Programming/atlas/Engine/AtlasEngine__GCC__Linux__Debug'
make: *** [libAtlasEngine.so.1.0.0] Error 1
15:20:52: The process "/usr/bin/make" exited with code 2.
Error while building project AtlasEngine (target: Desktop)
When executing build step 'Make'
My adjustments look as follows:
# Copy over build artifacts
SRCDIR = $$ATLAS_PROJ_ROOT
DESTDIR = $$ATLAS_INCLUDE
# Look for header files there too
INCLUDEPATH += $$SRCDIR
# Dependencies: mylib. Only specify the libs you depend on.
# Leave out for building a shared library without dependencies.
#win32:LIBS += $$quote($$SRCDIR/mylib.dll)
# unix:LIBS += $$quote(-L$$SRCDIR) -lmylib
DDIR = \"$$SRCDIR/\" #<--DEFAULTS
SDIR = \"$$IN_PWD/\"
# Replace slashes in paths with backslashes for Windows
win32:file ~= s,/,\\,g
win32:DDIR ~= s,/,\\,g
win32:SDIR ~= s,/,\\,g
for(file, HEADERS) {
QMAKE_POST_LINK += $$QMAKE_COPY $$quote($${SDIR}$${file}) $$quote($$DDIR) $$escape_expand(\\n\\t)
}

I have managed to overcome this using some Qmake magic that works cross-platform. It copies over the shared libraries (either .dll or .so files) along with the header files to a directory in a directory dll at a level next to your current project.
Put this in the end of your .pro files and change the paths/libs accordingly.
# Copy over build artifacts
MYDLLDIR = $$IN_PWD/../dlls
DESTDIR = \"$$MYDLLDIR\"
# Look for header files there too
INCLUDEPATH += $$MYDLLDIR
# Dependencies: mylib. Only specify the libs you depend on.
# Leave out for building a shared library without dependencies.
win32:LIBS += $$quote($$MYDLLDIR/mylib.dll)
unix:LIBS += $$quote(-L$$MYDLLDIR) -lmylib
DDIR = \"$$MYDLLDIR/\"
SDIR = \"$$IN_PWD/\"
# Replace slashes in paths with backslashes for Windows
win32:file ~= s,/,\\,g
win32:DDIR ~= s,/,\\,g
win32:SDIR ~= s,/,\\,g
for(file, HEADERS) {
QMAKE_POST_LINK += $$QMAKE_COPY $$quote($${SDIR}$${file}) $$quote($$DDIR) $$escape_expand(\\n\\t)
}
Then adjust the LD_LIBRARY_PATH in the 'Run settings' of your project to point to that same dll directory (relatively).
Yes, it's ugly with escaping for paths with spaces and backslashes, but I found this to be working well cross-platform. Windows (XP, 7) and Linux tested. And yes it includes environment settings to be changed for running your project, but at least you don't need external (Python) scripts anymore or to install it to system directory requiring root privileges.
Improvements are welcome.

I'm not sure if anyone else would be having issues with this, but for whatever reason Qmake wasn't able to access my user specified environment variables properly.
So, since this was the case, one solution I came up with was to add the variables as Qmake configuration variable.
If you're in a UNIX based system, the first thing you're going to want to do is append the location of qmake - which should lie in your QtSDK folder - to your system $PATH, like so:
export PATH=$PATH:/path/to/QtSDK/...../qmake_root
From there, you can do something along the lines of:
qmake -set "VARIABLE" "VALUE"
In this case, I simply did:
qmake -set "ATLAS_PROJ_ROOT" $ATLAS_PROJ_ROOT.
And then I accessed it in my Qmake project file (.pro) with:
VAR = $$[ATLAS_PROJ_ROOT]
More info can be found here.

Related

How do I add .a files to Cmake [duplicate]

How to get CMake to link an executable to an external shared library that is not build within the same CMake project?
Just doing target_link_libraries(GLBall ${CMAKE_BINARY_DIR}/res/mylib.so) gives the error
make[2]: *** No rule to make target `res/mylib.so', needed by `GLBall'. Stop.
make[1]: *** [CMakeFiles/GLBall.dir/all] Error 2
make: *** [all] Error 2
(GLBall is the executable)
after I copied the library into the binary dir bin/res.
I tried using find_library(RESULT mylib.so PATHS ${CMAKE_BINARY_DIR}/res)
Which fails with RESULT-NOTFOUND.
arrowdodger's answer is correct and preferred on many occasions. I would simply like to add an alternative to his answer:
You could add an "imported" library target, instead of a link-directory. Something like:
# Your-external "mylib", add GLOBAL if the imported library is located in directories above the current.
add_library( mylib SHARED IMPORTED )
# You can define two import-locations: one for debug and one for release.
set_target_properties( mylib PROPERTIES IMPORTED_LOCATION ${CMAKE_BINARY_DIR}/res/mylib.so )
And then link as if this library was built by your project:
TARGET_LINK_LIBRARIES(GLBall mylib)
Such an approach would give you a little more flexibility: Take a look at the add_library( IMPORTED) command and the many target-properties related to imported libraries.
I do not know if this will solve your problem with "updated versions of libs".
Set libraries search path first:
link_directories(${CMAKE_BINARY_DIR}/res)
And then just do
target_link_libraries(GLBall mylib)
I assume you want to link to a library called foo, its filename is usually something link foo.dll or libfoo.so.
1. Find the library
You have to find the library. This is a good idea, even if you know the path to your library. CMake will error out if the library vanished or got a new name. This helps to spot error early and to make it clear to the user (may yourself) what causes a problem.
To find a library foo and store the path in FOO_LIB use
find_library(FOO_LIB foo)
CMake will figure out itself how the actual file name is. It checks the usual places like /usr/lib, /usr/lib64 and the paths in PATH.
You already know the location of your library. Add it to the CMAKE_PREFIX_PATH when you call CMake, then CMake will look for your library in the passed paths, too.
Sometimes you need to add hints or path suffixes, see the documentation for details:
https://cmake.org/cmake/help/latest/command/find_library.html
2. Link the library
From 1. you have the full library name in FOO_LIB. You use this to link the library to your target GLBall as in
target_link_libraries(GLBall PRIVATE "${FOO_LIB}")
You should add PRIVATE, PUBLIC, or INTERFACE after the target, cf. the documentation:
https://cmake.org/cmake/help/latest/command/target_link_libraries.html
If you don't add one of these visibility specifiers, it will either behave like PRIVATE or PUBLIC, depending on the CMake version and the policies set.
3. Add includes (This step might be not mandatory.)
If you also want to include header files, use find_path similar to find_library and search for a header file. Then add the include directory with target_include_directories similar to target_link_libraries.
Documentation:
https://cmake.org/cmake/help/latest/command/find_path.html
and
https://cmake.org/cmake/help/latest/command/target_include_directories.html
If available for the external software, you can replace find_library and find_path by find_package.
Let's say you have an executable like:
add_executable(GLBall GLBall.cpp)
If the external library has headers, give the path to its include folder:
target_include_directories(GLBall PUBLIC "/path/to/include")
Add the library directory path:
target_link_directories(GLBall PUBLIC "/path/to/lib/directory")
Finally, link the library name
target_link_libraries(GLBall mylib)
Note that the prefix and extension of the library file are removed:
libmylib.a ➜ mylib
mylib.so ➜ mylib
One more alternative, in the case you are working with the Appstore, need "Entitlements" and as such need to link with an Apple-Framework.
For Entitlements to work (e.g. GameCenter) you need to have a "Link Binary with Libraries"-buildstep and then link with "GameKit.framework". CMake "injects" the libraries on a "low level" into the commandline, hence Xcode doesn't really know about it, and as such you will not get GameKit enabled in the Capabilities screen.
One way to use CMake and have a "Link with Binaries"-buildstep is to generate the xcodeproj with CMake, and then use 'sed' to 'search & replace' and add the GameKit in the way XCode likes it...
The script looks like this (for Xcode 6.3.1).
s#\/\* Begin PBXBuildFile section \*\/#\/\* Begin PBXBuildFile section \*\/\
26B12AA11C10544700A9A2BA \/\* GameKit.framework in Frameworks \*\/ = {isa = PBXBuildFile; fileRef = 26B12AA01C10544700A9A2BA \/\* GameKit.framework xxx\*\/; };#g
s#\/\* Begin PBXFileReference section \*\/#\/\* Begin PBXFileReference section \*\/\
26B12AA01C10544700A9A2BA \/\* GameKit.framework xxx\*\/ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = GameKit.framework; path = System\/Library\/Frameworks\/GameKit.framework; sourceTree = SDKROOT; };#g
s#\/\* End PBXFileReference section \*\/#\/\* End PBXFileReference section \*\/\
\
\/\* Begin PBXFrameworksBuildPhase section \*\/\
26B12A9F1C10543B00A9A2BA \/\* Frameworks \*\/ = {\
isa = PBXFrameworksBuildPhase;\
buildActionMask = 2147483647;\
files = (\
26B12AA11C10544700A9A2BA \/\* GameKit.framework in Frameworks xxx\*\/,\
);\
runOnlyForDeploymentPostprocessing = 0;\
};\
\/\* End PBXFrameworksBuildPhase section \*\/\
#g
s#\/\* CMake PostBuild Rules \*\/,#\/\* CMake PostBuild Rules \*\/,\
26B12A9F1C10543B00A9A2BA \/\* Frameworks xxx\*\/,#g
s#\/\* Products \*\/,#\/\* Products \*\/,\
26B12AA01C10544700A9A2BA \/\* GameKit.framework xxx\*\/,#g
save this to "gamecenter.sed" and then "apply" it like this ( it changes your xcodeproj! )
sed -i.pbxprojbak -f gamecenter.sed myproject.xcodeproj/project.pbxproj
You might have to change the script-commands to fit your need.
Warning: it's likely to break with different Xcode-version as the project-format could change, the (hardcoded) unique number might not really by unique - and generally the solutions by other people are better - so unless you need to Support the Appstore + Entitlements (and automated builds), don't do this.
This is a CMake bug, see http://cmake.org/Bug/view.php?id=14185 and http://gitlab.kitware.com/cmake/cmake/issues/14185
It has been a long time since the question was posted but I am leaving this one just for reference.
I have a blog post describing step-by-step almost what you (or anyone else) were trying to do.
Please check here: https://michae9.wordpress.com/2022/09/01/shared-lib-to-be-used-by-client-programs-with-cmake/

Create directory structure in /var/lib using autotools and automake

I'm using autotools on a C project that, after installation, needs a particular directory structure in /var/lib as follows:
/var/lib/my-project/
data/
configurations/
local/
extra/
inputs/
I'm currently using the directive AS_MKDIR_P in configure.ac like so:
AS_MKDIR_P(/var/lib/my-project/data)
AS_MKDIR_P(/var/lib/my-project/configurations/local)
AS_MKDIR_P(/var/lib/my-project/configurations/extra)
AS_MKDIR_P(/var/lib/my-project/inputs)
But it needs the configure script to be run with root permissions which I don't think is the way to go. I think the instructions to create this directory structure needs to be in Makefile.am, so that make install creates them rather than configure, but I have no idea how to do that.
You really, really, really do not want to specify /var/lib/my-project. As the project maintainer, you have the right to specify relative paths, but the user may change DESTDIR or prefix. If you ignore DESTDIR and prefix and just install your files in /var/lib without regard for the user's requests, then your package is broken. It is not just slightly damaged, it is completely unusable. The autotool packaging must not specify absolute paths; that is for downsteam packagers (ie, those that build *.rpm or *.deb or *.dmg or ...). All you need to do is add something like this to Makefile.am:
configdir = $(pkgdatadir)/configurations
localdir = $(configdir)/local
extradir = $(configdir)/extra
inputdir = $(pkgdatadir)/inputs
mydatadir = $(pkgdatadir)/data
config_DATA = cfg.txt
local_DATA = local.txt
extra_DATA = extra.txt
input_DATA = input.txt
mydata_DATA = data.txt
This will put input.txt in $(DESTDIR)$(pkgdatadir)/inputs, etc. If you want that final path to be /var/lib/my-project, then you can specify datadir appropriately at configure time. For example:
$ CONFIG_SITE= ./configure --datadir=/var/lib > /dev/null
This will assign /var/lib to datadir, so that pkgdatadir will be /var/lib/my-project and a subsequent make install DESTDIR=/path/to/foo will put the files in /path/to/foo/var/lib/my-package/. It is essential that your auto-tooled package honor things like prefix (which for these files was essentially overridden here by the explicit assignment of datadir) and DESTDIR. The appropriate time to specify paths like /var/lib is when you run the configure script. For example, you can add the options to the configure script in your rpm spec file or in debian/rules, or in whatever file your package system uses. The auto-tools provide a very flexible packaging system which can be easily used by many different packaging systems (unfortunately, the word "package" is highly overloaded!). Embrace that flexibility.
According to autotools documentation (here and here), there are hooks that you can specify in Makefile.am that will run at specific times during the installation. For my needs I will use install-exec-hook (or install-data-hook) which will be run after all executables (or data) have been installed:
install-exec-hook:
$(MKDIR_P) /var/lib/my-project/data
$(MKDIR_P) /var/lib/my-project/configurations/local
$(MKDIR_P) /var/lib/my-project/configurations/extra
$(MKDIR_P) /var/lib/my-project/inputs
MKDIR_P is a variable containing the command mkdir -p, or an equivalent to it if the system doesn't have mkdir. To make it available in Makefile.am you have to use the macro AC_PROG_MKDIR_P in configure.ac.

How to use autotools for deep projects?

I have a C project that has the following structure
Main/
Makefile.am
bin/
src/
Makefile.am
main.c
SomeLibrarySource/
SomeFuncs.c
SomeFuncs.h
The main.c contains the main function that uses functions defined in the SomeFuncs.{h/c} files.
I want to use autotools for this project. I read a couple of resources on autotools. But, I was only able to manage using autotools for a single level project where all source, object and other files reside in the same directory.
Then I got some links that talked about using autotools for deep projects like this one and then I got confused.
Right now I have two Makefile.am as follows
Makefile.am
SUBDIRS=src
src/Makefile.am
mainprgdir=../
mainprg_PROGRAMS=main
main_SOURCES=main.c
I am pretty sure that these files should not be as I have them now :P
How do I use autotools for the above project structure? (At least what should be there in those Makefile.am(s) and where should I place them.
EDIT:
One more thing! At the end I would like to have the object files created in the bin directory.
Thanks
mainprogdir=../ does not make a whole lot of sense (you don't know what it is relative to on installation). Probably intended:
# Main/Makefile.am
# .━━ target for `make install`
# |
# ↓ ↓━━ target for compilation
bin_PROGRAMS = bin/main
# ↓━━ based upon compilation target name
bin_main_SOURCES = src/main.c
There are two main approaches. If the functions in SomeLibrarySource are used only by main, then there's no need to build a separate library and you can simply specify the source files in src/Makefile.am
main_SOURCES = main.c SomeLibrarySource/SomeFuncs.c
However, if you actually want to use the functions in other code in your tree, you do not want to compile SomeFuncs.c multiple times but should use a convenience library.
# Assigning main_SOURCES is redundant
main_SOURCES = main.c
main_LDADD = SomeLibrarySource/libSomeFuncs.a
noinst_LIBRARIES = SomeLibrarySource/libSomeFuncs.a
AM_CPPFLAGS = -I$(srcdir)/SomeLibrarySource
(You'll need AC_PROG_RANLIB in configure.ac to use convenience libraries.)
If the source file is named SomeFuncs.c, automake will not need Makefile.am to specify SomeLibrarySource_libSomeFuncs_a_SOURCES, but if the name of the source code file does not match the name specified in noinst_LIBRARIES, SomeLibrarySource_libSomeFuncs_a_SOURCES should be set to the list of files used to build the library. Note that you do not need to specify main_SOURCES, since main.c is the default value if left unspecified (but it's not a bad idea to be explicit.) (In all of this, I am not comfortable use CamlCase names, but the system I'm using uses a case insensitive file system (biggest mistake apple ever made) and the examples I give here are working for me. YMMV)
You could of course do a recursive make, or build the library as a separate project and install it. (I like the final option. Libraries with useful features should exist on their own.)

Crossprofiling with gcov, but GCOV_PREFIX and GCOV_PREFIX_STRIP is ignored

I want to use GCOV to make code coverage but the tests will run on another machine. So the hard wired path to .gcda files in the executable won't work.
In order to change this default directory I can use the GCOV_PREFIX and GCOV_PREFIX_STRIP env vars, as it's said here.
Here my commands I used:
$ export GCOV_PREFIX="/foo/bar"
$ export GCOV_PREFIX_STRIP="3"
$ gcc main.c -fprofile-arcs -ftest-coverage
$ strings a.out | grep gcda
/home/calmarius/blahblah/main.c.gcda
The path remains the same.
Anyone have experience with this?
The environment variables are taken into account when you run the code.
Set them to the appropriate values on the target machine before you run your tests, and the .gcda files will be generated where you want them.
************ ARRRRGGGGGHHHHH ************
Please, please vote for Mat's answer.
The environment variables are taken into account when you run the
code.
This one sentence is apparently missing from EVERY document I have read regarding how to relocate the output !
In fact , allow me to expand that answer just a bit.
GCOV_PREFIX is a runtime - as apposed to build time - environment variable and determines the root directory where the gcov output files (*.gcda) are written.
GCOV_PREFIX_STRIP=X is also a runtime variable, and has the effect of stripping X elements from the path found in the object files (strings XXXX.o)
What this means is:
When you build your project, the object files are written with the full path to the location of each source file responsible for each object file embedded within them.
So, imagine you are writing an executable MyApp and a library MyLib in a directory stricture like this:
/MyProject
|-MyApp
|--MyLib
Notice MyLib is a subdirectory of MyApp
Let's say MyApp has 2 source files, and MyLib has 3
After building with the "-coverage" flag, you will have generated
5 .gcno files, 1 for each object file.
Embedded in the .o files for MyApp will be the absolute path **/MyProject/MyApp/**a_source_file.cpp Similarly, embedded in the .o files for MyLib will be the path **/MyProject/MyApp/MyLib/**another_source_file.cpp
Now, let's say you're like me, and move those files onto a completely different machine with a different directory structure from where they got built. In my case the target machine is actually a totally different architecture. I deploy to /some/deploy/path not /MyProject on that machine.
If you simply run the app, gcov data will try to write corresponding .gcda files to /MyProject/MyApp and /MyProject/MyApp/MyLib for each object file in your project, because that's the path indicated by the .o files, and after all, MyApp, and MyLib are simply collections of .o files archived together, with some other magic to fix up funcitons pointers and stuff.
Chances are, those directories don't exist, and you probably aren't running as root (are you?), so those directories won't be created either. Soooo.. you won't see any gcda files within the deploy location /my/deploy/path.
That's totally confusing, right !?!??!?!?!?
Here's where GCOV_PREFIX and GCOV_PREFIX_STRIP come in.
(BAM ! fist hits forehead)
You need to instruct the ****runtime**** that the embedded path in the .o files isn't really what you want. You want to "strip" some of the path off, and replace it with the deploy directory.
So, you set the deploy directory via GCOV_PREFIX=/some/deploy/path and you want to strip the /MyProject from the generated .gcda paths so you set GCOV_PREFIX_STRIP=1
With these two environment variables set, you run your app and then look in
/some/deploy/path/MyApp and /some/deploy/path/MyApp/MyLib and lo and behold, the 5 gcda files miraculously appear, one for each object file.
Note: the problem is compounded if you do out of source builds. The .o points to the source, but the gcda will be written relative to the build directory.

How do I use waf to build a shared library?

I want to build a shared library using waf as it looks much easier and less cluttered than GNU autotools.
I actually have several questions so far related to the wscript I've started to write:
VERSION='0.0.1'
APPNAME='libmylib'
srcdir = '.'
blddir = 'build'
def set_options(opt):
opt.tool_options('compiler_cc')
pass
def configure(conf):
conf.check_tool('compiler_cc')
conf.env.append_value('CCFLAGS', '-std=gnu99 -Wall -pedantic -ggdb')
def build(bld):
bld.new_task_gen(
features = 'cc cshlib',
source = '*.c',
target='libmylib')
The line containing source = '*.c' does not work. Must I specify each and every .c file instead of using a wildcard?
How can I enable a debug build for example (currently the wscript is using the debug builds CFLAGS, but I want to make this optional for the end user).
It is planned for the library sources to be within a sub directory, and programs that use the lib each in their own sub directories.
Assuming you are using the latest version of waf (1.5.9 at the time of writing), wild cards can be specified via the glob() method on the build context. So you can write the following:
bld.new_task_gen(
features = 'cc cshlib',
source = bld.glob('*.c'),
target='mylib')
If you were using an older version of waf that doesn't have glob, then there is a method find_sources_in_dirs that you can use:
lib = bld.new_task_gen(
features = 'cc cshlib',
target = 'mylib')
lib.find_sources_in_dirs('.')
This method is still in Waf but is slated for deprecation and may eventually disappear.
The srcdir and blddir variables are optional now so you don't need them - they default to "." and "build" anyway. You shouldn't prepend "lib" to the target name, this is done automatically in a platform specific way (on Windows no lib is added and shared libraries use .dll). Debug vs Release build is a surprisingly thorny issue. Originally Waf included this feature, but it was dropped at some point and never re-added. It's a common request on the mailing list so may resurface in the future. Meanwhile you could do a lot worse than use gjc's cflags module. Just add it to your project directory. The final wscript would then be:
VERSION='0.0.1'
APPNAME='mylib'
def set_options(opt):
opt.tool_options('compiler_cc')
opt.tool_options('cflags', tooldir='.')
def configure(conf):
conf.check_tool('compiler_cc')
conf.check_tool('cflags', tooldir='.')
def build(bld):
bld.new_task_gen(
features = 'cc cshlib',
source = bld.glob('*.c'),
target=APPNAME)
And to set up a debug build you would run the following:
./waf configure -d debug
If you are using libraries in their own sub-directories, then you should probably have a top level wscript and use the bld.add_subdirs() technique to add library/program directories. Each sub-directory would have its own wscript_build file. You can then use the export_incdirs and uselib_local properties to specify the correct include directories between library and program "modules".
waf has changed a lot over the years so neither the code in the question nor the answer works with current waf anymore. Nowadays you just write:
def options(ctx):
ctx.load('compiler_c')
def configure(ctx):
ctx.load('compiler_c')
def build(ctx):
ctx.shlib(source = ctx.path.ant_glob('src/*.c'),
target = 'name')
Note that waf will automatically add the lib prefix so you don't write target
= 'libname'. On Windows, you also have to add the defs keyword argument to the shlib function call.
Personally I would recommend against recursive build scripts involving wscript_build files. Not that it doesn't work (like recursive makefiles), it's just much simpler to keep all logic in one medium-sized build script.

Resources