Create directory structure in /var/lib using autotools and automake - c

I'm using autotools on a C project that, after installation, needs a particular directory structure in /var/lib as follows:
/var/lib/my-project/
data/
configurations/
local/
extra/
inputs/
I'm currently using the directive AS_MKDIR_P in configure.ac like so:
AS_MKDIR_P(/var/lib/my-project/data)
AS_MKDIR_P(/var/lib/my-project/configurations/local)
AS_MKDIR_P(/var/lib/my-project/configurations/extra)
AS_MKDIR_P(/var/lib/my-project/inputs)
But it needs the configure script to be run with root permissions which I don't think is the way to go. I think the instructions to create this directory structure needs to be in Makefile.am, so that make install creates them rather than configure, but I have no idea how to do that.

You really, really, really do not want to specify /var/lib/my-project. As the project maintainer, you have the right to specify relative paths, but the user may change DESTDIR or prefix. If you ignore DESTDIR and prefix and just install your files in /var/lib without regard for the user's requests, then your package is broken. It is not just slightly damaged, it is completely unusable. The autotool packaging must not specify absolute paths; that is for downsteam packagers (ie, those that build *.rpm or *.deb or *.dmg or ...). All you need to do is add something like this to Makefile.am:
configdir = $(pkgdatadir)/configurations
localdir = $(configdir)/local
extradir = $(configdir)/extra
inputdir = $(pkgdatadir)/inputs
mydatadir = $(pkgdatadir)/data
config_DATA = cfg.txt
local_DATA = local.txt
extra_DATA = extra.txt
input_DATA = input.txt
mydata_DATA = data.txt
This will put input.txt in $(DESTDIR)$(pkgdatadir)/inputs, etc. If you want that final path to be /var/lib/my-project, then you can specify datadir appropriately at configure time. For example:
$ CONFIG_SITE= ./configure --datadir=/var/lib > /dev/null
This will assign /var/lib to datadir, so that pkgdatadir will be /var/lib/my-project and a subsequent make install DESTDIR=/path/to/foo will put the files in /path/to/foo/var/lib/my-package/. It is essential that your auto-tooled package honor things like prefix (which for these files was essentially overridden here by the explicit assignment of datadir) and DESTDIR. The appropriate time to specify paths like /var/lib is when you run the configure script. For example, you can add the options to the configure script in your rpm spec file or in debian/rules, or in whatever file your package system uses. The auto-tools provide a very flexible packaging system which can be easily used by many different packaging systems (unfortunately, the word "package" is highly overloaded!). Embrace that flexibility.

According to autotools documentation (here and here), there are hooks that you can specify in Makefile.am that will run at specific times during the installation. For my needs I will use install-exec-hook (or install-data-hook) which will be run after all executables (or data) have been installed:
install-exec-hook:
$(MKDIR_P) /var/lib/my-project/data
$(MKDIR_P) /var/lib/my-project/configurations/local
$(MKDIR_P) /var/lib/my-project/configurations/extra
$(MKDIR_P) /var/lib/my-project/inputs
MKDIR_P is a variable containing the command mkdir -p, or an equivalent to it if the system doesn't have mkdir. To make it available in Makefile.am you have to use the macro AC_PROG_MKDIR_P in configure.ac.

Related

Using pg_config with waf

I use waf as my build system and I want to compile a small C program using Postgres. I have included postgres.h in my program so I need to find the path to it in my wscript file. I know that I can get the path I need by running:
pg_config --includedir-server
which gives me:
/usr/include/postgresql/9.3/server
So I thought I could use something like this:
cfg.check_cfg(
path='pg_config',
package='',
uselib_store='PG',
args='--includedir-server',
)
And then build my program by:
bld.program(
source=['testpg.c'],
target='testpg',
includes=['.', '../src'],
use=['PQ', 'PG'],
)
But this fails with postgres.h: No such file or directory. I ran ./waf -v and confirmed that the proper -I flag is not being passed to gcc. My guess is this happens because pg_config does not add a -I prefix to the path it returns. Is there a way I can make waf to add the prefix, or make pg_config to add it?
Should pg_config had the standard output of pkg_config like programs (ie outputs something like -Ixxx -Iyyy), your code would work, has check_cfg parse this kind of output.
As there is no complicated parsing, you can go for:
import subprocess
includes = subprocess.check_output(["pg_config", "--includedir-server"])
includes.replace("\n", "")
conf.env.INCLUDES_PG = [includes]
And then use it:
bld.program(
source=['testpg.c'],
target='testpg',
use=['PG'],
)
See the library integration in the waf book. It explains the naming rule that make it works.
You can write a small plugin to ease the use :)

Building a Shared Library, Updating Header Files to Compiler/System Directories

A friend and I are using Qt Creator with Boost to build a game engine. So far we have this idea that the engine is going to be a shared library, with the idea that we can run it with a test executable which will turn into the game we eventually want to make.
The problem is header files, mainly. I'd like to find some way for Qt Creator to be able to recognize the header files as soon as the latest builds of the engine have been built or even when they're added. At first I was thinking a script in Python which executed as a build step in Qt Creator after the engine had been built, would simply copy the header files to a system directory (/usr/include, for example - if operating on a *nix system), where the IDE would then recognize the header files when linking the engine with the test executable, and we'd also have full auto completion support.
Of course, environmental variables would be used, and while I prefer developing in Linux, my friend prefers Windows, so we agreed to take care of development in regards to our respective platform preferences.
While this seems like a good solution, I think this Python script idea may be over kill. Is there a better way to do this?
Update
From to the suggested Qmake script, I end up getting this error.
cp -f "/home/amsterdam/Programming/atlas/Engine/AtlasEngine/"AtlasEngine_global.h "/"
cp: cannot create regular file `/AtlasEngine_global.h': Permission denied
make: Leaving directory `/home/amsterdam/Programming/atlas/Engine/AtlasEngine__GCC__Linux__Debug'
make: *** [libAtlasEngine.so.1.0.0] Error 1
15:20:52: The process "/usr/bin/make" exited with code 2.
Error while building project AtlasEngine (target: Desktop)
When executing build step 'Make'
My adjustments look as follows:
# Copy over build artifacts
SRCDIR = $$ATLAS_PROJ_ROOT
DESTDIR = $$ATLAS_INCLUDE
# Look for header files there too
INCLUDEPATH += $$SRCDIR
# Dependencies: mylib. Only specify the libs you depend on.
# Leave out for building a shared library without dependencies.
#win32:LIBS += $$quote($$SRCDIR/mylib.dll)
# unix:LIBS += $$quote(-L$$SRCDIR) -lmylib
DDIR = \"$$SRCDIR/\" #<--DEFAULTS
SDIR = \"$$IN_PWD/\"
# Replace slashes in paths with backslashes for Windows
win32:file ~= s,/,\\,g
win32:DDIR ~= s,/,\\,g
win32:SDIR ~= s,/,\\,g
for(file, HEADERS) {
QMAKE_POST_LINK += $$QMAKE_COPY $$quote($${SDIR}$${file}) $$quote($$DDIR) $$escape_expand(\\n\\t)
}
I have managed to overcome this using some Qmake magic that works cross-platform. It copies over the shared libraries (either .dll or .so files) along with the header files to a directory in a directory dll at a level next to your current project.
Put this in the end of your .pro files and change the paths/libs accordingly.
# Copy over build artifacts
MYDLLDIR = $$IN_PWD/../dlls
DESTDIR = \"$$MYDLLDIR\"
# Look for header files there too
INCLUDEPATH += $$MYDLLDIR
# Dependencies: mylib. Only specify the libs you depend on.
# Leave out for building a shared library without dependencies.
win32:LIBS += $$quote($$MYDLLDIR/mylib.dll)
unix:LIBS += $$quote(-L$$MYDLLDIR) -lmylib
DDIR = \"$$MYDLLDIR/\"
SDIR = \"$$IN_PWD/\"
# Replace slashes in paths with backslashes for Windows
win32:file ~= s,/,\\,g
win32:DDIR ~= s,/,\\,g
win32:SDIR ~= s,/,\\,g
for(file, HEADERS) {
QMAKE_POST_LINK += $$QMAKE_COPY $$quote($${SDIR}$${file}) $$quote($$DDIR) $$escape_expand(\\n\\t)
}
Then adjust the LD_LIBRARY_PATH in the 'Run settings' of your project to point to that same dll directory (relatively).
Yes, it's ugly with escaping for paths with spaces and backslashes, but I found this to be working well cross-platform. Windows (XP, 7) and Linux tested. And yes it includes environment settings to be changed for running your project, but at least you don't need external (Python) scripts anymore or to install it to system directory requiring root privileges.
Improvements are welcome.
I'm not sure if anyone else would be having issues with this, but for whatever reason Qmake wasn't able to access my user specified environment variables properly.
So, since this was the case, one solution I came up with was to add the variables as Qmake configuration variable.
If you're in a UNIX based system, the first thing you're going to want to do is append the location of qmake - which should lie in your QtSDK folder - to your system $PATH, like so:
export PATH=$PATH:/path/to/QtSDK/...../qmake_root
From there, you can do something along the lines of:
qmake -set "VARIABLE" "VALUE"
In this case, I simply did:
qmake -set "ATLAS_PROJ_ROOT" $ATLAS_PROJ_ROOT.
And then I accessed it in my Qmake project file (.pro) with:
VAR = $$[ATLAS_PROJ_ROOT]
More info can be found here.

How to use autotools for deep projects?

I have a C project that has the following structure
Main/
Makefile.am
bin/
src/
Makefile.am
main.c
SomeLibrarySource/
SomeFuncs.c
SomeFuncs.h
The main.c contains the main function that uses functions defined in the SomeFuncs.{h/c} files.
I want to use autotools for this project. I read a couple of resources on autotools. But, I was only able to manage using autotools for a single level project where all source, object and other files reside in the same directory.
Then I got some links that talked about using autotools for deep projects like this one and then I got confused.
Right now I have two Makefile.am as follows
Makefile.am
SUBDIRS=src
src/Makefile.am
mainprgdir=../
mainprg_PROGRAMS=main
main_SOURCES=main.c
I am pretty sure that these files should not be as I have them now :P
How do I use autotools for the above project structure? (At least what should be there in those Makefile.am(s) and where should I place them.
EDIT:
One more thing! At the end I would like to have the object files created in the bin directory.
Thanks
mainprogdir=../ does not make a whole lot of sense (you don't know what it is relative to on installation). Probably intended:
# Main/Makefile.am
# .━━ target for `make install`
# |
# ↓ ↓━━ target for compilation
bin_PROGRAMS = bin/main
# ↓━━ based upon compilation target name
bin_main_SOURCES = src/main.c
There are two main approaches. If the functions in SomeLibrarySource are used only by main, then there's no need to build a separate library and you can simply specify the source files in src/Makefile.am
main_SOURCES = main.c SomeLibrarySource/SomeFuncs.c
However, if you actually want to use the functions in other code in your tree, you do not want to compile SomeFuncs.c multiple times but should use a convenience library.
# Assigning main_SOURCES is redundant
main_SOURCES = main.c
main_LDADD = SomeLibrarySource/libSomeFuncs.a
noinst_LIBRARIES = SomeLibrarySource/libSomeFuncs.a
AM_CPPFLAGS = -I$(srcdir)/SomeLibrarySource
(You'll need AC_PROG_RANLIB in configure.ac to use convenience libraries.)
If the source file is named SomeFuncs.c, automake will not need Makefile.am to specify SomeLibrarySource_libSomeFuncs_a_SOURCES, but if the name of the source code file does not match the name specified in noinst_LIBRARIES, SomeLibrarySource_libSomeFuncs_a_SOURCES should be set to the list of files used to build the library. Note that you do not need to specify main_SOURCES, since main.c is the default value if left unspecified (but it's not a bad idea to be explicit.) (In all of this, I am not comfortable use CamlCase names, but the system I'm using uses a case insensitive file system (biggest mistake apple ever made) and the examples I give here are working for me. YMMV)
You could of course do a recursive make, or build the library as a separate project and install it. (I like the final option. Libraries with useful features should exist on their own.)

Locating data files in C program built with Autotools

I have a C program built using Autotools. In src/Makefile.am, I define a macro with the path to installed data files:
AM_CPPFLAGS = -DAM_INSTALLDIR='"$(pkgdatadir)"'
The problem is that I need to run make install before I can test the binary (since it needs to be able to find the data files).
I can define another macro with the path of the source tree so the data files can be located without installing:
AM_CPPFLAGS = -DAM_INSTALLDIR='"$(pkgdatadir)"' -DAM_TOPDIR='"$(abs_top_srcdir)"'
Now, I would like the following behavior:
If the binary was installed via make install, use AM_INSTALLDIR to fetch data files.
If the binary was not installed, use AM_TOPDIR to fetch data files.
Is this possible? Is there a better approach to this problem?
What I do (in https://http://rhdunn.github.com/cainteoir/) is:
const char *basedir = getenv("CAINTEOIR_DATADIR");
if (!basedir)
basedir = DATADIR "/" PACKAGE; // e.g. /usr/share/cainteoir-engine
and then run it (in tests/harness.py) as:
CAINTEOIR_DATADIR=`pwd`/data src/apps/metadata/metadata test_file.epub
This then allows the user to change the location of where to get the data if they wish.
Making the program able to use a run-time configuration as proposed by reece is a good solution. If for some reason you do not want it to be configurable at run-time, a common solution is to build a test binary differently than the installed binary (there are other problems associated with this, in particular ensuring that the program you are testing has behavior that is consistent with the program that is installed.) An easy way to do that is something like:
bin_PROGRAMS = foo
check_PROGRAMS = test-foo
test_foo_SOURCES = $(foo_SOURCES)
AM_CPPFLAGS = -DINSTALLDIR='"$(pkgdatadir)"'
test_foo_CPPFLAGS = -DINSTALLDIR='"$(abs_top_srcdir)"'
Rather than using a binary with a different name, you might want to have a dedicated tests directory and build the program using the same name as the original.
Note that I've changed the name from AM_INSTALLDIR to INSTALLDIR. Automake reserves names
beginning with "AM_" for its own use, and by using that name you are stomping on Automake's
namespace.
A bit of additional information first: The data files are under active development, and I have various scripts that need to call binaries using local data files, whereas installed binaries should use stable, installed data files.
My original solution made use of an environment variable, as proposed by reece. But I didn't want to manage setting up environment variables in various places, and I didn't want any risk of the wrong data files being picked up due to a mistake.
So the solution I ended up with was to define macros for both locations at build time, and add a flag (-local) to the binaries to force local data files to be used.

How do I use waf to build a shared library?

I want to build a shared library using waf as it looks much easier and less cluttered than GNU autotools.
I actually have several questions so far related to the wscript I've started to write:
VERSION='0.0.1'
APPNAME='libmylib'
srcdir = '.'
blddir = 'build'
def set_options(opt):
opt.tool_options('compiler_cc')
pass
def configure(conf):
conf.check_tool('compiler_cc')
conf.env.append_value('CCFLAGS', '-std=gnu99 -Wall -pedantic -ggdb')
def build(bld):
bld.new_task_gen(
features = 'cc cshlib',
source = '*.c',
target='libmylib')
The line containing source = '*.c' does not work. Must I specify each and every .c file instead of using a wildcard?
How can I enable a debug build for example (currently the wscript is using the debug builds CFLAGS, but I want to make this optional for the end user).
It is planned for the library sources to be within a sub directory, and programs that use the lib each in their own sub directories.
Assuming you are using the latest version of waf (1.5.9 at the time of writing), wild cards can be specified via the glob() method on the build context. So you can write the following:
bld.new_task_gen(
features = 'cc cshlib',
source = bld.glob('*.c'),
target='mylib')
If you were using an older version of waf that doesn't have glob, then there is a method find_sources_in_dirs that you can use:
lib = bld.new_task_gen(
features = 'cc cshlib',
target = 'mylib')
lib.find_sources_in_dirs('.')
This method is still in Waf but is slated for deprecation and may eventually disappear.
The srcdir and blddir variables are optional now so you don't need them - they default to "." and "build" anyway. You shouldn't prepend "lib" to the target name, this is done automatically in a platform specific way (on Windows no lib is added and shared libraries use .dll). Debug vs Release build is a surprisingly thorny issue. Originally Waf included this feature, but it was dropped at some point and never re-added. It's a common request on the mailing list so may resurface in the future. Meanwhile you could do a lot worse than use gjc's cflags module. Just add it to your project directory. The final wscript would then be:
VERSION='0.0.1'
APPNAME='mylib'
def set_options(opt):
opt.tool_options('compiler_cc')
opt.tool_options('cflags', tooldir='.')
def configure(conf):
conf.check_tool('compiler_cc')
conf.check_tool('cflags', tooldir='.')
def build(bld):
bld.new_task_gen(
features = 'cc cshlib',
source = bld.glob('*.c'),
target=APPNAME)
And to set up a debug build you would run the following:
./waf configure -d debug
If you are using libraries in their own sub-directories, then you should probably have a top level wscript and use the bld.add_subdirs() technique to add library/program directories. Each sub-directory would have its own wscript_build file. You can then use the export_incdirs and uselib_local properties to specify the correct include directories between library and program "modules".
waf has changed a lot over the years so neither the code in the question nor the answer works with current waf anymore. Nowadays you just write:
def options(ctx):
ctx.load('compiler_c')
def configure(ctx):
ctx.load('compiler_c')
def build(ctx):
ctx.shlib(source = ctx.path.ant_glob('src/*.c'),
target = 'name')
Note that waf will automatically add the lib prefix so you don't write target
= 'libname'. On Windows, you also have to add the defs keyword argument to the shlib function call.
Personally I would recommend against recursive build scripts involving wscript_build files. Not that it doesn't work (like recursive makefiles), it's just much simpler to keep all logic in one medium-sized build script.

Resources