Hello everybody out there using GNU autoconf,
What is the best practice to look for libftdi and including it with autoconf for compiling a C program using it?
The following snippet from a configure.ac file works, but I'm not sure, whether it is best practice:
PKG_CHECK_MODULES([LIBFTDI], [libftdi])
#AC_CHECK_LIB([ftdi],[ftdi],[ftdi]) # Why doesn't this work?
#AC_SEARCH_LIBS([ftdi],[ftdi],[ftdi]) # Why doesn't this work?
#AC_CHECK_HEADERS([ftdi.h],[],[echo "error: missing libftdi header files" && exit 1])
LIBS="-lftdi $LIBS $LDFLAGS" # works, but is this the best way?
I'm building the program with autoconf (GNU Autoconf) 2.69 and compiling it with gcc version 7.5.0 on Ubuntu 18.04.
Why your other attempts failed
Library tests
Your commented-out AC_CHECK_LIB and AC_SEARCH_LIBS examples do not demonstrate correct usage. Usage details are presented in the manual, but to sum up:
the arguments to AC_CHECK_LIB are
The simple name of the library, i.e. ftdi
The name of a characteristic function provided by the library
(optional) Code for configure to execute in the event that the library is found. Default is to prepend a link option to $LIBS and define a HAVE_LIB* preprocessor macro.
(optional) Code for configure to execute in the event that the library is not found
(optional) Additional library link options (not already in $LIBS) that are needed to link a program that uses the library being checked
the arguments to AC_SEARCH_LIBS are
The name of the function to search for
A list of one or more library names to search
(optional) Code for configure to execute in the event that the library is found, in addition to prepending a link option to $LIBS (but not defining any preprocessor macro)
(optional) Code for configure to execute in the event that the library is not found
(optional) Additional library link options (not already in $LIBS that are needed to link a program that uses the library being checked
Neither your AC_CHECK_LIB example nor your AC_SEARCH_LIBS example properly designates an existing libftdi function to check for. Moreover, the third argument in each case is unlikely to be valid shell / Autoconf code, so in the event that the library were found, configure would probably crash. Better might be:
AC_CHECK_LIB([ftdi], [ftdi_init])
or
AC_SEARCH_LIBS([ftdi_init], [ftdi])
Depending on what exactly you want to do, on details of libftdi, and on the configure.ac context, you might need to provide appropriate values for some or all of the optional arguments.
The main reasons for a library check to fail despite the library in fact being installed are
the library being installed in a location that is not in the default search path
the library having link dependencies on other libraries, and those have not (yet) been accounted for at the time of the check
The former is analogous to header installation location considerations discussed in the next section. The latter can be addressed by adding explicit extra link flags via the fifth argument to AC_CHECK_LIB or AC_SEARCH_LIBS, but is more often addressed semi-automatically by performing AC_CHECK_LIB or AC_SEARCH_LIBS tests in reverse prerequisite order, so that the value of LIBS is built up with an appropriately-ordered list of link flags, ready at each point to support the next check, and ultimately appropriate for supporting the overall compilation.
Note also that libftdi provides both C and C++ interfaces. In ftdi_init, I have been careful to choose a function that has C linkage, so as to avoid C++ name-mangling issues (see How to test a C++ library usability in configure.in?). You may also need to ensure that the tests are run with the C compiler (see Language Choice in the Autoconf manual).
Header test
Your AC_CHECK_HEADERS usage, on the other hand, does not appear to be inherently wrong. If the resulting configure script does not detect ftdi.h, then that implies that the header isn't in the compiler's default header search path. That might happen, for example, if it is installed in a subdirectory, such as /usr/include/ftdi. This would be a matter of both ftdi and system installation convention.
If it is ftdi convention for the headers to be installed in a subdirectory, then your source files should specify that in their #include directives:
#include <ftdi/ftdi.h>
If your source files in fact do that, then that should also be what you tell Autoconf to look for:
AC_CHECK_HEADERS([ftdi/ftdi.h])
Regardless of whether a subdirectory prefix is expected or used, it is good practice to accommodate the possibility of headers and / or libraries being installed in a non-standard location. Although one can always do that by specifying appropriate flags in the CPPFLAGS variable in configure's environment, I prefer and recommend using AC_ARG_WITH to designate a --with argument or AC_ARG_VAR to designate an environment variable that configure will consult for the purpose. For example,
AC_ARG_WITH([ftdi-includedir],
[AS_HELP_STRING([--with-ftdiincludedir=dir],
[specifies a custom directory for the libftdi header files])],
[CPPFLAGS="$CPPFLAGS -I$withval"]
)
Exposing an argument or environment variable for the specific purpose highlights (in the output of ./configure --help) the fact that this is a knob that the user might need to adjust. Additionally, receiving the include directory via a for-purpose vector is sometimes useful for limiting in which compilations the designated include directory is made available.
On PKG_CHECK_MODULES
The Autotools objective and philosophy is to support the widest possible array of build machines and environments by minimizing external dependencies and writing the most portable configuration and build code possible. To this end, the Autotools are designed so that they themselves are not required to build projects on supported systems. Rather, Autoconf produces configure as a stand-alone, highly portable shell script, and Automake produces configurable templates for highly portable makefiles. These are intended to be included in source packages, to be used as-is on each build system. Making your configure script dependent on pkg-config being installed on every system where your project is to be built, as using PKG_CHECK_MODULES does, conflicts with those objectives.
How significant an issue that may be is a subject of some dispute. Where it is available, pkg-config can be very useful, especially for components that require complex build flags. PKG_CHECK_MODULES is thus very convenient for both package maintainer and package builder on those systems where it is present or readily available, for those components that provide pkg-config metadata.
But pkg-config is not necessarily available for every system targeted by your software. It cannot reasonably be assumed present or obtainable even on systems for which it is nominally available. And even on systems that have it, pkg-config metadata for the libraries of interest are not necessarily installed with the libraries.
As such, I urge you to avoid using PKG_CHECK_MODULES in your Autoconf projects. You need to know how to do without it in any case, because it is not an option for some libraries. Where appropriate, provide hooks by which the builder can supply appropriate flags, and let them choose whether to use pkg-config in conjunction with those. Decoupling configure from pkg-config in this way makes a bit more work for you, and in some cases for builders, but it is more flexible.
Your PKG_CHECK_MODULES example
Your example invocation appears ok in itself, supposing that "libftdi" is the appropriate pkg-config module name (you have to know the appropriate name):
PKG_CHECK_MODULES([LIBFTDI], [libftdi])
But although that may yield a configure script that runs successfully, it does not, in itself, do much for you. In particular, it verifies that pkg-config metadata for the named module is present, but
it does not verify the presence or test the use of the library or header
although it does set some output variables containing compile and link flags, you do not appear to be using those
specifically, if you're going to rely on pkg-config, then you should use the link flags it reports to you instead of hardcoding -lftdi, and that alone.
Furthermore, it is more typical to use the output variables created by PKG_CHECK_MODULES in your makefile than to use them to update $LIBS or other general variables inside configure. If you do use them in configure, however, then it is essential to understand that LIBS and LDFLAGS have different roles with little overlap. It is generally inappropriate, not to mention unnecessary, to include the LDFLAGS in LIBS. If you want to update LIBS inside configure, then this would be the way to do it:
LIBS="$LIBFTDI_LIBS $LIBS"
And if you're going to do that, then you probably should do the same with the compiler flags reported by pkg-config, if any:
CFLAGS="$CFLAGS $LIBFTDI_CFLAGS"
You can check it like this:
AC_CHECK_LIB([ftdi],[ftdi_init],[],[echo "error: missing libftdi library" && exit 1],[])
LDFLAGS="-lftdi $LDFLAGS"
The second argument for AC_CHECK_LIB is a function exported by the library, and in this case the init call works well.
If libftdi uses pkg-config (and it appears to, given you said that snippet works), the PKG_CHECK_MODULES is what you want. The default for action-if-not-found is to error out, so if this is a required dependency, it's exactly what you want.
But you shouldn't use LIBS that way. First because LDFLAGS does not have the same semantics as LIBS, second because the pkg-config file might have provided you with further search paths that are required.
Instead you should add to your Makefile.am the flags as you need them:
mytarget_CFLAGS = $(LIBFTDI_CFLAGS)
mytarget_LDADD = $(LIBFTDI_LIBS)
You can refer to my Autotools Mythbuster — Dependency Discovery for further details on how to use pkg-config for dependencies. You can see there how you would generally use AC_CHECK_LIB or AC_SEARCH_LIB, but seriously if pkg-config works, stick to that, as it's more reliable and consistent.
I have been reading up on make and looking at the Makefiles for popular C projects on GitHub to cement my understanding.
One thing I am struggling to understand is why none of the examples I've looked at (e.g. lz4, linux and FFmpeg) seem to account for header file dependencies.
For my own project, I have header files that contain:
Numeric and string constants
Macros
Short, inline functions
It would seem essential, therefore, to take any changes to these into account when determining whether to recompile.
I have discovered that gcc can automatically generate Makefile fragments from dependencies as in this SO answer but I haven't seen this used in any of the projects I've looked at.
Can you help me understand why these projects apparently ignore header file dependencies?
I'll attempt to answer.
The source distros of some projects include a configure script which creates a makefile from a template/whatever.
So the end user which needs to recompile the package for his/her target just has to do:
$ configure --try-options-until-it-works
$ make
Things can go wrong during the configure phase, but this has nothing to do with the makefile itself. User has to download stuff, adjust paths or configure switches and run again until makefile is successfully generated.
But once the makefile is generated, things should go pretty smooth from there for the user which only needs to build the product once to be able to use it.
A few portion of users will need to change some source code. In that case, they'll have to clean everything, because the makefile provided isn't the way the actual developpers manage their builds. They may use other systems (code::blocks, Ant, gprbuild...) , and just provide the makefile to automate production from scratch and avoid to depend on a complex production system. make is fairly standard even on Windows/MinGW.
Note that there are some filesystems which provide build audit (Clearcase) where the dependencies are automatically managed (clearmake).
If you see the makefile as a batch script to build all the sources, you don't need to bother adding a dependency system using
a template makefile
a gcc -MM command to append dependencies to it (which takes time)
Note that you can build it yourself with some extra work (adding a depend target to your makefile)
Configuration
I use autotools (autoreconf -iv and ./configure) to generate correct makefiles. On my development machine (Fedora) everything works correctly. For make check I use the library libcheck and from autotools I use Libtools. On Fedora the library for check is dynamic: libcheck.so.0.0.0 or something such. It works.
The Issue
When I push the commits to my repo on github and do a pull request, the result is tested on Travis CI which uses Ubuntu as a platform. Now on Ubuntu the libcheck is a static library: libcheck.a and a libcheck_pic.a.
When Travis does a make check I get the following error message:
/usr/bin/ld: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../..
/../libcheck.a(check.o): relocation R_X86_64_32 against.rodata.str1.1' can
not be used when making a shared object; recompile with -fPIC`
/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../libcheck.a: could not read
symbols: Bad value
Which means I have to somehow let configure determine what library I need. I suspect I need libcheck_pic.a for Ubuntu and the regular libcheck.so for Fedora.
The question
Does anyone know how to integrate this into configure.ac and test/Makefile.am using libtool? I would prefer to stay in line with autotools way of life.
I couldn't find usable information using google, but there is a lot of questions out there about what the difference is between static and dynamic (which is not what I need).
I would much appreciate if anyone could point me in the right direction, or maybe even solved this already?
I suspect you're right that the library you want to use on the CI system is libcheck_pic.a, for its name suggests that the routines within are compiled as position-independent code, exactly as the error message you receive suggests that you do.
One way to approach the problem, then, would be to use libcheck_pic if it is available, and to fall back to plain libcheck otherwise. That's not too hard to configure your Autotools-based build system to do. You then record the appropriate library name in an output variable, and use that in your (auto)make files.
Autoconf's SEARCH_LIBS macro specifically serves this kind of prioritized library search requirement, but it has the side effect, likely unwanted in this case, of modifying the LIBS variable. You can nevertheless make it work. Something like this might do it, for example:
LIBS_save=$LIBS
AC_SEARCH_LIBS([ck_assert], [check_pic check], [
# Optional: add a test to verify that the chosen lib really provides PIC code.
# Set LIBCHECK to the initial substring of $LIBS up to but excluding the first space.
LIBCHECK=${LIBS%% *}
], [
# or maybe make it a warning, and disable your test suite when libcheck
# is not available.
AC_MSG_ERROR([A PIC version of libcheck is required])
])
AC_OUTPUT([LIBCHECK])
LIBS=$LIBS_save
I presume you know what to do with $(LIBCHECK) on the Make side.
As written, that has the limitation that if there is no PIC version of libcheck available then you won't find out until make or maybe make check. That's undesirable, and you could add Autoconf code to detect that situation if it is undesirable enough.
As an altogether different approach, you could consider building your tests statically (add -static to the appropriate *_LDFLAGS variable). Of course, this has the opposite problem: if a static version of the library is not available, then the build or tests fail. Also, it requires building a static version of your own code if you're not doing so already, and furthermore, it is the static version that will be tested.
For greatest flexibility, you could consider combining those two approaches. You might set up a fallback from one to the other, or you might set up separate targets for testing static code and PIC code, and exercise whichever of them (possibly both) is supported by the libraries available on the build system.
I recently built an older version of GCC and installed it in my home directory (spec. ~/local/gcc-5.3.0). However, I need this compiler only for CUDA projects, and will be working with the system compiler (GCC 6.2.1) rest of the time. So, I guess I need to find a way to switch between these as and when needed, and in a way that also changes the library and include paths appropriately.
I understand that update-alternatives is one way to do so, but it seems to require root permissions to be set up, which I don't have.
The next best thing might be to write a shell function in .bashrc that ensures the following:
Each call switches between system and local gcc
Whenever a switch is made, it adjusts paths so that when local gcc is chosen, it first looks for header files and libraries that were installed by itself before looking in system paths like /usr/local/include or usr/local/lib. A previous answer suggests that modifying LD_LIBRARY_PATH should be sufficient, because a GCC installation "knows" where its own header files and static libraries are (I am not sure if it's correct, I was thinking I might need to modify CPATH, etc).
Is the above the best way to achieve this? If so, what paths should I set while implementing such a function?
Is the above the best way to achieve this? If so,
what paths should I set while implementing such a function?
As others pointed out, PATH and LD_LIBRARY_PATH are mandatory. You may also update MANPATH for completeness.
Rather than reinventing the wheel in .bashrc I suggest to employ a little known but extremely handy and modular Environment Modules that were designed for this specific purpose. You could use them like (once you set up config for gcc/3.1.1):
$ module load gcc/3.1.1
$ which gcc
/usr/local/gcc/3.1.1/linux/bin/gcc
$ module unload gcc
$ which gcc
gcc not found
Ubuntu has libiconv built-in to it’s standard c library and does not require it in LDFLAGS. OS X does not have it built-in and requires the flag to be set.
My current approach is using ifeq in my Makefile to conditionally set LDFLAG += -liconv when on OS X.
I am wondering if there is a better approach? I am heavily influenced by the feature-detection mindset of web development and hope I can use a similar approach to detect whether the flag is required on the current system or not.
For manually written makefiles individually setting the contents of the LDFLAGS and friends is an acceptable way of doing it. This may be simpler if there is only couple of features needed.
If you want to generically detect the feature one route to go would be to use a makefile generator like CMake or Automake.
The final option would be to manually perform the runtime checks that Automake does. That is to have a sample file that you compile, link with one of the options and see if it works. If it doesn't try the next and so on. This way you are testing what flags are required for each feature. Automake does this onece, when the project is ./configured but you could do it every time make is run or whenever is good for your build setup.