I am working on a free software application targeted at meteorologists and climate scientists called PIO: https://github.com/NCAR/ParallelIO.
As implied by the name it is a parallel I/O library using MPI.
I am trying to turn on profiling for the accompanying MPE library. There are lots of different documents on line that all mention the option -mpilog, but it's not clear where it should be used, or if something else should be used.
When I try adding -mpilog to either CC, CFLAGS, or LDFLAGS, configure fails with the message that the C compiler does not work.
So how do I turn on logging with MPE? I am using MPICH 3.2 on a Linux system.
I have gotten this working, and here's how.
Firstly, thanks to Gilles Gouaillardet for pointing me at the MPE github site. There is a lot of old information out there about MPE, and it has changed a lot over the years, so reading about it on the GitHub site cleared up a lot of confusion.
For my PIO project, not only did I have to build PIO with MPE, but also HDF5, netcdf-c, and pnetcdf.
HDF5 was built like this:
CC='gcc' CPPFLAGS='-I/usr/local/zlib-1.2.11/include' LDFLAGS='-L/usr/local/zlib-1.2.11/lib' LIBS='-llmpe -lmpe -lmpi -lpthread' ./configure --prefix=/usr/local/hdf5-1.10.5_mpe_static --disable-shared --enable-parallel
netCDF was built like this:
CC='gcc' CPPFLAGS='-I/usr/local/zlib-1.2.11/include -I/usr/local/hdf5-1.10.5_mpe_static/include' LDFLAGS='-L/usr/local/zlib-1.2.11/lib -L/usr/local/hdf5-1.10.5_mpe_static/lib' ./configure --prefix=/usr/local/netcdf-c-4.7.0_hdf5-1.10.5_mpe_static_nodap --disable-shared --disable-dap
pnetcdf was built like this:
CC=gcc LIBS='-llmpe -lmpe -lmpi -lpthread' ./configure --prefix=/usr/local/pnetcdf-1.11.0_mpe --disable-shared --disable-cxx --disable-fortran
The PIO is built like this:
autoreconf -i && CC='gcc' CPPFLAGS='-I/usr/local/pnetcdf-1.11.0_mpe/include -I/usr/local/zlib-1.2.11/include -I/usr/local/hdf5-1.10.5_mpe_static/include -I/usr/local/netcdf-c-4.7.0_hdf5-1.10.5_mpe_static_nodap/include' LDFLAGS='-L/usr/local/pnetcdf-1.11.0_mpe/lib -L/usr/local/zlib-1.2.11/lib -L/usr/local/hdf5-1.10.5_mpe_static/lib -L/usr/local/netcdf-c-4.7.0_hdf5-1.10.5_mpe_static_nodap/lib' LIBS=' -lhdf5_hl -lhdf5 -lz -ldl -lm -llmpe -lmpe -lmpi -lpthread' ./configure --disable-shared --enable-mpe
Note that all the mpicc/mpecc compiler wrappers seemed unable to yield the correct results for this build, because the libraries would be listed in an incorrect order. Only by using CC=gcc, and explicitly linking to -llmpe -lmpe -lmpi -lpthread can this be made to build.
Once built, it gives very nice charts:
Related
I am trying to wrap the socket() system call with my own custom __wrap_socket() system call in a wrap.c file in lighttpd (a web server application).
Lighttpd uses auto-tools to build, and I am attempting to build it completely static using musl.
The configure script I pass is as follows:
CC="/home/riscv64-linux-musl-cross/bin/riscv64-linux-musl-gcc" CFLAGS="-g -Wl,--wrap=socket -static --static -static-libstdc++ -static-libgcc /home/lighttpd/wrap.c" LDFLAGS="-L/musl/lib/ -lcrypt -lc" ./configure --prefix=/lighttpd/install/ --enable-static --without-zlib --without-bzip2 --without-pcre
However, due to lighttpd's build script and Makefile generated by autotools, I get the following errors (I have only included some of the many errors):
/home/riscv64-linux-musl-cross/bin/riscv64-linux-musl-gcc -DHAVE_CONFIG_H -DHAVE_VERSIONSTAMP_H -DLIBRARY_DIR="\"/home/lighttpd/install/lib\"" -DSBIN_DIR="\"/home/lighttpd/install/sbin\"" -I. -I.. -DDEBUG_PROC_OPEN -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGE_FILES -g -Wl,--wrap=socket -static --static -static-libstdc++ -static-libgcc /home/lighttpd/wrap.c -Wall -W -Wshadow -pedantic -MT proc_open-proc_open.o -MD -MP -MF .deps/proc_open-proc_open.Tpo -c -o proc_open-proc_open.o `test -f 'proc_open.c' || echo './'`proc_open.c
riscv64-linux-musl-gcc: fatal error: cannot specify ‘-o’ with ‘-c’, ‘-S’ or ‘-E’ with multiple files
However, if I do not include the path to the wrap.c file in the CFLAGS argument, I get the following errors:
/home/riscv64-linux-musl-cross/bin/../lib/gcc/riscv64-linux-musl/10.2.1/../../../../riscv64-linux-musl/bin/ld: /home/musl/lib//libc.a(if_nametoindex.lo):/home/musl-1.1.23/src/network/if_nametoindex.c:13: more undefined references to `__wrap_socket' follow
and
/home/riscv64-linux-musl-cross/bin/../lib/gcc/riscv64-linux-musl/10.2.1/../../../../riscv64-linux-musl/bin/ld: /home/lighttpd/src/fdevent.c:491: undefined reference to `__wrap_socket'
I have the wrap.c function (__wrap_socket()) in a file wrap.h which I have included in lighttpd's main file (server.c). Any help would be greatly appreciated.
There are lots of issues here. Among them:
You are using the CC variable to sub in a cross compiler, but that's the wrong way to perform a cross build with an Autotools build system. The right way is to specify the system type for cross compilation is via a --host triplet. It looks like you may want --host=riscv64-linux-musl in particular. As long as the cross tools are in the executable search path, with the expected names, configure will then find them -- including the correct linker for the toolchain, whose use should reduce the number of flags you need to specify. configure will also behave a bit differently than it does for configuring an ordinary build.
You are specifying link flags in your CFLAGS variable, which is for compilation flags. This might not always cause a problem in practice, but it is a muddle that you should avoid. LDFLAGS is the correct variable for link flags other than -l and maybe -L options, including -Wl,--wrap=socket, -static, --static, -static-libstdc++, and -static-libgcc, though some of those should not be necessary at all.
You have put a source file name into your CFLAGS. This flat out will not work. The build system will build each source file to an object file, separately, using the CFLAGS where those are C sources. You are tricking it into trying to compile two sources with each such command, and that conflicts with some of the other flags that the build system chooses.
You appear to have a C++ project, but you are specifying C compiler flags (CFLAGS) and no C++ compiler flags (CXXFLAGS). If there are sources in both languages then you may need both. Do not expect the CFLAGS to be used when compiling C++ sources.
You are specifying link libraries in LDFLAGS. This probably won't break your build in itself, but it also will not get you the link behavior you probably expect, because link commands are sensitive to the order of command-line arguments. You should use LIBS for this, instead.
You cannot add a source file to an Autotools build without modifying the autotooling. If you want to avoid that then my suggestion would be to instead (cross-)build the extra sources into a library, and add that to the link. I leave building the library to you, but supposing that you build it as libsocket_wrap.a, you might then inject it into lighthttpd by configuring something like this:
CFLAGS="-g" \
CXXFLAGS="-g" \
LDFLAGS="-Wl,--wrap=socket -static-libstdc++ -static-libgcc" \
LIBS="/path/to/libsocket_wrap.a -lcrypt" \
./configure \
--host=riscv64-linux-musl \
--prefix=/lighttpd/install \
--enable-static \
--disable-shared \
--without-zlib \
--without-bzip2 \
--without-pcre
I cannot guarantee that that will work as-is, because much depends on details of your build environment and the specifics of the project you are trying to build. But it's a much better position to work from than the one presented in the question.
For a given sample program I want to know which part of GCC's source code is used during compilation.
BUILDING GCC:
I tried building gcc-8.2 source code using
./src/configure --enable-languages=c,c++
--disable-bootstrap --disable-multilib BOOT_CFLAGS="-O2 -g -ftest-coverage -fprofile-arcs"
I have a sample program named program.c
I am compiling the program.c with xgcc as
/path/to/xgcc -B path/to/stage1build/gcc -ftest-coverage program.c
The above command gives me code coverage of program .The problem is I don't want to find code coverage of program.c using gcov and lcov.
I want to find code coverage of gcc's source code.
I tried searching on web but almost all of the answers tell me how do I use gcov and lcov by giving example of a sample .c program .
NOTE:
I know how to use gcov and lcov
As per searching online for the BOOT_CFLAGS, it seems it sets the default compiler flags for the code that you would be compiling.
Refer here:1, 2
Probably you need to run configure something like this:
./src/configure --enable-languages=c,c++ --disable-bootstrap --disable-multilib CFLAGS="-O2 -g -ftest-coverage -fprofile-arcs" CXXFLAGS="-O2 -g -ftest-coverage -fprofile-arcs"
But you might face other issues while building with coverage enabled GCC Build with coverage.
Better would be to post on GCC mailing list to see if someone has a probable solution.
I've build a shared library on my desktop that uses statically linked gstreamer and gstreamer plugins (base, good, rtsp-server).
Now I'm trying to compile the library using yocto but its giving me a linker error:
version node not found for symbol _IO_do_write##GLIBC_2.17
failed to set dynamic section sizes: Bad value
The solutions I found on stack overflow did not seem to help me.
use compiler with --disable-symvers
link libc libs in different orders (-ldl -lm -lc -lpthread -ltinfo -lrt)
link libc libs statically/shared
What I find particularly odd is that the linker is looking for GLIBC_2.17 while yocto uses 2.27 and my system is using 2.24. I don't know if this matters or if it is normal (the function did not change since 2.17?).
NM -C shows the symbol in libc.a:
nm -C recipe-sysroot/usr/lib/libc.a | grep IO_do_write
U _IO_do_write
U _IO_do_write
0000000000001ba8 W _IO_do_write
So I would thinks that lib is linked incorrectly?
The linker command is a long one because of all the shared libs so I shortend it a bit (removed boost and custom libs):
aarch64-poky-linux-g++ -fPIC --sysroot=recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=recipe-root/git-r0 -fdebug-prefix-map=recipe-sysroot= -fdebug-prefix-map=recipe-sysroot-native= -fvisibility-inlines-hidden --sysroot=recipe-sysroot -Wl,-allow-multiple-definition -Wall -Wextra -Wpedantic -Wsuggest-override -Wswitch-default -Wduplicated-cond -Wshadow -Werror -ftemplate-depth=1024 -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -lc -Wl,--no-as-needed -Wl,--no-undefined -pthread -ldl -shared -Wl,-soname,rtsp_streamer.so -o rtsp_streamer.so ... custom static libs .and boost static libs ... -lpthread recipe-sysroot/usr/lib/gstreamer-1.0/libgstrtsp.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstrtp.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstrtpmanager.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstcoreelements.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstadder.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstapp.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstaudioconvert.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstaudiorate.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstaudioresample.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstaudiotestsrc.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstgio.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstpango.a recipe-sysroot/usr/lib/gstreamer-1.0/libgsttypefindfunctions.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstvideoconvert.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstvideorate.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstvideoscale.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstvideotestsrc.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstvolume.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstautodetect.a recipe-sysroot/usr/lib/gstreamer-1.0/libgstvideofilter.a recipe-sysroot/usr/lib/libBrokenLocale.a recipe-sysroot/usr/lib/libBrokenLocale_pic.a recipe-sysroot/usr/lib/libanl.a recipe-sysroot/usr/lib/libanl_pic.a recipe-sysroot/usr/lib/libatomic.a recipe-sysroot/usr/lib/libatomic_ops.a recipe-sysroot/usr/lib/libatomic_ops_gpl.a ... more boost static libs ... recipe-sysroot/usr/lib/libc.a recipe-sysroot/usr/lib/libc_nonshared.a recipe-sysroot/usr/lib/libc_pic.a recipe-sysroot/usr/lib/libcidn_pic.a recipe-sysroot/usr/lib/libcrypt.a recipe-sysroot/usr/lib/libcrypt_pic.a recipe-sysroot/usr/lib/libcrypto.a recipe-sysroot/usr/lib/libdl.a recipe-sysroot/usr/lib/libdl_pic.a recipe-sysroot/usr/lib/libg.a recipe-sysroot/usr/lib/libgomp.a recipe-sysroot/usr/lib/libgstallocators-1.0.a recipe-sysroot/usr/lib/libgstaudio-1.0.a recipe-sysroot/usr/lib/libgstbase-1.0.a recipe-sysroot/usr/lib/libgstcheck-1.0.a recipe-sysroot/usr/lib/libgstcontroller-1.0.a recipe-sysroot/usr/lib/libgstfft-1.0.a recipe-sysroot/usr/lib/libgstpbutils-1.0.a recipe-sysroot/usr/lib/libgstreamer-1.0.a recipe-sysroot/usr/lib/libgstriff-1.0.a recipe-sysroot/usr/lib/libgstrtp-1.0.a recipe-sysroot/usr/lib/libgstrtsp-1.0.a recipe-sysroot/usr/lib/libgstrtspserver-1.0.a recipe-sysroot/usr/lib/libgstapp-1.0.a recipe-sysroot/usr/lib/libgstnet-1.0.a recipe-sysroot/usr/lib/libgstsdp-1.0.a recipe-sysroot/usr/lib/libgsttag-1.0.a recipe-sysroot/usr/lib/libgstvideo-1.0.a recipe-sysroot/usr/lib/libhistory.a recipe-sysroot/usr/lib/libitm.a recipe-sysroot/usr/lib/liblicensing.a recipe-sysroot/usr/lib/libm.a recipe-sysroot/usr/lib/libm_pic.a recipe-sysroot/usr/lib/libmcheck.a recipe-sysroot/usr/lib/libncurses++.a recipe-sysroot/usr/lib/libncurses++w.a recipe-sysroot/usr/lib/libnsl.a recipe-sysroot/usr/lib/libnsl_pic.a recipe-sysroot/usr/lib/libnss_compat_pic.a recipe-sysroot/usr/lib/libnss_db_pic.a recipe-sysroot/usr/lib/libnss_dns_pic.a recipe-sysroot/usr/lib/libnss_files_pic.a recipe-sysroot/usr/lib/libnss_hesiod_pic.a recipe-sysroot/usr/lib/libnss_nis_pic.a recipe-sysroot/usr/lib/libnss_nisplus_pic.a recipe-sysroot/usr/lib/libprotobuf-lite.a recipe-sysroot/usr/lib/libprotobuf.a recipe-sysroot/usr/lib/libprotoc.a recipe-sysroot/usr/lib/libpthread.a recipe-sysroot/usr/lib/libpthread_nonshared.a recipe-sysroot/usr/lib/libreadline.a recipe-sysroot/usr/lib/libresolv.a recipe-sysroot/usr/lib/libresolv_pic.a recipe-sysroot/usr/lib/librpcsvc.a recipe-sysroot/usr/lib/librt.a recipe-sysroot/usr/lib/librt_pic.a recipe-sysroot/usr/lib/libsqlite3.a recipe-sysroot/usr/lib/libssl.a recipe-sysroot/usr/lib/libssp.a recipe-sysroot/usr/lib/libssp_nonshared.a recipe-sysroot/usr/lib/libstdc++.a recipe-sysroot/usr/lib/libstdc++fs.a recipe-sysroot/usr/lib/libsupc++.a recipe-sysroot/usr/lib/libthread_db_pic.a recipe-sysroot/usr/lib/libutil.a recipe-sysroot/usr/lib/libutil_pic.a recipe-sysroot/usr/lib/libz.a recipe-sysroot/usr/lib/librt.a recipe-sysroot/usr/lib/libpthread.a recipe-sysroot/usr/lib/libm.a recipe-sysroot/usr/lib/libc.a
Does anybody know what is wrong? If more info is needed please ask. Thanks in advance!
Does anybody know what is wrong?
I suspect that you are not linking against GLIBC-2.27 from Yocto, but against some other GLIBC, though it is hard to see how that could happen.
Your first step should be to find out which libc.so.6 is actually being used. You can do so by adding -Wl,-t flag to your link line. Also add -Wl,-y,_IO_do_write while you are at it.
After you know which libc.so.6 is being used, run readelf -Ws /path/to/libc.so.6 | grep _IO_do_write to see what (if any) versioned symbols are defined in it.
I don't know if this matters or if it is normal (the function did not change since 2.17)?
Yes: that is normal -- the function didn't change its ABI since GLIBC-2.17, so that's the version that is attached to it.
I figured out what went wrong. The shared library is build using a CMAKE project and our own written FindGSTREAMER.cmake. To find gstreamer, among other things, a glob is used to find all the static libs. Because on my desktop I have gstreamer installed in its seperate location this works. With Yocto however this causes every static lib in the recipe-sysroot/usr/lib directory to be linked. Including every libc library (.a, _pic.a and .so). Apparently this causes the linker unable to resolve the symbols.
Correctly filtering the libraries needed by gstreamer fixed the problem.
I'm trying to fix a bug in libxml2. I cannot get it to compile with --with-icu when using --prefix=/Server/software. I have submitted a bug report here, but I need to get it to compile for resolving a conflict when compiling PHP with intl support. I suspect it's a problem with the Makefile. My experience with Makefile's is limited. The desired result is coming up with a patch that can be submitted to the linked bug report.
The --with-icu flag causes LIBXML_ICU_ENABLED to be defined. The included code is supposed to resolve a conflict when including headers from both icu and libxml2 (specifically, both use UChar). The PHP plugin intl, activated with --enable-intl, requires icu. libxml2 is needed by PHP for DOM/XML functions.
There are two problems.
First, this config:
./configure --prefix=/Server/software --enable-shared --enable-static --with-icu
Results in:
configure: error: libicu config program icu-config not found
This happens because of this code in configure.in:
WITH_ICU=0
if test "$with_icu" != "yes" ; then
echo Disabling ICU support
else
ICU_CONFIG=icu-config
if ${ICU_CONFIG} --cflags >/dev/null 2>&1
then
ICU_LIBS=`icu-config --ldflags`
LDFLAGS="$LDFLAGS $ICU_LIBS"
WITH_ICU=1
echo Enabling ICU support
else
AC_MSG_ERROR([libicu config program icu-config not found])
fi
fi
Specifically ICUCONFIG=icu-config isn't respecting --prefix=/Server/software. I can work around this by doing export PATH=/Server/software/bin:$PATH.
This "fixes" the ./configure problem.
Second, when I run make I get errors, the most relavent being:
./include/libxml/encoding.h:31:26: error: unicode/ucnv.h: No such file or
directory
The unicode/uncv.h file is in /Server/software/include/unicode/uncv.h. I suspect the compiler is looking for this file in the local directory and in my /usr directory.
This is what the error is referring to:
#ifdef LIBXML_ICU_ENABLED
#include <unicode/ucnv.h>
#endif
Clearly this is a path issue when using --with-icu and --prefix=/Server/software. Without --with-icu it compiles fine, but this is needed to resolve a define UChar conflict when compiling PHP with both icu and libxml2.
The result of icu-config --cflags is:
-O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long
This is being piped into /dev/null.
The result of icu-config --ldflags is:
-lpthread -lm -L/Server/software/lib -licui18n -licuuc -licudata -lpthread -lm
What needs to be changed to resolve these issues?
So, take a look at where it's using icu-config. It should be doing something like icu-config --cppflags which should set -I/Server/Software/include or similar. You could work around it by setting CPPFLAGS to include such a parameter yourself.
Can you include the actual compile command line immediately before the error?
Sounds like a bug in libxml - it ought to search ${PREFIX}/bin for icu-config.
Also, ICU now exports pkg-config files, which are more of a standard way to find such items.
Try this before WITH_ICU :
ICU_CPPFLAGS=`icu-config --cppflags`
CPPFLAGS="$CPPFLAGS $ICU_CPPFLAGS"
update I'm going to quote Luke's last response. Glad it's working.
I solved the linker problems, so now it all works. For this question
using libxml 2.7.7 was the solution. It seems OX X 10.6 ships with
2.7.8. So for it to work you have to compile libxml2 yourself with 2.7.7. The linker problems are solved by adding LIBS="-lresolv -lstdc++" just before PHP's ./configure. If installing to a non-standard location you also need to compile ICU with
--enable-rpath. I've accepted your answer. Feel free to update it with this information :). – Luke 17 hours ago
After having trouble extending a standard makefile for having (examples, clean, (un) install, etc) I decided to try autoconf.
The only documentation I could find is how to supply a file to be compiled and installed, but not a file that will be linked and installed in to a library location.
An example a tutorial gives me:
# what flags you want to pass to the C compiler & linker
CFLAGS = --pedantic -Wall -std=c99 -O2
LDFLAGS =
# this lists the binaries to produce, the (non-PHONY, binary) targets in
# the previous manual Makefile
bin_PROGRAMS = targetbinary1 targetbinary2 [...] targetbinaryN
targetbinary1_SOURCES = targetbinary1.c myheader.h [...]
targetbinary2_SOURCES = targetbinary2.c
.
.
targetbinaryN_SOURCES = targetbinaryN.c
Of course this seems a bit limiting as it does not expose where the Cflags (or LDFLAGS in the case I wish to have) go or if they are automatically applied.
bin_PROGRAMS can be a few other options but they only seem to be for including libraries (.la) rather than compiling one, or installing the compiled shared library to system.
Am I being a bit silly packaging my library so it can be ./configured and alike? Are there any base line guides for GNU make with this sort of packaging or informal conventions that would be easier doing this? I am looking forward to researching all my options.
I found the answer by using libtool with automake, this makes a lot more sense than handling platform-specific flags and such on my own.