Context :
Trying to understand how lto (link time compilation) works
Code:
I have those files :
julia.h:
#ifndef JULIA_H
#define JULIA_H
#include <stdio.h>
int julian();
#endif // JULIA_H
julia.c :
#include "julia.h"
int julian()
{
printf("Hello Worldu!\n");
return 0;
}
compiled as a shared library like so :
gcc -O3 -fPIC -shared julia.c -o libjulia.so -L$PWD -I$PWD -flto
and my main program :
main.c
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include "julia.h"
int main()
{
julian();
return 0;
}
compiled with :
gcc -O3 main.c -I/path/to/inc -L/path/to/lib -Wl,-rpath=/path/to/lib -ljulia -flto
It compiles fines.
Question :
So, this is a hello world program but am I doing it right with LTO ?
Is is all it takes to optimize the linkage ?
Thanks
As keltar saied, LTO doesn't affect shared libraries. But...
LTO works with static libraries
Just replace ar by gcc-ar and add the option --plugin gccpath/liblto_plugin.so. This LTO plugin will copy the declarations, types, callgraph and GIMPLE representation from LTO-compiled objects into the static lib. (same for ranlib to be replaced by gcc-ranlib)
In your example
# First retrieve the GCC path
gccpath=$(gcc -print-search-dirs | awk '/install/{print $2}')
# Compile the static library
gcc julia.c -o julia.o -flto -ffat-lto-objects
gcc-ar rcs libjulia.a julia.o --plugin $gccpath/liblto_plugin.so
# Compile & link the executable
gcc main.c libjulia.a -flto -Ofast -march=native
Note: -Ofast introduced in GCC-4.6 [ref] (else use -03)
Update Makefile
GCCPATH = $(shell gcc -print-search-dirs | awk '/install/{print $$2}')
AR = gcc-ar
RANLIB = gcc-ranlib
ARFLAGS += --plugin $(GCCPATH)/liblto_plugin.so
RANLIBFLAGS += --plugin $(GCCPATH)/liblto_plugin.so
CFLAGS += -flto -ffat-lto-objects
CXXFLAGS += -flto -ffat-lto-objects
LDFLAGS += -flto=8 # 8 -> compiles using 8 threads
Do not forget, the real compilation will be done at link time. Therefore, move your optimization flags from CFLAGS (and CXXFLAGS) to LDFLAGS ;-) One more thing, debugging info and LTO is still experimental in GCC-4.9. GCC-5.0 should improve this point...
LTO doesn't affect shared libraries; they're being linked with by dynamic linker, which is not aware of LTO and can't modify code at runtime.
Moreover, LTO doesn't even work with static libraries, but some day it presumably will (it is TODO on gcc wiki).
But yes, what it takes to enable is using -flto on both compilation and linking phases.
Related
I have a statically linked library, containing a global variable barvar. I can compile the library with no problems with either gcc-10 or clang (this is on macOS Catalina). Interestingly, the behavior differs between the two when I try to link it into a program that uses the library. Here's the code:
In globvars.h, int barvar is declared:
#ifndef H_GLOBVARS_H
#define H_GLOBVARS_H
extern int barvar;
#endif
In globvars.c, int barvar is defined:
#include "globvars.h"
int barvar;
In foo.c, the function foo sets and prints barvar:
#include <stdio.h>
#include "globvars.h"
void foo()
{
barvar = 10;
printf("barvar is: %d\n", barvar);
return;
}
Here's test.c, the program that uses the library:
void foo();
int main(int argc, char **argv)
{
foo();
return 0;
}
When I compile and link with gcc-10, no problems:
gcc-10 -c foo.c -o foo.o
gcc-10 -c globvars.c -o globvars.o
gcc-10 -c test.c -o test.o
gcc-ar-10 rcs liblinktest.a foo.o globvars.o
gcc -o testlinkrun test2.o -L. -llinktest
When I compile and link with clang, I get an undefined symbol error at the last step:
cc -c foo.c -o foo.o
cc -c globvars.c -o globvars.o
cc -c test.c -o test.o
ar rcs liblinktest.a foo.o globvars.o
cc -o testlinkrun test2.o -L. -llinktest
with error:
Undefined symbols for architecture x86_64:
"_barvar", referenced from:
_foo in liblinktest.a(foo.o)
Any ideas? Interestingly, it appears the only step that has to be done with gcc-10 is compiling globvars.c. I can use clang and the clang linker for all other steps, and everything is fine. Is it possible that clang is optimizing away all the variables in globvars.c? How can I prevent this?
As #EricPostpischil observed in this comment, the issue is that clang defaults to treating barvar as a common symbol. Either changing int barvar; to int barvar = 0;, or compiling with -fno-common, fix the issue.
Beginning with gcc-10, gcc's default behavior is -fno-common instead of -fcommon.
I have a program which is linked (dynamically) with libm.
There are also several plugins for this program.
Plugins are loaded explicitely with dlopen().
Some of these plugins use round() from libm.
On one system (Linux Mint 19.1 gcc 7.5.0) the program
does not work because of unresolved round.
Here is simple example:
Library (lib.c)
#include <stdio.h>
#include <math.h>
void func(double a, double b)
{
double c;
c = round(a + b);
printf("c = %lf\n", c);
}
Main program (main.c)
#include <stdio.h>
#include <dlfcn.h>
void *dll;
void (*f)(double, double);
double a = 1.234, b = 4.321;
int main(void)
{
dll = dlopen("./lib.so", RTLD_LAZY);
f = dlsym(dll, "func");
f(a,b);
return 0;
}
Building (Makefile)
all:
gcc -Wall -Os -shared -fPIC lib.c -o lib.so
gcc -Wall -Os -rdynamic -fPIC main.c -o main -ldl -lm
Run on Debian 8, gcc 4.9.2
./main
c = 6.000000
Run on Linux Mint 19.1, gcc 7.5.0
./main
./main: symbol lookup error: ./lib.so: undefined symbol: round
Now, add -lm for dll compilation
gcc -Wall -Os -shared -fPIC lib.c -o lib.so -lm
./main
c = 6.000000
So, the question is - why on this particular system one must use -lm not only for main program but for plugin also?
Just like an executable program, shared libraries are linked entities (unlike static libraries which are archives of object files).
Since shared libraries are linked like executables, you also need to link with the libraries that your library depends on:
gcc -Wall -Os -shared -fPIC lib.c -o lib.so -lm
I am experimenting with externs and various methods of linking to better understand the linking process.
I have three files:
foo.c:
#include "foo.h"
int a = 4;
test.c:
#include <stdio.h>
#include "foo.h"
int main(int, char**);
int mymain();
int mymain() {
main(0, 0);
printf("test\r\n");
return 0;
}
int main(int argc, char** argv) {
printf("extern a has %d\r\n", a);
return 0;
}
foo.h:
extern int a; // defined in foo.c
If I build each file together and link at compile time using gcc like this:
gcc *.c -o final.bin
I can execute final.bin as:
./final.bin
and get expected output
extern a has 4
However, if I compile (but don't link) test.c and foo.c separately, then try and link the object files together at runtime to produce a binary, I get a segmentation fault 11 (which from what I can gather is some generic memory corruption bug like a normal segfault(?)
Here is my makefile I'm using to compile and link separately. Note I am specifying my own entry point and linking against libc to get printf()...
all: test.o foo.o
#echo "Making all..."
ld test.o foo.o -o together.bin -lc -e _mymain
test.o: test.c
#echo "Making test..."
gcc -c test.c -o test.o
foo.o: foo.c
#echo "Making foo..."
gcc -c foo.c -o foo.o
Output when running 'together.bin':
./together.bin
extern a has 4
test
Segmentation fault: 11
Perhaps my function signature for 'mymain' is wrong? My guess is that something is wrong with my 'myentry' usage.
Also, if anyone has any recommendations on good books for how linkers and loaders work, I am certainly in the market for one. I've heard mixed things about 'Linkers and Loaders', so I'm waiting on more opinions before I invest the time in that book in particular.
Thanks for any help on this... My understanding of linkers is sub-par to say the least.
Unless if you have a good reason to do so, just use gcc to link:
$ gcc test.o foo.o "-Wl,-e,_mymain" -o ./final.bin; ./final.bin
extern a has 4
test
gcc calls ld---though, with a few more arguments than you are providing in your example. If you want to know exactly how gcc invokes ld, use the -v option. Example:
$ gcc -v test.o foo.o "-Wl,-e,_mymain" -o ./final.bin
Apple LLVM version 8.0.0 (clang-800.0.38)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -dynamic -arch x86_64 -macosx_version_min 10.12.0 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -o ./final.bin test.o foo.o -e _mymain -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin/libclang_rt.osx.a
I have the next code :
test.c
#include "a1.h"
int main() {
int a = 8;
foo(a);
return a;
}
a1.h
void foo (int a);
a1.c
int f = 0;
void foo (int a, int b){
f=5+a+b;
return;
}
Pay attention that in a1.c foo has 1 more parameter than the prototype defined in a1.h.
The compiler isn't issue a warning or an error and so as coverity :
make all
Building file: ../src/a1.c
Invoking: GCC C Compiler
gcc -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/a1.d" -MT"src/a1.d" -o "src/a1.o" "../src/a1.c"
Finished building: ../src/a1.c
Building file: ../src/test.c
Invoking: GCC C++ Compiler
gcc -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/test.d" -MT"src/test.d" -o "src/test.o" "../src/test.c"
Finished building: ../src/test.c
Building target: test
Invoking: GCC C++ Linker
gcc -o "test" ./src/a1.o ./src/test.o
Finished building target: test
How can I defend myself in those cases ? I know that if I will add #include "a1.h" in the a1.c file I will get an error but is there a way to get an error without the "include " ?
Compiler isn't issuing a warning because it does not know that foo(int) from a1.h header and foo(int,int) from a1.c file is the same function. C++ allows functions to be overloaded, so both functions could potentially coexist. That is why C++ compiler cannot detect this problem, so you need to wait until the linking stage.
If you were compiling using C, not C++, you could have the compiler detect this condition simply by including a1.h at the top of a1.c file.
You're overloading foo. The version with only one parameter is never defined, hence you should get a linker error when using it.
How can I defend myself in those cases ?
You can't defend yourself from function overloading. Just make sure that you've got the same signature in both the header as the source file.
I try to write simple mongo c client. Source file (a.c):
#include <stdio.h>
#define MONGO_HAVE_STDINT
#include <mongo.h>
void mongo_init_c(mongo *con)
{
mongo_init(con);
}
int main() {
return 0;
}
And i try to compile it with:
gcc -I/usr/local/include -L/usr/local/lib -lmongoc a.c
But get an error:
a.c:(.text+0xd): undefined reference to `mongo_init'
Files /usr/local/include/mongo.h and /usr/local/lib/libmongoc.so exists
How can I correctly compile a.c?
p.s. mongo-2.0.4, gcc-4.6, mongo-c-driver - pulled from github
update
$ nm /usr/local/lib/libmongoc.so | grep init
000034e0 T _init
0000dd10 T bson_init
0000c740 T bson_init_data
0000c7b0 T bson_init_finished_data
0000dc10 T bson_init_size
0000d060 T bson_iterator_init
0000a5e0 T gridfile_init
00009af0 T gridfile_writer_init
000095e0 T gridfs_init
00010a18 R initialBufferSize
00005f40 T mongo_cursor_init
00008da0 T mongo_env_sock_init
00005d90 T mongo_init
000057b0 T mongo_init_sockets
00004800 T mongo_md5_init
00005e40 T mongo_replica_set_init
00005f00 T mongo_replset_init
00005b80 T mongo_write_concern_init
$ gcc -I/usr/local/include -L/usr/local/lib -Wall -Werror -lmongoc a.c
/tmp/cccuNEp1.o: In function `mongo_init_c':
a.c:(.text+0xd): undefined reference to `mongo_init'
Try linking the library after the source file, like gcc a.c -lmongoc. This is because you're using a traditional single-pass linker, which expects to satisfy dependencies with subsequent, not previous, objects specified on the command line.