I would like to compile a shared library using both symbol versioning and link-time optimization (LTO). However, as soon as I turn on LTO, some of the exported symbols vanish. Here is a minimal example:
Start by defining two implementations of a function fun:
$ cat fun.c
#include <stdio.h>
int fun1(void);
int fun2(void);
__asm__(".symver fun1,fun#v1");
int fun1() {
printf("fun1 called\n");
return 1;
}
__asm__(".symver fun2,fun##v2");
int fun2() {
printf("fun2 called\n");
return 2;
}
Create a version script to ensure that only fun is exported:
$ cat versionscript
v1 {
global:
fun;
local:
*;
};
v2 {
global:
fun;
} v1;
First attempt, compile without LTO:
$ gcc -o fun.o -Wall -Wextra -O2 -fPIC -c fun.c
$ gcc -o libfun.so.1 -shared -fPIC -Wl,--version-script,versionscript fun.o
$ nm -D --with-symbol-versions libfun.so.1 | grep fun
00000000000006b0 T fun##v2
0000000000000690 T fun#v1
..exactly as it should be. But if I compile with LTO:
$ gcc -o fun.o -Wall -Wextra -flto -O2 -fPIC -c fun.c
$ gcc -o libfun.so.1 -flto -shared -fPIC -Wl,--version-script,versionscript fun.o
$ nm -D --with-symbol-versions libfun.so.1 | grep fun
..no symbols exported anymore.
What am I doing wrong?
WHOPR Driver Design gives some strong hints to what is going on. The function definitions fun1 and fun2 are not exported according to the version script. The LTO plugin is able to use this information, and since GCC does not peek into the asm directives, it knows nothing about the .symver directive, and therefore removes the function definition.
For now, adding __attribute__ ((externally_visible)) is the workaround for this. You also need to build with -flto-partition=none, so that the .symver directives do not land by accident in a different intermediate assembler file than the function definition (where it will not have the desired effect).
GCC PR 48200 tracks an enhancement request for symbol versioning at the compiler level, which would likely address this issue as well.
It looks like my externally_visible fix works. This is:
#define DLLEXPORT __attribute__((visibility("default"),externally_visible))
DLLEXPORT int fun1(void);
Also see: https://gcc.gnu.org/onlinedocs/gccint/WHOPR.html
But I think your versionscript is wrong.
If I take out the visibility overrides and change your versionscript by adding fun1 and fun2 then it works. Like:
v1 {
global:
fun; fun1;
local:
*;
};
v2 {
global:
fun; fun2;
} v1;
The symbol alias targets have to be visible as well as the alias.
I just hit the same problem - so thank you for asking this. However I've found it to be more clean to use __attribute__((used)). Since gcc is not scanning the top level assembler, it can't figure out that fun1 and fun2 are being used ... so it removes them. So it looks to me that changing definition to:
__asm__(".symver fun1,fun#v1");
int __attribute__((used)) fun1() {
printf("fun1 called\n");
return 1;
}
should be sufficient.
Related
I have a statically linked library, containing a global variable barvar. I can compile the library with no problems with either gcc-10 or clang (this is on macOS Catalina). Interestingly, the behavior differs between the two when I try to link it into a program that uses the library. Here's the code:
In globvars.h, int barvar is declared:
#ifndef H_GLOBVARS_H
#define H_GLOBVARS_H
extern int barvar;
#endif
In globvars.c, int barvar is defined:
#include "globvars.h"
int barvar;
In foo.c, the function foo sets and prints barvar:
#include <stdio.h>
#include "globvars.h"
void foo()
{
barvar = 10;
printf("barvar is: %d\n", barvar);
return;
}
Here's test.c, the program that uses the library:
void foo();
int main(int argc, char **argv)
{
foo();
return 0;
}
When I compile and link with gcc-10, no problems:
gcc-10 -c foo.c -o foo.o
gcc-10 -c globvars.c -o globvars.o
gcc-10 -c test.c -o test.o
gcc-ar-10 rcs liblinktest.a foo.o globvars.o
gcc -o testlinkrun test2.o -L. -llinktest
When I compile and link with clang, I get an undefined symbol error at the last step:
cc -c foo.c -o foo.o
cc -c globvars.c -o globvars.o
cc -c test.c -o test.o
ar rcs liblinktest.a foo.o globvars.o
cc -o testlinkrun test2.o -L. -llinktest
with error:
Undefined symbols for architecture x86_64:
"_barvar", referenced from:
_foo in liblinktest.a(foo.o)
Any ideas? Interestingly, it appears the only step that has to be done with gcc-10 is compiling globvars.c. I can use clang and the clang linker for all other steps, and everything is fine. Is it possible that clang is optimizing away all the variables in globvars.c? How can I prevent this?
As #EricPostpischil observed in this comment, the issue is that clang defaults to treating barvar as a common symbol. Either changing int barvar; to int barvar = 0;, or compiling with -fno-common, fix the issue.
Beginning with gcc-10, gcc's default behavior is -fno-common instead of -fcommon.
I have a function in my C code that is being called implicitly, and getting dumped by the linker. how can I prevent this phenomena?
I'm compiling using gcc and the linker flag -gc-sections, and I don't want to exclude the whole file from the flag. I tried using attributes: "used" and "externally_visible" and neither has worked.
void __attribute__((section(".mySec"), nomicromips, used)) func(){
...
}
on map file I can see that the function has compiled but didn't linked. am I using it wrong? is there any other way to do it?
You are misunderstanding the used attribute
used
This attribute, attached to a function, means that code must be emitted for the function even if it appears that the function is not referenced...
i.e the compiler must emit the function definition even the function appears
to be unreferenced. The compiler will never conclude that a function is unreferenced
if it has external linkage. So in this program:
main1.c
static void foo(void){}
int main(void)
{
return 0;
}
compiled with:
$ gcc -c -O1 main1.c
No definition of foo is emitted at all:
$ nm main1.o
0000000000000000 T main
because foo is not referenced in the translation unit, is not external,
and so may be optimised out.
But in this program:
main2.c
static void __attribute__((used)) foo(void){}
int main(void)
{
return 0;
}
__attribute__((used)) compels the compiler to emit the local definition:
$ gcc -c -O1 main2.c
$ nm main2.o
0000000000000000 t foo
0000000000000001 T main
But this does nothing to inhibit the linker from discarding a section
in which foo is defined, in the presence of -gc-sections, even if foo is external, if that section is unused:
main3.c
void foo(void){}
int main(void)
{
return 0;
}
Compile with function-sections:
$ gcc -c -ffunction-sections -O1 main3.c
The global definition of foo is in the object file:
$ nm main3.o
0000000000000000 T foo
0000000000000000 T main
But after linking:
$ gcc -Wl,-gc-sections,-Map=mapfile main3.o
foo is not defined in the program:
$ nm a.out | grep foo; echo Done
Done
And the function-section defining foo was discarded:
mapfile
...
...
Discarded input sections
...
...
.text.foo 0x0000000000000000 0x1 main3.o
...
...
As per Eric Postpischil's comment, to force the linker to retain
an apparently unused function-section you must tell it to assume that the program
references the unused function, with linker option {-u|--undefined} foo:
main4.c
void __attribute__((section(".mySec"))) foo(void){}
int main(void)
{
return 0;
}
If you don't tell it that:
$ gcc -c main4.c
$ gcc -Wl,-gc-sections main4.o
$ nm a.out | grep foo; echo Done
Done
foo is not defined in the program. If you do tell it that:
$ gcc -c main4.c
$ gcc -Wl,-gc-sections,--undefined=foo main4.o
$ nm a.out | grep foo; echo Done
0000000000001191 T foo
Done
it is defined. There's no use for attribute used.
Apart from -u already mentioned here are two other ways to keep the symbol using GCC.
Create a reference to it without calling it
This approach does not require messing with linker scripts, which means it will work for hosted programs and libraries using the operating system's default linker script.
However it varies with compiler optimization settings and may not be very portable.
For example, in GCC 7.3.1 with LD 2.31.1, you can keep a function without actually calling it, by calling another function on its address, or branching on a pointer to its address.
bool function_exists(void *address) {
return (address != NULL);
}
// Somewhere reachable from main
assert(function_exists(foo));
assert(foo != NULL); // Won't work, GCC optimises out the constant expression
assert(&foo != NULL); // works on GCC 7.3.1 but not GCC 10.2.1
Another way is to create a struct containing function pointers, then you can group them all together and just check the address of the struct. I use this a lot for interrupt handlers.
Modify the linker script to keep the section
If you are developing a hosted program or a library, then it's pretty tricky to change the linker script.
Even if you do, its not very portable, for example gcc on OSX does not actually use the GNU linker since OSX uses the Mach-O format instead of ELF.
Your code already shows a custom section though, so it's possible you are working on an embedded system and can easily modify the linker script.
SECTIONS {
// ...
.mySec {
KEEP(*(.mySec));
}
}
I'm in a situation that's quite similar to the following. There's libA.so that depending on some compile time flags exhibits slightly different behaviour (it's an external lib, and I can't modify the source). Then, I have libB.so that depends on libA.so (compiled with say -DVALUE=1), and in my executable I depend both on libB.so, as well as on libA.so, but compiled with -DVALUE=0. However, once I launch it, ld resolves all symbols with one of libA.so versions, so both my executable and libB.so are using the same functions.
Is there any way to specify that I want to load resolve undefined symbols of libB.so only using its dependencies? I've tried using -Wl,-Bgroup flag when building libB.so, but it didn't change anything. I know there's dlmopen that can load the library in a new namespace, but I'd like to have it loaded automatically at startup.
I'm attaching a set of files that reproduce the behaviour:
libA.so:
#include <stdio.h>
#define _STR(x) #x
#define STR(x) _STR(x)
#ifndef VALUE
#define VALUE default
#endif
void func2() {
printf(STR(VALUE) "\n");
}
void func() {
func2();
}
libB.so:
#include <stdio.h>
extern void func(void);
void b_func() {
func();
}
executable:
#include <stdio.h>
extern void b_func(void);
extern void func(void);
int main() {
func(); // should print "default"
b_func(); // should print "other"
}
build commands:
gcc -fPIC -shared A.c -o libA.so
gcc -fPIC -shared -DVALUE=other A.c -o libA2.so
gcc -fPIC -shared B.c -L. -lA2 -o libB.so
gcc main.c -L. -lA -lB -o main
Curiously, it all works fine on OS X.
I am building a shared library in C that is dynamically loaded by a program that I do not have source access to. The target platform is a 64-bit Linux platform and we're using gcc to build. I was able to build a reproduction of the issue in ~100 lines, but it's still a bit to read. Hopefully it's illustrative.
The core issue is I have two non-static functions (bar and baz) defined in my shared library. Both need to be non-static as we expect the caller to be able to dlsym them. Additionally, baz calls bar. The program that is using my library also has a function named bar, which wouldn't normally be an issue, but the calling program is compiled with -rdynamic, as it has a function foo that needs to be called in my shared library. The result is my shared library ends up linked to the calling program's version of bar at runtime, producing unintuitive results.
In an ideal world I would be able to include some command line switch when compiling my shared library that would prevent this from happening.
The current solution I have is to rename my non-static functions as funname_local and declare them static. I then define a new function:
funname() { return funname_local(); }, and change any references to funname in my shared library to funname_local. This works, but it feels cumbersome, and I'd much prefer to just tell the linker to prefer symbols defined in the local compilation unit.
internal.c
#include <stdio.h>
#include "internal.h"
void
bar(void)
{
printf("I should only be callable from the main program\n");
}
internal.h
#if !defined(__INTERNAL__)
#define __INTERNAL__
void
bar(void);
#endif /* defined(__INTERNAL__) */
main.c
#include <dlfcn.h>
#include <stdio.h>
#include "internal.h"
void
foo(void)
{
printf("It's important that I am callable from both main and from any .so "
"that we dlopen, that's why we compile with -rdynamic\n");
}
int
main()
{
void *handle;
void (*fun1)(void);
void (*fun2)(void);
char *error;
if(NULL == (handle = dlopen("./shared.so", RTLD_NOW))) { /* Open library */
fprintf(stderr, "dlopen: %s\n", dlerror());
return 1;
}
dlerror(); /* Clear any existing error */
*(void **)(&fun1) = dlsym(handle, "baz"); /* Get function pointer */
if(NULL != (error = dlerror())) {
fprintf(stderr, "dlsym: %s\n", error);
dlclose(handle);
return 1;
}
*(void **)(&fun2) = dlsym(handle, "bar"); /* Get function pointer */
if(NULL != (error = dlerror())) {
fprintf(stderr, "dlsym: %s\n", error);
dlclose(handle);
return 1;
}
printf("main:\n");
foo();
bar();
fun1();
fun2();
dlclose(handle);
return 0;
}
main.h
#if !defined(__MAIN__)
#define __MAIN__
extern void
foo(void);
#endif /* defined(__MAIN__) */
shared.c
#include <stdio.h>
#include "main.h"
void
bar(void)
{
printf("bar:\n");
printf("It's important that I'm callable from a program that loads shared.so"
" as well as from other functions in shared.so\n");
}
void
baz(void)
{
printf("baz:\n");
foo();
bar();
return;
}
compile:
$ gcc -m64 -std=c89 -Wall -Wextra -Werror -pedantic -o main main.c internal.c -l dl -rdynamic
$ gcc -m64 -std=c89 -Wall -Wextra -Werror -pedantic -shared -fPIC -o shared.so shared.c
run:
$ ./main
main:
It's important that I am callable from both main and from any .so that we dlopen, that's why we compile with -rdynamic
I should only be callable from the main program
baz:
It's important that I am callable from both main and from any .so that we dlopen, that's why we compile with -rdynamic
I should only be callable from the main program
bar:
It's important that I'm callable from a program that loads shared.so as well as from other functions in shared.so
Have you tried -Bsymbolic linker option (or -Bsymbolic-functions)? Quoting from ld man:
-Bsymbolic
When creating a shared library, bind references to global symbols to the definition within the shared library, if any. Normally, it is possible for a program linked against a shared library to override the definition within the shared library. This option can also be used with the --export-dynamic option, when creating a position independent executable, to bind references to global symbols to the definition within the executable. This option is only meaningful on ELF platforms which support shared libraries and position independent executables.
It seems to solve the problem:
$ gcc -m64 -std=c89 -Wall -Wextra -Werror -pedantic -shared -fPIC -o shared.so shared.c
$ gcc -m64 -std=c89 -Wall -Wextra -Werror -pedantic -o main main.c internal.c -l dl -rdynamic
$ ./main
main:
It's important that I am callable from both main and from any .so that we dlopen, that's why we compile with -rdynamic
I should only be callable from the main program
baz:
It's important that I am callable from both main and from any .so that we dlopen, that's why we compile with -rdynamic
I should only be callable from the main program
bar:
It's important that I'm callable from a program that loads shared.so as well as from other functions in shared.so
$ gcc -m64 -std=c89 -Wall -Wextra -Werror -pedantic -shared -fPIC -Wl,-Bsymbolic -o shared.so shared.c
$ ./main
main:
It's important that I am callable from both main and from any .so that we dlopen, that's why we compile with -rdynamic
I should only be callable from the main program
baz:
It's important that I am callable from both main and from any .so that we dlopen, that's why we compile with -rdynamic
bar:
It's important that I'm callable from a program that loads shared.so as well as from other functions in shared.so
bar:
It's important that I'm callable from a program that loads shared.so as well as from other functions in shared.so
A common solution for this problem is to not actually depend on a global symbol not being overriden. Instead do the following:
Call the function bar from your library mylib_bar or something like that
Hide mylib_bar with __attribute__((visibility("hidden"))) or similar
Make bar a weak symbol, referring to mylib_bar like this:
#pragma weak bar = mylib_bar
Make your library call mylib_bar everywhere instead of bar
Now everything works as expected:
When your library calls mylib_bar, this always refers to the definition in the library as the visibility is hidden.
When other code calls bar, this calls mylib_bar by default.
If someone defines his own bar, this overrides bar but not mylib_bar, leaving your library untouched.
I have the next code :
test.c
#include "a1.h"
int main() {
int a = 8;
foo(a);
return a;
}
a1.h
void foo (int a);
a1.c
int f = 0;
void foo (int a, int b){
f=5+a+b;
return;
}
Pay attention that in a1.c foo has 1 more parameter than the prototype defined in a1.h.
The compiler isn't issue a warning or an error and so as coverity :
make all
Building file: ../src/a1.c
Invoking: GCC C Compiler
gcc -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/a1.d" -MT"src/a1.d" -o "src/a1.o" "../src/a1.c"
Finished building: ../src/a1.c
Building file: ../src/test.c
Invoking: GCC C++ Compiler
gcc -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/test.d" -MT"src/test.d" -o "src/test.o" "../src/test.c"
Finished building: ../src/test.c
Building target: test
Invoking: GCC C++ Linker
gcc -o "test" ./src/a1.o ./src/test.o
Finished building target: test
How can I defend myself in those cases ? I know that if I will add #include "a1.h" in the a1.c file I will get an error but is there a way to get an error without the "include " ?
Compiler isn't issuing a warning because it does not know that foo(int) from a1.h header and foo(int,int) from a1.c file is the same function. C++ allows functions to be overloaded, so both functions could potentially coexist. That is why C++ compiler cannot detect this problem, so you need to wait until the linking stage.
If you were compiling using C, not C++, you could have the compiler detect this condition simply by including a1.h at the top of a1.c file.
You're overloading foo. The version with only one parameter is never defined, hence you should get a linker error when using it.
How can I defend myself in those cases ?
You can't defend yourself from function overloading. Just make sure that you've got the same signature in both the header as the source file.