When linking a program against a shared object, ld will ensure that symbols can be resolved. This basically ensures that the interfaces between the program and its shared objects are compatible. After reading Linking with dynamic library with dependencies, I learnt that ld will descend into linked shared objects and attempt to resolve their symbols too.
Aren't my shared object's references already checked when the shared objects are themselves linked?
I can understand the appeal of finding out at link time whether a program has all the pieces it requires to start, but does it seems irrelevant in the context of packages building where shared objects may be distributed separately (Debian's lib* packages, for instance). It introduces recursive build dependencies on systems uninterested in executing built programs.
Can I trust the dependencies resolved when the shared object was built? If so, how safe is it to use -unresolved-symbols=ignore-in-shared-libs when building my program?
You're wondering why a program's linkage should bother to resolve symbols originating in
the shared libraries that it's linked with because:
Aren't my shared object's references already checked when the shared objects are themselves linked?
No they're not, unless you expressly insist on it when you link the shared library,
Here I'm going to build a shared library libfoo.so:
foo.c
extern void bar();
void foo(void)
{
bar();
}
Routinely compile and link:
$ gcc -fPIC -c foo.c
$ gcc -shared -o libfoo.so foo.o
No problem, and bar is undefined:
$ nm --undefined-only libfoo.so | grep bar
U bar
I need to insist to get the linker to object to that:
$ gcc --shared -o libfoo.so foo.o -Wl,--no-undefined
foo.o: In function `foo':
foo.c:(.text+0xa): undefined reference to `bar'
Of course:
main.c
extern void foo(void);
int main(void)
{
foo();
return 0;
}
it won't let me link libfoo with a program:
$ gcc -c main.c
$ gcc -o prog main.o -L. -lfoo
./libfoo.so: undefined reference to `bar'
unless I also resolve bar in the same linkage:
bar.c
#include <stdio.h>
void bar(void)
{
puts("Hello world!");
}
maybe by getting it from another shared library:
gcc -fPIC -c bar.c
$ gcc -shared -o libbar.so bar.o
$ gcc -o prog main.o -L. -lfoo -lbar
And then everything's fine.
$ export LD_LIBRARY_PATH=.; ./prog
Hello world!
It's of the essense of a shared library that it doesn't by default have
to have all of its symbols resolved at linktime. That way that a program - which
typically does need all its symbols resolved a linktime - can get all its symbols
resolved by being linked with more than one library.
Aren't my shared object's references already checked
when the shared objects are themselves linked?
Well, shared libs might have been linked with -Wl,--allow-shlib-undefined or with dummy dependencies so it still makes sense to check them.
Can I trust the dependencies resolved when the shared object was built?
Probly not, current linking environment and the environment used to link original shlibs may be different.
If so, how safe is it to use -unresolved-symbols=ignore-in-shared-libs
when building my program?
You may be missing potential errors in this case (or rather delaying them to runtime which is still bad). Imagine a situation where some of the symbols needed by shared objects are to come from executable itself or from one of the libs which is linked by executable (but not by the shlib which is missing the symbols).
EDIT
Although above is correct, Mike Kinghan's answer gives stronger argument in favor of symbol resolution in libraries during executable link.
Related
I have a project with thousands of C files, many libraries, and dozens of programs to link, and to speed up the compilation, I am combining C files into translation units that include multiple C files. This is sometimes referred to as single compilation unit, single translation unit, or unity build.
I have multiple of these translation units compiled into different libraries, and these libs were previously created by compiling each C file individually.
For example:
old library.lib:
file1.o
file2.o
file3.o
file4.o
file5.o
file6.o
new library.lib:
translation_unit_1.o
translation_unit_2.o
translation_unit_1.c:
#include "file1.c"
#include "file2.c"
#include "file3.c"
translation_unit_2.c:
#include "file4.c"
#include "file5.c"
#include "file6.c"
So these compile into: translation_unit_1.o and translation_unit_2.o. And the library is the new library.lib shown above.
Now say I have a program that I want to link to library.lib that refers to a function in file2.c. But has a different version of file1.c that it compiles that duplicates symbols in the file1.c in the library, so it only needs file2.c from the library.lib to link. Or perhaps I have a need to link code from file1.c but can't link file2.c because it has a dependency that I don't want to rely on (example below).
program:
main.o
file1.o
library.lib
Is there a way with any linker that you know of to get the linker to only pull the code from file2.c out of translation_unit_1.o object code and use that to link main.o to make the program?
An alternative would be to split the translation_unit_1.o out into file1.o, file2.o, file3.o if that is possible, then feed that to the linker.
Thanks for any help.
edit 1
This is for single code base that is compiled for both a bare metal ARM platform that uses ELF compiled with ARM ADS 1.2 toolchain and for a Windows platform that uses the Visual Studio toolchain. However thoughts on how to approach the problem on other platforms and toolchains are welcome.
Here is a concrete example on MacOS using clang.
example code below is here: https://github.com/awmorgan/single_translation_unit_lib_link
library:
file1.c this file is needed to link
file2.c this file is not used to link and has an unresolved dependency which could be in another library or object
main.c:
int main( void ) {
extern int file1_a( void );
int x = file1_a();
}
file1.c:
int file1_a(void) {
return 1;
}
file2.c:
int file2_a( void ) {
extern int file3_a( void );
return file3_a(); // file3_a() is located somewhere else
}
single_translation_unit.c:
#include "file1.c"
#include "file2.c"
this works to produce program1.out:
++ clang -c file1.c -o file1.o
++ clang -c file2.c -o file2.o
++ libtool -static file1.o file2.o -o library1.lib
++ clang -c main.c -o main1.o
++ clang main1.o library1.lib -o program1.out
this fails to produce program2.out:
++ clang -c single_translation_unit.c -o single_translation_unit.o
++ libtool -static single_translation_unit.o -o library2.lib
++ clang -c main.c -o main2.o
++ clang main2.o library2.lib -o program2.out
Undefined symbols for architecture x86_64:
"_file3_a", referenced from:
_file2_a in library2.lib(single_translation_unit.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
changing the link order does not work either:
++ clang library2.lib main2.o -o program2.out
Undefined symbols for architecture x86_64:
"_file3_a", referenced from:
_file2_a in library2.lib(single_translation_unit.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Is there a way with clang, gcc, microsoft or any linker
None of clang, gcc or microsoft is a linker (the first two are compilers, and the third is a corporation).
The answer also depends on the platform (which you didn't specify).
IF you are building on a Linux, or another ELF platform, you could compile your code with -ffunction-sections -fdata-sections, and the linker will automagically do what you want.
Is there a way to have a linker pull part of an object file from a library for linking?
In general, linkers operate on sections, and can't split sections apart (you get all or nothing).
Without -ffunction-sections, all functions in a single translation unit end up in a single .text section (this is an approximation -- template instantiations and out-of-line function definitions for inline functions usually end up in a section of their own). Therefore, the linker can't select some, but not all, parts of the .text.
With the GCC/binutils ELF toolchain, or suitably compatible tools, you can do this by:
Compiling single_translation_unit.c with the options -ffunction-sections, -fdata-sections
Linking program2.out with the linker option option -gc-sections.
E.g. (on Linux):
$ gcc -ffunction-sections -fdata-sections -c single_translation_unit.c -o single_translation_unit.o
$ ar rcs library2.a single_translation_unit.o # On Mac OS, use libtool to make the static library if you prefer.
$ gcc -c main.c -o main2.o
$ gcc main2.o library2.a -Wl,-gc-sections -o program2.out
You may replace gcc with clang throughout.
The linkage succeeds because:
In compilation, -ffunction-sections directed the compiler to emit each function definition
in a distinct code section of the object file, containing nothing else, rather than merging them all into
a single .text section, as per default.
In the linkage, -Wl,-gc-sections directed the linker to discard unused sections,
i.e. sections in which no symbols were referenced by the program.
The definition of the unreferenced function file2_a acquired a distinct code section,
containing nothing else, which was therefore unused. The linker was able to discard this unused section, and along with it
the unresolved reference to file3_a within the definition of file2_a.
So no references to file2_a or file3_a were finally linked, as we can see:
$ nm program2.out | egrep '(file2_a|file3_a)'; echo Done
Done
And if we re-do the linkage requesting a mapfile:
$ gcc main2.o library2.a -Wl,-gc-sections,-Map=mapfile -o program2.out
then the mapfile will show us:
...
...
Discarded input sections
...
...
.text.file2_a 0x0000000000000000 0xb library2.a(single_translation_unit.o)
...
...
that the function section text.file2.a originating in library2.a(single_translation_unit.o)
was indeed thrown away.
BTW...
Because of the way a static library is used in linkage,
there is no point in archiving the single object file single_translation_unit.o alone into a static library
library2 and then linking your program against library2, if you know that your program references any
symbol defined in single_translation_unit.o. You might as well skip creating library2 and just link single_translation_unit.o instead.
Given that symbols defined in single_translation_unit.o are needed, the linkage:
$ gcc main2.o library2.a [-Wl,-gc-sections] -o program2.out
is exactly the same linkage as:
$ gcc main2.o single_translation_unit.o [-Wl,-gc-sections] -o program2.out
with or without -Wl,-gc-sections.
And...
I trust you're aware that while a unity build well be fastest for your builds-from-clean,
it may equally well be slow for most incremental builds, as against an automated build system, typically Make based,
that is well-crafted to minimise the amount of rebuilding required per source change. Chances are if you can
benefit from a unity build, it's only from a unity build as well as an efficient incremental build.
I am trying to understand static libraries and shared objects in C. I am trying to understand whether one type of library can depend on other type.
Consider a scenario:
libA.so has a function foo_A_dyn():
libA.so ---> foo_A_dyn()
foo_A_dyn() uses a function foo_B_static() which is defined in libB.a which is a static library.
libB.a ---> foo_B_static()
I have built my libraries in the following way:
gcc -c foo_B.c -o foo_B.o
ar -cvq libB.a foo_B.o
gcc -fPIC -c foo_A.c -o foo_A.o
gcc -shared libA.so foo_A.o -I.
gcc main.c -lA -lB -L. -I. -o EXE
Note: main.c makes call to foo_A_dyn() and does NOT call foo_B_static() directly.
And now when I am trying to build my executable EXE, I am getting the error "undefined reference to foo_B_static".
I think the error seems genuine but I am not able to decode the rationale behind this and put it to words.
Can someone please help?
From gcc link options:
-llibrary
-l library
...
It makes a difference where in the command you write this option; the linker searches and processes libraries and object files in the order they are specified. Thus, ‘foo.o -lz bar.o’ searches library ‘z’ after file foo.o but before bar.o. If bar.o refers to functions in ‘z’, those functions may not be loaded.
Try:
gcc main.c -lB -lA -L. -I. -o EXE
Here's what the linker is doing. When we link our executable ('EXE' above) it has some symbols (functions and other things) that are unresolved. It will look down the list of libraries that follow in sequential order, trying to resolve unresolved symbols. Along the way, it finds that some of the symbols are provided by libB.so, so it notes that they are now resolved by this library. While going through libB.so it finds some symbols which are unresolved and it tries to resolve them by looking up the library that follows.
When we are ordering the libraries like:
gcc main.c -lA -lB -L. -I. -o EXE
Linker is not able to lookup for the definition of symbols used in libB into libA. Reason could be that backward reference is not available.
I have also figured out that:
shared object can depend on a static archive,
a static archive can depend on a shared object, and
one static archive can depend on another static archive
Please let me know if I have erred somewhere.
I have files foo.c bar.c and baz.c, plus wrapper code myfn.c defining a function myfn() that uses code and data from those other files.
I would like to create something like an object file or archive, myfn.o or libmyfn.a, so that myfn() can be made available to other projects without also exporting a load of symbols from {foo,bar,baz}.o as well.
What's the right way to do that in Linux/gcc? Thanks.
Update: I've found one way of doing it. I should've emphasised originally that this was about static archives, not DSOs. Anyway, the recipe:
#define PUBLIC __attribute__ ((visibility("default"))) then mark myfn() as PUBLIC in myfn.c. Don't mark anything else PUBLIC.
Compile objects with gcc -c foo.c bar.c baz.c myfn.c -fvisibility=hidden, which marks everything as hidden except for myfn().
Create a convenience archive using ld's partial-linking switch: ld -r foo.o bar.o baz.o myfn.o -o libmyfn.a
Localise everything that wasn't PUBLIC like so: objcopy --localize-hidden libmyfn.a
Now nm says myfn is the only global symbol in libmyfn.a and subsequent linking into other programs works just fine: gcc -o main main.c -L. -lmyfn (here, the program calls myfn(); if it tried to call foo() then compilation would fail).
If I use ar instead of ld -r in step 3 then compilation fails in step 5: I guess ar hasn't linked foo etc to myfn, and no longer can once those functions are localised, whereas ld -r resolves the link before it gets localised-away.
I'd welcome any response that confirms this is the "right" way, or describes a slicker way of achieving the same.
Unfortunately, C linkage for globals is all-or-nothing, in the sense that the globals of all modules would be available in libmyfn.a's final list of external symbols.
gcc tool chain offers an extension that lets you hide symbols from outside users, while making them available to other translation units in your library:
foo.h:
void foo();
foo.c:
void foo() __attribute__ ((visibility ("hidden")));
myfn.h:
void myfn();
myfn.c:
#include <stdio.h>
#include "foo.h"
void myfn() {
printf("calling foo...\n");
foo();
printf("calling foo again...\n");
foo();
}
For portability, you would probably benefit from making a macro for __attribute__ ((visibility ("hidden"))), and placing it in a conditional compilation block conditioned on gcc.
In addition, Linux offers a utility called strip, which lets you remove some of the symbols from compiled object files. Options -N and -K let you identify individual symbols that you want to keep or remove.
Start with this to build a static library
gcc -c -O2 foo.c bar.c baz.c myfn.c
ar av libmyfunctions.a foo.o bar.o baz.o myfn.o
Compile and link with other programs like:
gcc -O2 program.c -lmyfunctions -o myprogram
Now your libmyfunctions.a will ultimately have extra stuff from the source that isn't required by the code in myfn.c But the linker should do a reasonable job of removing this when it creates the final program.
Suppose myfn.c has function myfun() which you want to use in other three files foo.c, bar.c & baz.c
Now create a shared library from code in myfn.c viz libmyf.a
Use this function call myfun() in other three files. Declare function as extern in these files. Now you can create object code of these thee files and link the libmyf.a at linking phase.
Refer to following link for using shared libraries.
http://www.cprogramming.com/tutorial/shared-libraries-linux-gcc.html
Im wondering why Im getting no undefined references when compiling as a shared library using -shared as an option for gcc. Consider the following case:
#include <confuse.h>
int
main(int argc, char **argv)
{
cfg_opt_t opts[1];
cfg_t *cfg = cfg_init(opts, CFGF_NOCASE);
return 0;
}
libconfuse is needed here in order to run the program properly. If im compiling it as a "normal" application without specifying that libconfuse is needed (-lconfuse) im getting the following (from my perspective regular) error:
$ gcc -Wall -Wno-unused-variable test.c -o test
/tmp/ccTVz6an.o: In function `main':
test.c:(.text+0x20): undefined reference to `cfg_init'
collect2: error: ld returned 1 exit status
If im compiling the same code as a shared library im not getting any error message regarding the library:
$ gcc -Wall -Wno-unused-variable test.c -o test.so -shared
$ echo $?
0
Can someone please bring light into darkness?
You are compiling a (shared) library, not a program, and libraries are expected not to be complete, so undefined references do not prevent the shared library from building.
Yes, it may sound a bit surprising, particularly if you come from a OS (Windows?) where the shared libraries are always fully linked, but that's how ELF works.
If you want to ensure that all your references are are resolved when building a shared library you can use the linker option --no-undefined:
gcc -Wall -Wno-unused-variable test.c -o test.so -shared -Wl,--no-undefined
Note that libraries used to resolve references when linking a shared library will be added to the header of that library, as NEEDED entries, and then linked automatically when using that library. See objdump -p for details.
With shared libraries (not static ones), you could link a library libA.so into your other library libB.so.
So you could compile your shared library as:
gcc -Wall -Wno-unused-variable -fPIC test.c \
-o libmytest.so -shared -lconfuse
Later you would link that library to some main.c with
gcc -Wall main.c -lmytest -L. -o myprog
or you could load it at runtime, using dlopen(3) on ./libmytest.so (read the man page about why ./ is significant to dlopen)
Simple explanations are in the Program Library HowTo. But Drepper's paper: How to Write Shared Libraries is the best reference.
And it is permitted for a shared library (or even a static one) to have undefined references, since quite often the user of that library would explicitly link the lower level libraries needed by it.
I'm following this answer to override a call to a C function of a C library.
I think I did everything correctly, but it doesn't work:
I want to override the "DibOpen"-function. This is my code of the library which I pass to LD_PRELOAD environment-variable when running my application:
DIBSTATUS DibOpen(void **ctx, enum Board b)
{
printf("look at me, I wrapped\n");
static DIBSTATUS (*func)(void **, enum Board) = NULL;
if(!func)
func = dlsym(RTLD_NEXT, "DibOpen");
printf("Overridden!\n");
return func(pContextAddr, BoardType, BoardHdl);
}
The output of nm lib.so | grep DibOpen shows
000000000001d711 T DibOpen
When I run my program like this
LD_PRELOAD=libPreload.so ./program
I link my program with -ldl but ldd program does not show a link to libdl.so
It stops with
symbol lookup error: libPreload.so: undefined symbol: dlsym
. What can I do to debug this further? Where is my mistake?
When you create a shared library (whether or not it will be used in LD_PRELOAD), you need to name all of the libraries that it needs to resolve its dependencies. (Under some circumstances, a dlopened shared object can rely on the executable to provide symbols for it, but it is best practice not to rely on this.) In this case, you need to link libPreload.so against libdl. In Makefile-ese:
libPreload.so: x.o y.o z.o
$(CC) -shared -Wl,-z,defs -Wl,--as-needed -o $# $^ -ldl
The option -Wl,-z,defs tells the linker that it should issue an error if a shared library has unresolved undefined symbols, so future problems of this type will be caught earlier. The option -Wl,--as-needed tells the linker not to record a dependency on libraries that don't actually satisfy any undefined symbols. Both of these should be on by default, but for historical reasons, they aren't.