I'm creating a little static library for having thread pools, and it depends on 2 other homemade static libraries (a homemade printf and a homemade mini libc).
But sub-functions like ft_bzero are not linked in the project unless I use them on the root project, the one that needs to use thread pools library. So I have the linking error coming from my thpool lib.
Sample :
cc -Wall -Werror -Wextra -MD -I ./ -I ./jqueue -I ../libft/incs -I
../printf/incs -o .objs/thpool_create.o -c ./thpool_create.c
ar rc libthpool.a ./.objs/thpool_create.o etcetc
In the libraries, I compile every .o and use an ar rc libthpool.a *.o. Then I compile .o from main project (a single test.c actually), and then
cc .objs/test.o -o test -L./libft -L./printf -L./thpool -lft -lftprintf -lthpool -lpthread
How can I solve my errors?
Since the code in the ftpool library uses code from ft and ftprintf, you (almost certainly) need to list the libraries in the reverse order:
cc .objs/test.o -o test -L./libft -L./printf -L./thpool -lthpool -lftprintf -lft -lpthread
When scanning a static library, the linker looks for definitions of symbols that are currently undefined. If your test code only calls functions from thpool, then none of the symbols in ft are referenced when the ft library is scanned, so nothing is included from the library; if none of the symbols from ftprintf are referenced when the ftprintf library is scanned, nothing is included from ftprintf either. When it comes across the symbols in thpool that reference things from ft or ftprintf, it's too late; the linker doesn't rescan the libraries. Hence you need to list the libraries in an order such that all references from one library (A) to another (B) are found by linking (A) before (B). If the test code references some of the functions in ft or ftprintf, you may get lucky, or a bit lucky; some symbols may be linked in. But if there are functions in thpool that make the first reference to a function in ft, with the order in the question, you've lost the chance to link everything. Hence the suggested reordering.
Another (very grubby, but nonetheless effective) technique is to rescan the static libraries by listing them several times on the command line.
With shared libraries, the rules of linking are different. If a shared library satisfies any symbol, the whole library will be available, so the linker remembers all the defined symbols, and you might well get away with the original link order.
You might need to look up 'topological sort'. You should certainly aim to design your static libraries so that there are no loops in the dependencies; that leads to cycles of dependencies, and the only reliable solutions are either to rescan the libraries or combine the libraries.
Related
I am trying to understand more about linking and shared library.
Ultimately, I wonder if it's possible to add a method to a shared library. For instance, suppose one has a source file a.c, and a library lib.so (without the source file). Let's furthermore assume, for simplicity, that a.c declares a single method, whose name is not present in lib.so. I thought maybe it might be possible to, at linking time, link a.o to lib.so while instructing to create newLib.so, and forcing the linker to export all methods/variable in lib.so to that the newLib.so is now basically lib.so with the added method from a.so.
More generally, if one has some source file depending on a shared library, can one create a single output file (library or executable) that is not dependent on the shared library anymore ? (That is, all the relevant methods/variable from the library would have been exported/linked/inlined to the new executable, hence making the dependency void). If that's not possible, what is technically preventing it ?
A somehow similar question has been asked here: Merge multiple .so shared libraries.
One of the reply includes the following text: "If you have access to either source or object files for both libraries, it is straightforward to compile/link a combined SO from them.: without explaining the technical details. Was it a mistake or does it hold ? If so, how to do it ?
Once you have a shared library libfoo.so the only ways you can use it
in the linkage of anything else are:-
Link a program that dynamically depends on it, e.g.
$ gcc -o prog bar.o ... -lfoo
Or, link another shared library that dynamically depends on it, e.g.
$ gcc -shared -o libbar.so bar.o ... -lfoo
In either case the product of the linkage, prog or libbar.so
acquires a dynamic dependency on libfoo.so. This means that prog|libfoo.so
has information inscribed in it by the linker that instructs the
OS loader, at runtime, to find libfoo.so, load it into the
address space of the current process and bind the program's references to libfoo's exported symbols to
the addresses of their definitions.
So libfoo.so must continue to exist as well as prog|libbar.so.
It is not possible to link libfoo.so with prog|libbar.so in
such a way that libfoo.so is physically merged into prog|libbar.so
and is no longer a runtime dependency.
It doesn't matter whether or not you have the source code of the
other linkage input files - bar.o ... - that depend on libfoo.so. The
only kind of linkage you can do with a shared library is dynamic linkage.
This is in complete contrast with the linkage of a static library
You wonder about the statement in this this answer where it says:
If you have access to either source or object files for both libraries, it is straightforward to compile/link a combined SO from them.
The author is just observing that if I have source files
foo_a.c foo_b.c... bar_a.c bar_b.c
which I compile to the corresponding object files:
foo_a.o foo_b.o... bar_a.o bar_b.o...
or if I simply have those object files. Then as well as - or instead of - linking them into two shared libraries:
$ gcc -shared -o libfoo.so foo_a.o foo_b.o...
$ gcc -shared -o libbar.so bar_a.o bar_b.o...
I could link them into one:
$ gcc -shared -o libfoobar.so foo_a.o foo_b.o... bar_a.o bar_b.o...
which would have no dependency on libfoo.so or libbar.so even if they exist.
And although that could be straightforward it could also be false. If there is
any symbol name that is globally defined in any of foo_a.o foo_b.o... and
also globally defined in any of bar_a.o bar_b.o... then it will not matter
to the linkage of either libfoo.so or libbar.so (and it need not be dynamically
exported by either of them). But the linkage of libfoobar.so will fail for
multiple definition of name.
If we build a shared library libbar.so that depends on libfoo.so and has
itself been linked with libfoo.so:
$ gcc -shared -o libbar.so bar.o ... -lfoo
and we then want to link a program with libbar.so, we can do that in such a way
that we don't need to mention its dependency libfoo.so:
$ gcc -o prog main.o ... -lbar -Wl,-rpath=<path/to/libfoo.so>
See this answer to follow that up. But
this doesn't change the fact that libbar.so has a runtime dependency on libfoo.so.
If that's not possible, what is technically preventing it?
What technically prevents linking a shared library with some program
or shared library targ in a way that physically merges it into targ is that a
shared library (like a program) is not the sort of thing that a linker knows
how to physically merge into its output file.
Input files that the linker can physically merge into targ need to
have structural properties that guide the linker in doing that merging. That is the structure of object files.
They consist of named input sections of object code or data that are tagged with various attributes.
Roughly speaking, the linker cuts up the object files into their sections and distributes them into
output sections of the output file according to their attributes, and makes
binary modifications to the merged result to resolve static symbol references
or enable the OS loader to resolve dynamic ones at runtime.
This is not a reversible process. The linker can't consume a program or
shared library and reconstruct the object files from which it was made to
merge them again into something else.
But that's really beside the point. When input files are physically
merged into targ, that is called static linkage.
When input files are just externally referenced in targ to
make the OS loader map them into a process it has launched for targ,
that is called dynamic linkage. Technical development has given us
a file-format solution to each of these needs: object files for static linkage, shared libraries
for dynamic linkage. Neither can be used for the purpose of the other.
With GCC on Linux, is it possible to link a .a into another .a and then only link the resultant .a to my application? Or must my application know of the dependence between one archive and another and link them both?
My understanding is that I must know of the dependencies and link all archives at the end, not in an intermediary step, which seems a little ugly.
This is slightly different than How to merge two "ar" static libraries into one as I'm after a clear description that this is only possible by working around the problem and that linking the two archives together in the naive way is incorrect and will not work, along with the reason as to why.
Yes, your application has to know the dependencies between your different static libraries.
Let's say you have two static libraries a and b.
a has a function void print_a(), and b has a function void print_b() that is calling to print_a(). So, b depends on a.
Their binaries will look like liba.a and libb.a.
Let's say that library b has a reference to a function defined in library a - void print_b(void).
When compiling library b only its symbols are defined in the binary's code section while the others are still undefined:
host$ nm libb.a | grep print
U _print_a <--- Undefined
0000000000000000 T _print_b <--- Defined, in code section
0000000000000068 S _print_b.eh
U _printf
Therefore, when compiling the application that wants to use both of the libraries, linking only to libb.a won't be enough. You'll have to link your application to both libraries. Each library will provide its own symbols addresses in the code section and then your application will be able to link to both.
Something like:
gcc -o main main.c libb.a liba.a
BTW: When compiling library b that uses a, you can but it's not necessary to link to a. The result will be just the same.
Why is this the behavior
When compiling + linking the application that uses static libraries, the symbols in the application source files have to be defined somewhere (with the exception of dynamic linking, but this is done only with dynamic libraries/shared objects. Here we deal with static ones).
Now, remember that a static library is just an archive of objects. When it's created there's no linking phase. Just:
Compiling source code (*.c) to objects (*.o)
Archiving them together in a libXXXX.a file.
It means that if this library (library b in my example) uses some function (void print_a(void)) that is defined in another library (a), this symbol won't be resolved (not as a compilation error, but as the normal behavior). It will be set as Undefined symbol (as we see in the output of nm command) after the library creation, and it will wait to be linked later to its definition. And it's OK because a static library is not executable.
Now returning to application - the linking phase of the application needs to find all the definitions of all the symbols. If you just gave it libb.a as an argument, it wouldn't be able to find the definition to print_a(), because it's not there, it's still undefined. It exists only in liba.a.
Therefore, you must provide both of the libraries.
Let libx.a and liby.a be the modules you want to combine. You can try:-
mkdir tmp # create temporary directory for extracting
cd tmp
ar x ../libx.a # extract libx.a
cp ../liby.a ../libxy.a
ar -q ../libxy.a * # add extracted files to libxy.a
cd ..
rm -rf tmp
libxy.a thus created contains .o files from both .a files
I understand object ordering is very important during linking. I've had a lot of headache before trying to get ld to resolve all symbols. This time ld didn't generate any error, but the output is wrong!
The project is big (50K+ lines of C++) and I can't generate a simplified version, so I'll try to describe what I encountered. Hopefully some expert can help me figure out.
g++ -o bad.out a.o b.o ... x.so y.so
g++ -o good.out a.o b.o ... y.so x.so
While good.out runs correctly, bad.out does not. Both x.so and y.so are provided by independent vendors, so their ordering should not matter. Here are more clues:
a.o uses x.so
b.o uses y.so
a.o and b.o are independent, but they both use some common classes
The incorrect behavior manifests as a function OnRspLogin() never called back. This is a pure virtual function defined in y.so and implemented in b.o. "grep OnRspLogin *.o *.so" only found match in y.so and b.o.
Apparently ld didn't resolved OnRspLogin() to the one in b.o, but which one did it resolve to? This worries me because linker didn't generate any error or warning.
I'm using gcc 4.4.7-4 on CentOS 6.5.
EDIT:
I found x.so and y.so both contain some common symbols (e.g. T TcpClient), so I guess linker picked x.so:TcpClient (instead of y.so:TcpClient) when resolving b.o:TcpClient. While changing .so order may solve this problem, I'm afraid that linker may incorrectly resolve some other symbols in a.o. So is there anyway to tell the linker to resolve b.o using only y.so? Note that these .so files are provided by 3rd parties and I cannot change them.
I found x.so and y.so both contain some common symbols (e.g. T TcpClient)
That is a problem. If these symbols are supposed to be distinct, then you can't link these two libraries together -- they are not link compatible.
The usual way that vendors resolve these kind of problems is that they use distinct, vendor-specific prefix on all of their exported symbols (e.g. vendorA_TcpClient) and hide all other symbols.
Note that these .so files are provided by 3rd parties and I cannot change them.
You can tell vendors that you can't use their library, unless they avoid defining symbols that aren't prefixed with their unique identifiers, and that you are not going to pay them unless they resolve this problem. Vendors often become quite responsive when do that.
I am learning C and I just read the term multiple compilation.Till now I had a single file.c and I used the command gcc file.c to compile it and then ./a.out to execute it. But I got confused a little bit. When should I use the multiple compilation instead of the single and which would be the possible reasons that they will lead me to prefer a multiple compilation instead of the single? I searched it and I found some articles but they didn't cover fully my questions. 1 (this is for c++) , 2If i undestood well if I have some files.c in my project eg file1.c, file2.c and then i want to link them, i execute
gcc file1.c
gcc file.c
gcc file1.o file2.o //somehow i have to create the .o files..
Thank you..
Compilation takes time. There's no point in re-compiling C code that hasn't changed. So, for large projects, it makes sense to split the code into multiple files (typically not randomly of course, but into modules of different functionality) and compile them only when needed.
Linking is the process of taking a bunch of object code (what the .o files are called) and turning them into a single program.
There are many steps to compiling.
When you invoke gcc it will create by default an executable file i.e. all steps in one go:
.c -> .i preprocessor
.i -> .s compiler
.s -> .o assembler
*.o -> a.out linker
Generally the first two take up the most time. If you have a large project then recompiling the entire project may take a lot of time when you are developing. So the compiler allows you to stop at a certain point and reuse previous results of files that have not changed:
gcc -E for preprocess only (rarely used)
gcc -S compile, but don't assemble. Useful for debugging or optimising assembly
gcc -c compile, assemble, but don't link. This is the most commonly used one and produces object files. Those contain your assembled functions (object code), but it's not capable of running because not all functions may be present yet, library functions are missing and the executable header has not been linked in.
The final step gcc -o executable *.o will then take all those and link them together to create an executable. Optionally linking libraries into it.
Generally having all functions in one source file will allow the compiler to do the more optimisations (i.e. inlining), but at the cost of compile time.
Have a look at https://cs.senecac.on.ca/~btp200/pages/images/compile_link.png
I'm attempting to do a release of some software and am currently working through a script for the build process. I'm stuck on something I never thought I would be, statically linking LAPACK on x86_64 linux. During configuration AC_SEARCH_LIB([main],[lapack]) works, but compilation of the lapack units do not work, for example undefiend reference to 'dsyev_' --no lapack/blas routine goes unnoticed.
I've confirmed I have the libraries installed and even compiled them myself with the appropriate options to make them static with the same results.
Here is an example I had used in my first experience with LAPACK a few years ago that works dynamically, but not statically: http://pastebin.com/cMm3wcwF
The two methods I'm using to compile are the following,
gcc -llapack -o eigen eigen.c
gcc -static -llapack -o eigen eigen.c
Your linking order is wrong. Link libraries after the code that requires them, not before. Like this:
gcc -o eigen eigen.c -llapack
gcc -static -o eigen eigen.c -llapack
That should resolve the linkage problems.
To answer the subsequent question why this works, the GNU ld documentation say this:
It makes a difference where in the command you write this option; the
linker searches and processes libraries and object files in the order
they are specified. Thus, foo.o -lz bar.o' searches libraryz' after
file foo.o but before bar.o. If bar.o refers to functions in `z',
those functions may not be loaded.
........
Normally the files found this way are library files—archive files
whose members are object files. The linker handles an archive file by
scanning through it for members which define symbols that have so far
been referenced but not defined. But if the file that is found is an
ordinary object file, it is linked in the usual fashion.
ie. the linker is going to make one pass through a file looking for unresolved symbols, and it follows files in the order you provide them (ie. "left to right"). If you have not yet specified a dependency when a file is read, the linker will not be able to satisfy the dependency. Every object in the link list is parsed only once.
Note also that GNU ld can do reordering in cases where circular dependencies are detected when linking shared libraries or object files. But static libraries are only parsed for unknown symbols once.