I'm attempting to do a release of some software and am currently working through a script for the build process. I'm stuck on something I never thought I would be, statically linking LAPACK on x86_64 linux. During configuration AC_SEARCH_LIB([main],[lapack]) works, but compilation of the lapack units do not work, for example undefiend reference to 'dsyev_' --no lapack/blas routine goes unnoticed.
I've confirmed I have the libraries installed and even compiled them myself with the appropriate options to make them static with the same results.
Here is an example I had used in my first experience with LAPACK a few years ago that works dynamically, but not statically: http://pastebin.com/cMm3wcwF
The two methods I'm using to compile are the following,
gcc -llapack -o eigen eigen.c
gcc -static -llapack -o eigen eigen.c
Your linking order is wrong. Link libraries after the code that requires them, not before. Like this:
gcc -o eigen eigen.c -llapack
gcc -static -o eigen eigen.c -llapack
That should resolve the linkage problems.
To answer the subsequent question why this works, the GNU ld documentation say this:
It makes a difference where in the command you write this option; the
linker searches and processes libraries and object files in the order
they are specified. Thus, foo.o -lz bar.o' searches libraryz' after
file foo.o but before bar.o. If bar.o refers to functions in `z',
those functions may not be loaded.
........
Normally the files found this way are library files—archive files
whose members are object files. The linker handles an archive file by
scanning through it for members which define symbols that have so far
been referenced but not defined. But if the file that is found is an
ordinary object file, it is linked in the usual fashion.
ie. the linker is going to make one pass through a file looking for unresolved symbols, and it follows files in the order you provide them (ie. "left to right"). If you have not yet specified a dependency when a file is read, the linker will not be able to satisfy the dependency. Every object in the link list is parsed only once.
Note also that GNU ld can do reordering in cases where circular dependencies are detected when linking shared libraries or object files. But static libraries are only parsed for unknown symbols once.
Related
I am trying to understand more about linking and shared library.
Ultimately, I wonder if it's possible to add a method to a shared library. For instance, suppose one has a source file a.c, and a library lib.so (without the source file). Let's furthermore assume, for simplicity, that a.c declares a single method, whose name is not present in lib.so. I thought maybe it might be possible to, at linking time, link a.o to lib.so while instructing to create newLib.so, and forcing the linker to export all methods/variable in lib.so to that the newLib.so is now basically lib.so with the added method from a.so.
More generally, if one has some source file depending on a shared library, can one create a single output file (library or executable) that is not dependent on the shared library anymore ? (That is, all the relevant methods/variable from the library would have been exported/linked/inlined to the new executable, hence making the dependency void). If that's not possible, what is technically preventing it ?
A somehow similar question has been asked here: Merge multiple .so shared libraries.
One of the reply includes the following text: "If you have access to either source or object files for both libraries, it is straightforward to compile/link a combined SO from them.: without explaining the technical details. Was it a mistake or does it hold ? If so, how to do it ?
Once you have a shared library libfoo.so the only ways you can use it
in the linkage of anything else are:-
Link a program that dynamically depends on it, e.g.
$ gcc -o prog bar.o ... -lfoo
Or, link another shared library that dynamically depends on it, e.g.
$ gcc -shared -o libbar.so bar.o ... -lfoo
In either case the product of the linkage, prog or libbar.so
acquires a dynamic dependency on libfoo.so. This means that prog|libfoo.so
has information inscribed in it by the linker that instructs the
OS loader, at runtime, to find libfoo.so, load it into the
address space of the current process and bind the program's references to libfoo's exported symbols to
the addresses of their definitions.
So libfoo.so must continue to exist as well as prog|libbar.so.
It is not possible to link libfoo.so with prog|libbar.so in
such a way that libfoo.so is physically merged into prog|libbar.so
and is no longer a runtime dependency.
It doesn't matter whether or not you have the source code of the
other linkage input files - bar.o ... - that depend on libfoo.so. The
only kind of linkage you can do with a shared library is dynamic linkage.
This is in complete contrast with the linkage of a static library
You wonder about the statement in this this answer where it says:
If you have access to either source or object files for both libraries, it is straightforward to compile/link a combined SO from them.
The author is just observing that if I have source files
foo_a.c foo_b.c... bar_a.c bar_b.c
which I compile to the corresponding object files:
foo_a.o foo_b.o... bar_a.o bar_b.o...
or if I simply have those object files. Then as well as - or instead of - linking them into two shared libraries:
$ gcc -shared -o libfoo.so foo_a.o foo_b.o...
$ gcc -shared -o libbar.so bar_a.o bar_b.o...
I could link them into one:
$ gcc -shared -o libfoobar.so foo_a.o foo_b.o... bar_a.o bar_b.o...
which would have no dependency on libfoo.so or libbar.so even if they exist.
And although that could be straightforward it could also be false. If there is
any symbol name that is globally defined in any of foo_a.o foo_b.o... and
also globally defined in any of bar_a.o bar_b.o... then it will not matter
to the linkage of either libfoo.so or libbar.so (and it need not be dynamically
exported by either of them). But the linkage of libfoobar.so will fail for
multiple definition of name.
If we build a shared library libbar.so that depends on libfoo.so and has
itself been linked with libfoo.so:
$ gcc -shared -o libbar.so bar.o ... -lfoo
and we then want to link a program with libbar.so, we can do that in such a way
that we don't need to mention its dependency libfoo.so:
$ gcc -o prog main.o ... -lbar -Wl,-rpath=<path/to/libfoo.so>
See this answer to follow that up. But
this doesn't change the fact that libbar.so has a runtime dependency on libfoo.so.
If that's not possible, what is technically preventing it?
What technically prevents linking a shared library with some program
or shared library targ in a way that physically merges it into targ is that a
shared library (like a program) is not the sort of thing that a linker knows
how to physically merge into its output file.
Input files that the linker can physically merge into targ need to
have structural properties that guide the linker in doing that merging. That is the structure of object files.
They consist of named input sections of object code or data that are tagged with various attributes.
Roughly speaking, the linker cuts up the object files into their sections and distributes them into
output sections of the output file according to their attributes, and makes
binary modifications to the merged result to resolve static symbol references
or enable the OS loader to resolve dynamic ones at runtime.
This is not a reversible process. The linker can't consume a program or
shared library and reconstruct the object files from which it was made to
merge them again into something else.
But that's really beside the point. When input files are physically
merged into targ, that is called static linkage.
When input files are just externally referenced in targ to
make the OS loader map them into a process it has launched for targ,
that is called dynamic linkage. Technical development has given us
a file-format solution to each of these needs: object files for static linkage, shared libraries
for dynamic linkage. Neither can be used for the purpose of the other.
I'm creating a little static library for having thread pools, and it depends on 2 other homemade static libraries (a homemade printf and a homemade mini libc).
But sub-functions like ft_bzero are not linked in the project unless I use them on the root project, the one that needs to use thread pools library. So I have the linking error coming from my thpool lib.
Sample :
cc -Wall -Werror -Wextra -MD -I ./ -I ./jqueue -I ../libft/incs -I
../printf/incs -o .objs/thpool_create.o -c ./thpool_create.c
ar rc libthpool.a ./.objs/thpool_create.o etcetc
In the libraries, I compile every .o and use an ar rc libthpool.a *.o. Then I compile .o from main project (a single test.c actually), and then
cc .objs/test.o -o test -L./libft -L./printf -L./thpool -lft -lftprintf -lthpool -lpthread
How can I solve my errors?
Since the code in the ftpool library uses code from ft and ftprintf, you (almost certainly) need to list the libraries in the reverse order:
cc .objs/test.o -o test -L./libft -L./printf -L./thpool -lthpool -lftprintf -lft -lpthread
When scanning a static library, the linker looks for definitions of symbols that are currently undefined. If your test code only calls functions from thpool, then none of the symbols in ft are referenced when the ft library is scanned, so nothing is included from the library; if none of the symbols from ftprintf are referenced when the ftprintf library is scanned, nothing is included from ftprintf either. When it comes across the symbols in thpool that reference things from ft or ftprintf, it's too late; the linker doesn't rescan the libraries. Hence you need to list the libraries in an order such that all references from one library (A) to another (B) are found by linking (A) before (B). If the test code references some of the functions in ft or ftprintf, you may get lucky, or a bit lucky; some symbols may be linked in. But if there are functions in thpool that make the first reference to a function in ft, with the order in the question, you've lost the chance to link everything. Hence the suggested reordering.
Another (very grubby, but nonetheless effective) technique is to rescan the static libraries by listing them several times on the command line.
With shared libraries, the rules of linking are different. If a shared library satisfies any symbol, the whole library will be available, so the linker remembers all the defined symbols, and you might well get away with the original link order.
You might need to look up 'topological sort'. You should certainly aim to design your static libraries so that there are no loops in the dependencies; that leads to cycles of dependencies, and the only reliable solutions are either to rescan the libraries or combine the libraries.
Yeah, I know many people asked that question before, but I still can't understand the problem in my case
I have 2 libs, let's say liba & libb. libb uses liba but is compiled in .a so it should link at compile time.
I have the following GCC command:
gcc -o my_program obj/mymain.o obj/myutils.o liba/liba.a libb/libb.a -Iinclude -Iliba -Ilibb
But GCC is returning me a lot of "Undefined reference to ..." from libb functions to liba functions.
What is happening? What should I do?
Thank you
The evaluation of commands on a link compile command is very important.
When the compiler sees .o files, they get added to the target binary automatically, so all .o files are present. That leaves a list of undefined entities which need to be found.
The next stage is to look through the libraries. Each library is searched, and the .o elements of each library which fulfills an undefined reference is added to the target binary. That always resolves some issues. However, it may also have further requirements. So adding part of a library may add to the required elements to be satisfied.
When a library requires another library, it needs to be specified after something which required it, and before the libraries which satisfy its requirements.
There is a chance if the .o files also require the same parts of a library, this issue can crop up when code is deleted from a .o (removing the mechanism which pulls in the library part).
I understand object ordering is very important during linking. I've had a lot of headache before trying to get ld to resolve all symbols. This time ld didn't generate any error, but the output is wrong!
The project is big (50K+ lines of C++) and I can't generate a simplified version, so I'll try to describe what I encountered. Hopefully some expert can help me figure out.
g++ -o bad.out a.o b.o ... x.so y.so
g++ -o good.out a.o b.o ... y.so x.so
While good.out runs correctly, bad.out does not. Both x.so and y.so are provided by independent vendors, so their ordering should not matter. Here are more clues:
a.o uses x.so
b.o uses y.so
a.o and b.o are independent, but they both use some common classes
The incorrect behavior manifests as a function OnRspLogin() never called back. This is a pure virtual function defined in y.so and implemented in b.o. "grep OnRspLogin *.o *.so" only found match in y.so and b.o.
Apparently ld didn't resolved OnRspLogin() to the one in b.o, but which one did it resolve to? This worries me because linker didn't generate any error or warning.
I'm using gcc 4.4.7-4 on CentOS 6.5.
EDIT:
I found x.so and y.so both contain some common symbols (e.g. T TcpClient), so I guess linker picked x.so:TcpClient (instead of y.so:TcpClient) when resolving b.o:TcpClient. While changing .so order may solve this problem, I'm afraid that linker may incorrectly resolve some other symbols in a.o. So is there anyway to tell the linker to resolve b.o using only y.so? Note that these .so files are provided by 3rd parties and I cannot change them.
I found x.so and y.so both contain some common symbols (e.g. T TcpClient)
That is a problem. If these symbols are supposed to be distinct, then you can't link these two libraries together -- they are not link compatible.
The usual way that vendors resolve these kind of problems is that they use distinct, vendor-specific prefix on all of their exported symbols (e.g. vendorA_TcpClient) and hide all other symbols.
Note that these .so files are provided by 3rd parties and I cannot change them.
You can tell vendors that you can't use their library, unless they avoid defining symbols that aren't prefixed with their unique identifiers, and that you are not going to pay them unless they resolve this problem. Vendors often become quite responsive when do that.
I used this command for compiling my program:
gcc -g -Wall -Werror -lpthread multi_thread_server.c -o multi_thread_socket_v4
It gave undefined reference to <function_name> error.
After of lot of trial and error to fix the error, finally I (by mistake) rearranged the options to gcc as:
gcc multi_thread_server.c -lpthread -Wall -Werror -o multi_thread_server -g
and it worked.
I read this question at SO, I got the reason as why it happened.
Here I want to know that, is there any rule for the sequence of options used for gcc?
P.S.: I know there are infinite options available for gcc, I want to know the sequence according to the category of options.
List libraries last.
After compiling, GCC passes files to the linker for linking (unless linking is not to be performed, as happens when you request compilation-only with the -c switch). It passes the files to the linker in the order you list them (or their corresponding inputs) on the command line.
You listed -lpthread (which means the pthread library, named libpthread.a or something similar) followed by multi_thread_server.c (which gets compiled to an object file named multi_thread_server.o. So the linker receives the library first, then the object file.
When the linker processes a library file, it extracts from it only the modules that supply a definition of a symbol that is needed to satisfy earlier references to the symbol. Since the library is the first file, there are no earlier references. When the linker processes multi_thread_server.o, it sees the references, but it is too late; the linker does not go back to the library.
If you list multi_thread_server.c first, the linker will see multi_thread_server.o first, and it will see that it has unsatisfied referencs. Then, when the linker processes the library, it will find the definitions for those references and will extract those modules from the library.