When linking an application against a dynamic shared library such as in
gcc -o myprog myprog.o -lmylib
I know the linker (ld on my Linux) use the -l option to store in the produced myprog ELF executable file the name of the library (mylib in this case) that will be used at load and link time (both when the program will be started if we ignore lazy dynamic linking). I am wondering what are the other jobs perform by ld (I am only speaking of the static linking step done at compilation time) regarding the dynamic shared library ?
ld must checks for undefined symbol existence in provided dynamic shared libraries
any other stuff ?
Moreover, I will be interested on pointers you are using (books, online documentation) regarding ELF format and dynamic linking and loading processes.
While you hit the most obvious things ld needs to do when linking to ELF shared libraries, there are a few more you missed. I'll re-state the ones you mentioned and add some more:
Ensuring that all undefined symbols are resolved (unless the output is a shared library itself, in which case undefined symbols are valid).
Storing a reference to the library in a DT_NEEDED record of the _DYNAMIC object of the output file.
If the output is not position-independent and references objects (in the sense of data, as opposed to functions) in the shared library, generating a copy relocation to copy the original image of the object into the main program's data segment at load time, and the proper symbol table entry so that references to the object in the shared library itself get resolved to the new copy in the main program, rather than the original copy in the library.
Generating PLT thunks for the destination of each function call in the output that's not resolved at ld-time to a definition in the output.
These are the tasks I can think of that are specific to use of shared libraries, and of course don't include all the work that the linker already does which would be the same as for static linking. One way to think of what ld does with dynamic linking is that it takes object files with a huge repertoire of relocation types (representing anything the compiler or assembler can produce) and resolves all but a small number of them (for static linking, that number would be zero), where all of the remaining relocations fit into a much more limited set of types resolvable by the dynamic linker at load time.
One important step is the creation of a dynamic symbol table, which the runtime linker ld.so can use to link the executable against the library at runtime. It will also write the dynamic relocation table to note which machine code locations need to be changed to point to dynamically linked symbols. To see details:
objdump -T myprog
objdump -R myprog
Also note that the string written to the executable will actually be the SONAME of the library, which might be something like mylib.so.0. This will ensure that even when you install a newer and incompatible mylib.so.1.42 at some later point, the executable will use the compatible ABI version 0 instead. For details:
ldd myprog
Of course, the linker will also link your object files against one another, but since it does that even in the absence of a dynamic shared library, I take it that you are not interested in this part of its operation.
Related
I use static linking to produce the executable object files and I use readelf to check the file and found there is one section called: .rela.plt
the keyword 'rela' indicates that this is related to relocation. but since I use static linking, not using any shared library, so the output executable file should be a fully linked executable file, so why this file still contain relocation information?
There are two ways run-time relocations can end up in statically-linked programs.
The GNU toolchain supports selecting different function implementations at run time using the IFUNC mechanism. On x86-64, these show up as R_X86_64_IRELATIVE relocations.
Some targets support statically linked position independent executables (via -static-pie in the GNU toolchain). Since the the load address differs from program run to program due to address-space layout randomization, any global data object that contains a pointer needs to be relocated at run time. On x86-64, these relocations show up as R_X86_64_RELATIVE.
(There might be other things that need relocations in statically linked programs on more obscure targets.)
I have a .so library and while building it I didn't get any undefined reference errors.
But now I am building an executable using the .so file and I can see the undefined reference errors during the linking stage as shown below:
xy.so: undefined reference to `MICRO_TO_NANO_ULL'
I referred to this and this but couldn't really understand the dynamic linking.
Also reading from here lead to more confusion:
Dynamic linking is accomplished by placing the name of a sharable
library in the executable image. Actual linking with the library
routines does not occur until the image is run, when both the
executable and the library are placed in memory. An advantage of
dynamic linking is that multiple programs can share a single copy of
the library.
My questions are:
Doesn't dynamic linking means that when I start the executable using
./executable_name then if the linker not able to locate the .so
file on which executable depends it should crash?
What actually is dynamic linking if all external entity references are
resolved while building? Is it some sort of pre-check performed by dynamic linker? Else
dynamic linker can make use of
LD_LIBRARY_PATH to get additional libraries to resolve the undefined
symbols.
Doesn't dynamic linking means that when I start the executable using ./executable_name then if the linker not able to locate the .so file on which executable depends it should crash?
No, linker will exit with "No such file or directory" message.
Imagine it like this:
Your executable stores somewhere a list of shared libraries it needs.
Linker, think of it as a normal program.
Linker opens your executable.
Linker reads this list. For each file.
It tries to find this file in linker paths.
If it finds the file, it "loads" it.
If it can't find the file, it get's errno with No Such file or directory from open() call. And then prints a message that it can't find the library and terminates your executable.
When running the executable, linker dynamically searches for a symbol in shared libraries.
When it can't find a symbol, it prints some message and the executable teerminates.
You can for example set LD_DEBUG=all to inspect what linker is doing. You can also inspect your executable under strace to see all the open calls.
What actually is dynamic linking if all external entity references are resolved while
building?
Dynamic linking is when you run the executable then the linker loads each shared library.
When building, your compiler is kind enough to check for you, that all symbols that you use in your program exist in shared libraries. This is just for safety. You can for example disable this check with ex. --unresolved-symbols=ignore-in-shared-libs.
Is it some sort of pre-check performed by dynamic linker?
Yes.
Else dynamic linker can make use of LD_LIBRARY_PATH to get additional libraries to resolve the undefined symbols.
LD_LIBRARY_PATH is just a comma separated list of paths to search for the shared library. Paths in LD_LIBRARY_PATH are just processed before standard paths. That's all. It doesn't get "additional libraries", it gets additional paths to search for the libraries - libraries stay the same.
It looks like there is a #define missing when you compile your shared library. This error
xy.so: undefined reference to `MICRO_TO_NANO_ULL'
means, that something like
#define MICRO_TO_NANO_ULL(sec) ((unsigned long long)sec * 1000)
should be present, but is not.
The compiler assumes then, that it is an external function and creates an (undefined) symbol for it, while it should be resolved at compile time by a preprocessor macro.
If you include the correct file (grep for the macro name) or put an appropriate definition at the top of your source file, then the linker error should vanish.
Doesn't dynamic linking means that when I start the executable using ./executable_name then if the linker not able to locate the .so file on which executable depends it should crash?
Yes. If the .so file is not present at run-time.
What actually is dynamic linking if all external entity references are resolved while building? Is it some sort of pre-check performed by dynamic linker? Else dynamic linker can make use of LD_LIBRARY_PATH to get additional libraries to resolve the undefined symbols.
It allows for libraries to be upgraded and have applications still be able to use the library, and it reduces memory usage by loading one copy of the library instead of one in every application that uses it.
The linker just creates references to these symbols so that the underlying variables or functions can be used later. It does not link the variables and functions directly into the executable.
The dynamic linker does not pull in any libraries unless those libraries are specified in the executable (or by extension any library the executable depends on). If you provide an LD_LIBRARY_PATH directory with a .so file of an entirely different version than what the executable requires the executable can crash.
In your case, it seems as if a required macro definition has not been found and the compiler is using implicit declaration rules. You can easily fix this by compiling your code with -pedantic -pedantic-errors (assuming you're using GCC).
Doesn't dynamic linking means that when I start the executable using
./executable_name then if the linker not able to locate the .so file
on which executable depends it should crash?
It will crash. The time of crash does depend on the way you call a certain exported function from the .so file.
You might retrieve all exported functions via functions pointers by yourself by using dlopen dlysm and co. In this case the program will crash at first call in case it does not find the exported method.
In case of the executable just calling an exported method from a shared object (part of it's header) the dynamic linker uses the information of the method to be called in it's executable (see second answer) and crashes in case of not finding the lib or a mismatch in symbols.
What actually is dynamic linking if all external entity references are resolved while building? Is it some sort of pre-check performed by dynamic linker? Else dynamic linker can make use of LD_LIBRARY_PATH to get additional libraries to resolve the undefined symbols.
You need to differentiate between the actual linking and the dynamic linking. Starting off with the actual linking:
In case of linking a static library, the actual linking will copy all code from the method to be called inside the executable/library using it.
When linking a dynamic library you will not copy code but symbols. The symbols contain offsets or other information pointing to the acual code in the dynamic library. If the executable does invoke a method which is not exported by the dynamic library though, it will already fail at the actual linking part.
Now when starting your executable, the OS will at some point try to load the shared object into memory where the code actually resides in. If it does not find it or also if it is imcotable (i.e.: the executable was linked to a library using different exports), it might still fail at runtime.
new to using C
Header files for libraries like stdlib do not contain the actual implementation code for the functions they provide access to. I understand that the actual source text for libraries like this aren't needed to compile, but how does this work specifically? Are the implementation details for these libraries contained within the compiler?
When you use a function like printf(), including the header file essentially pastes in code for the declaration of the function, but normally the implementation code would need to be available as well.
What form is it stored in? (and where?) Is this compiler specific? Would it be possible to write custom code and reference it in this way without modifying the behavior of the compiler?
I've been searching around and found some info that is relevant but nothing specific. This could be related to not formulating the question well. Thanks.
When you link a program, the compiler will implicitly add some extra libraries to your program:
$ ls
main.c
$ cc -c main.c
$ cc main.o
$ ls
main.c main.o a.out
You can discover the extra libraries a program uses with ldd. Here, there are three libraries linked into the program, and I didn't ask for any of them:
$ ldd a.out
linux-vdso.so => (0x00...)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00...)
/lib64/ld-linux-x86-64.so.2 (0x00...)
So, what happens if we link without these libraries? That's easy enough, just use the linker (ld) directly, instead of calling it through cc. When you use ld, it doesn't give you these extra libraries, so you get an error:
$ ld main.o
Undefined symbols:
"_printf", referenced from:
_main in main.o
The implementation for printf() is stored in the standard C library, which is usually just another library on your system... the only difference is that it gets automatically included into your program when you compile C.
You can use nm to find out what symbols are in a library, so I can use it to find printf() in libc:
$ nm -D /lib/x86_64-linux-gnu/libc-2.13.so | grep printf
...
000000000004e4b0 T printf
...
So, now that we know that libc has printf(), we can use -lc to tell the linker to include libc, and that will get rid of the errors about printf() being missing:
$ ld main.o -lc
There might be some other bits missing, and that's why we use cc to link our programs instead of ld: cc gives us all the default libraries.
When you compile a file you only need to promise the compiler that you have certain functions and symbols. A function call is in the compiled into a call [some_address]
The compiler will compile each C-file into object files that just have place holders for calls to functions declared in the headers. That is [some_address] does not need to be known at this point.
A number of oject files can be collected into what is known as a library.
After that it is the linkers job to look through all object files and libraries it know of and find out what the real value of all unknown [some_address] is and translate the call to, e.g. call 0x1234 if the particular function you are calling starts at 0x1234 (or it might be a relative offset from the current program pointer.
Stdlib and other library functions are implemented in an object library. A library is a collection of code that is linked with your program. By default C programs are linked against the stdlib library, which is usually provided by the operating system. Most modern operating systems use a dynamical linker. That is, your program is not linked against the library until it is executed. When it is being loaded, the linker-loader combines your code and the library code in your program's address space. You code and then make a call to the printf() code that is located in that library.
Usually a header file contains only a function prototype while the implementation is either in a separate source file or a precompiled library in the case of stdlib (and other libraries, both shipped with a compiler or available separately) the precompiled library gets linked at the end of the compilation process. (There's also a distinction between static and dynamic libraries, but I won't go into detail about that)
The implementation of standard libraries (which are shipped with a compiler) are usually compiler specific (there is a standard describing which functions have to be in a library, but the compiler programmer can decide how exactly he implements them) and it is (in theory) possible to exchange these libraries with some of your own without modifying the behaviour of the compiler (though not recommended as you would have to rewrite the entire library in order to ensure that all functions are contained).
I'm getting errors in the lua plugin that I'm writing that are symptomatic of linking in two copies of the lua runtime, as per this message:
http://lua-users.org/lists/lua-l/2008-01/msg00671.html
Quote:
Which in turn means the equality test for dummynode is failing.
This is the usual symptom, if you've linked two copies of the Lua
core into your application (causing two instances of dummynode to
appear).
A common error is to link C extension modules (shared libraries)
with the static library. The linker command line for extension
modules must not ever contain -llua or anything similar!
The Lua core symbols (lua_insert() and so on) are only to be
exported from the executable which contains the Lua core itself.
All C extension modules loaded afterwards can then access these
symbols. Under ELF systems this is what -Wl,-E is for on the
linker line. MACH-O systems don't need this since all non-static
symbols are exported.
This is exactly the error I'm seeing... what I don't know is what I should be doing instead.
I've added the lua src directory to the include path of the DLL that is the c component of my lua plugin, but when I link it I get a pile of errors like:
Creating library file: libmo.dll.a
CMakeFiles/moshared.dir/objects.a(LTools.c.obj): In function `moLTools_dump':
d:/projects/mo-pong/deps/mo/src/mo/lua/LTools.c:38: undefined reference to `lua_gettop'
d:/projects/mo-pong/deps/mo/src/mo/lua/LTools.c:47: undefined reference to `lua_type'
d:/projects/mo-pong/deps/mo/src/mo/lua/LTools.c:48: undefined reference to `lua_typename'
d:/projects/mo-pong/deps/mo/src/mo/lua/LTools.c:49: undefined reference to `lua_tolstring'
So, in summary, I have this situation:
A parent binary that is statically linked to the lua runtime.
A lua library that loads a DLL with C code in it.
The C code in the DLL needs to invoke the lua c api (eg. lua_gettop())
How do I link that? Surely the dynamic library can't 'see' the symbols in the parent binary, because the parent binary isn't loading them from a DLL, they're statically linked.
...but if I link the symbols in as part of the plugin, I get the error above.
Help? This seems like a problem that should turn up a lot (dll depends on symbols in parent binary, how do you link it?) but I can't seem to see any useful threads about it.
(before you ask, no, I dont have control over the parent binary and I cant get it to load the lua symbols from the DLL)
It's probably best to use libtool for this to make your linking easier and more portable. The executable needs to be linked with -export-dynamic to export all the symbols in it, including the Lua symbols from the static library. The module needs to then be linked with -module -shared -avoid-version and, if on Windows, additionall -no-undefined; if on MacOS, additionally -no-undefined -flat_namespace -undefined suppress -bundle; Linux and FreeBSD need no other symbols. This will leave the module with undefined symbols that are satisfied in the parent. If there are any missing, the module will fail to be dlopened by the parent.
The semantics are slightly different for each environment, so it might take some fiddling. Sometimes order of the flags matters. Again, libtool is recommended since it hides much of the inconsistency.
I recently ran nm -m -p -g on the System.B.dylib library from the iOS SDK4.3 and was surprised to find a lot of symbols marked (undefined) (external). Why and when would an undefined symbol be marked external? I can understand a undefined external symbol marked lazy or weak but these aren't. Many of the pthread_xxx functions fall in this category. When I link with this library however, all symbols are resolved. The pthread_xxx symbols are defined in one of the libraries in the \usr\lib\system folder so I am assume they are satisfied from there. How does that work during linking?
It's been a while since I was an nm and ld C-coding ninja, but I think this only means that there are other libraries this one links against.
Usually this is how dynamic linking works. If you were to nm a static archive of System.B, you would not have observed this behavior. The System.B.dylib on it's own would not do much; unless you make it as part of an ensemble set of dynamic and static libraries whose functions it makes use of. If you now try to compile your final binary BUT omit the library path '/usr/lib/system' then you linker will cry foul and exit with an error telling you that it cannot find a reference to pthread_XXX() (using your above example). During the final assembling of the binary, it needs to make sure it knows the location of each and every function used.
HTH