Routine is not loaded until it is called. All routines are kept on disk in a re-locatable load format. The main program is loaded into memory & is executed. This is called Dynamic Linking.
Why this is called Dynamic Linking? Shouldn't it be Dynamic Loading because Routine is not loaded until it is called in dynamic loading where as in dynamic linking, Linking postponed until execution time.
This answer assumes that you know basic Linux command.
In Linux, there are two types of libraries: static or shared.
In order to call functions in a static library you need to statically link the library into your executable, resulting in a static binary.
While to call functions in a shared library, you have two options.
First option is dynamic linking, which is commonly used - when compiling your executable you must specify the shared library your program uses, otherwise it won't even compile. When your program starts it's the system's job to open these libraries, which can be listed using the ldd command.
The other option is dynamic loading - when your program runs, it's the program's job to open that library. Such programs are usually linked with libdl, which provides the ability to open a shared library.
Excerpt from Wikipedia:
Dynamic loading is a mechanism by which a computer program can, at run
time, load a library (or other binary) into memory, retrieve the
addresses of functions and variables contained in the library, execute
those functions or access those variables, and unload the library from
memory. It is one of the 3 mechanisms by which a computer program can
use some other software; the other two are static linking and dynamic
linking. Unlike static linking and dynamic linking, dynamic loading
allows a computer program to start up in the absence of these
libraries, to discover available libraries, and to potentially gain
additional functionality.
If you are still in confusion, first read this awesome article: Anatomy of Linux dynamic libraries and build the dynamic loading example to get a feel of it, then come back to this answer.
Here is my output of ldd ./dl:
linux-vdso.so.1 => (0x00007fffe6b94000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f400f1e0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f400ee10000)
/lib64/ld-linux-x86-64.so.2 (0x00007f400f400000)
As you can see, dl is a dynamic executable that depends on libdl, which is dynamically linked by ld.so, the Linux dynamic linker when you run dl. Same is true for the other 3 libraries in the list.
libm doesn't show in this list, because it is used as a dynamically loaded library. It isn't loaded until ld is asked to load it.
Dynamic loading means loading the library (or any other binary for that matter) into the memory during load or run-time.
Dynamic loading can be imagined to be similar to plugins , that is an exe can actually execute before the dynamic loading happens(The dynamic loading for example can be created using LoadLibrary call in C or C++)
Dynamic linking refers to the linking that is done during load or run-time and not when the exe is created.
In case of dynamic linking the linker while creating the exe does minimal work.For the dynamic linker to work it actually has to load the libraries too.Hence it's also called linking loader.
Hence the sentences you refer may make sense but they are still quite ambiguous as we cannot infer the context in which it is referring in.Can you inform us where did you find these lines and at what context is the author talking about?
Dynamic loading refers to mapping (or less often copying) an executable or library into a process's memory after it has started. Dynamic linking refers to resolving symbols - associating their names with addresses or offsets - after compile time.
Here is the link to the full answer by Jeff Darcy at quora
http://www.quora.com/Systems-Programming/What-is-the-exact-difference-between-Dynamic-loading-and-dynamic-linking/answer/Jeff-Darcy
I am also reading the "dinosaur book" and was confused with the loading and linking concept. Here is my understanding:
Both dynamic loading and linking happen at runtime, and load whatever they need into memory.
The key difference is that dynamic loading checks if the routine was loaded by the loader while dynamic linking checks if the routine is in the memory.
Therefore, for dynamic linking, there is only one copy of the library code in the memory, which may be not true for dynamic loading. That's why dynamic linking needs OS support to check the memory of other processes. This feature is very important for language subroutine libraries, which are shared by many programs.
Dynamic linker is a run time program that loads and binds all of the dynamic dependencies of a program before starting to execute that program. Dynamic linker will find what dynamic libraries a program requires, what libraries those libraries require (and so on), then it will load all those libraries and make sure that all references to functions then correctly point to the right place. For example, even the most basic “hello world” program will usually require the C library to display the output and so the dynamic linker will load the C library before loading the hello world program and will make sure that any calls to printf() go to the right code.
Dynamic Loading: Load routine in main memory on call.
Dynamic Linking: Load routine in main memory during execution time,if call happens before execution time it is postponed till execution time.
Dynamic loading does not require special support from Operating system, it is the responsibility of the programmer to check whether the routine that is to be loaded does not exist in main memory.
Dynamic Linking requires special support from operating system, the routine loaded through dynamic linking can be shared across various processes.
Routine is not loaded until it is called. All routines are kept on disk in a re-locatable load format. The main program is loaded into memory & is executed. This is called Dynamic Linking.
The statement is incomplete."The main program is loaded into main memory & is executed." does not specify when the program is loaded.
If we consider that it is loaded on call as 1st statement specifies then its Dynamic Loading
We use dynamic loading to achieve better space utilization
With dynamic loading a program is not loaded until it is called.All routines are kept on a disk in a relocatable load format.The main program is loaded into memory and is executed.
When a routine needs to call another routine, the calling routine first checks to see whether has been loaded.If not , the relocatable linking loader is called to load the desired routine into memory and update program's address tables to reflect this change.Then control is passed to newly loaded routine
Advantages
An unused routine is never loaded .This is most useful when the program code
is large where infrequently occurring cases are needed to handle such as
error routines.In this case although the program code is large ,used code
will be small.
Dynamic loading doesn't need special support from O.S.It is the
responsibility of user to design their program to take advantage of
method.However, O.S can provide libraries to help the programmer
There are two types of Linking Static And Dynamic ,when output file is executed without any dependencies(files=Library) at run time this type of linking is called Static where as Dynamic is of Two types 1.Dynamic Loading Linking 2.Dynamic Runtime Linking.These are Described Below
Dynamic linking refers to linking while runtime where library files are brought to primary memory and linked ..(Irrespective of Function call these are linked).
Dynamic Runtime Linking refers to linking when required,that means whenever there is a function call happening at that time linking During runtime..Not all Functions are linked and this differs in Code writing .
Related
I understand that shared libraries are loaded into memory and used by various programs.
How can a program know where in memory the library is?
When a shared library is used, there are two parts to the linkage process. At compile time, the linker program, ld in Linux, links against the shared library in order to learn which symbols are defined by it. However, none of the code or data initializers from the shared library are actually included in the ultimate a.out file. Instead, ld just records which dynamic libraries were linked against and the information is placed into an auxiliary section of the a.out file.
The second phase takes placed at execution time, before main gets invoked. The kernel loads a small helper program, ld.so, into the address space and this gets executed. Therefore, the start address of the program is not main or even _start (if you have heard of it). Rather, it is actually the start address of the dynamic library loader.
In Linux, the kernel maps the ld.so loader code into a convenient place in the precess address space and sets up the stack so that the list of required shared libraries (and other necessary info) is present. The dynamic loader finds each of the required libraries by looking at a sequence of directories which are often point in the LD_LIBRARY_PATH environment variable. There is also a pre-defined list which is hard-coded into ld.so (and additional search places can be hard-coded into the a.out during link time). For each of the libraries, the dynamic loader reads its header and then uses mmap to create memory regions for the library.
Now for the fun part.
Since the actual libraries used at run-time to satisfy the requirements are not known at link-time, we need to figure out a way to access functions defined in the shared library and global variables that are exported by the shared library (this practice is deprecated since exporting global variables is not thread-safe, but it is still something we try to handle).
Global variables are assigned a statics address at link time and are then accessed by absolute memory address.
For functions exported by the library, the user of the library is going to emit a series of call assembly instructions, which reference an absolute memory address. But, the exact absolute memory address of the referenced function is not known at link time. How do we deal with this?
Well, the linker creates what is known as a Procedure Linkage Table, which is a series of jmp (assembly jump) instructions. The target of the jump is filled in at run time.
Now, when dealing with the dynamic portions of the code (i.e. the .o files that have been compiled with -fpic), there are no absolute memory references whatsoever. In order to access global variables which are also visible to the static portion of the code, another table called the Global Offset Table is used. This table is an array of pointers. At link time, since the absolute memory addresses of the global variables are known, the linker populates this table. Then, at run time, dynamic code is able to access the global variables by first finding the Global Offset Table, then loading the address of the correct variable from the appropriate slot in the table, and finally dereferencing the pointer.
In C, when is dynamic linking between a program and a shared library performed:
Once loading of the program into the memory, but before executing the main() of the program, or
After executing the main() of the program, when the first call to a routine from the library is executed? Will dynamic linking happen again when a second or third or... call to a routine from the library is executed?
I was thinking the first, until I read the following quote, and now I am not sure.
Not sure if OS matters, I am using Linux.
From Operating System Concepts:
With dynamic linking, a stub is included in the image for each
library- routine reference. The stub is a small piece of code that
indicates how to locate the appropriate memory-resident library
routine or how to load the library if the routine is not already
present.
When the stub is executed, it checks to see whether the needed routine is already in memory. If it is not, the program loads the
routine into memory. Either way, the stub replaces itself with the
address of the routine and executes the routine. Thus, the next time
that particular code segment is reached, the library routine is
executed directly, incurring no cost for dynamic linking. Under this
scheme, all processes that use a language library execute only one
copy of the library code.
I was thinking the first, until I read the following quote, and now I am not sure.
It's complicated (and depends on exactly what you call "dynamic linking").
The Linux kernel loads a.out into memory. It then examines PT_INTERP segment (if any).
If that segment is not present, the binary is statically linked and the kernel transfers control to the Elf{32,64}Ehdr.e_entry (usually the _start routine).
If the PT_INTERP segment is present, the kernel loads it into memory, and transfers control to it's .e_entry. It is here that the dynamic linking begins.
The dynamic loader relocates itself, then looks in a.outs PT_DYNAMIC segment for instructions on what else is necessary.
For example, it will usually find one or more DT_NEEDED entries -- shared libraries that a.out was directly linked against. The loader loads any such libraries, initializes them, and resolves any data references between them.
IF a.outs PT_DYNAMIC has a DT_FLAGS entry, and IF that entry contains DF_BIND_NOW flag, then function references from a.out will also be resolved. Otherwise (and assuming that LD_BIND_NOW is not set in the environment), lazy PLT resolution will be performed (resolving functions as part of first call to any given function). Details here.
When the stub is executed, it checks to see whether the needed routine is already in memory. If it is not, the program loads the routine into memory.
I don't know which book you are quoting from, but no current UNIX OS works that way.
The OS (and compiler, etc.) certainly matters: the language itself has nothing to say about dynamic libraries (and very little about linking in general). Even if we know that dynamic linking is occurring, a strictly-conforming program cannot observe any effect from timing among its translation units (since non-local initialization cannot have side effects).
That said, the common toolchains on Linux do support automatic initialization upon loading a dynamic library (for implementing C++, among other things). Executables and the dynamic libraries on which they depend (usually specified with -l) are loaded and initialized recursively to allow initialization in each module to (successfully) use functions from its dependencies. (There is an unfortunate choice of order in some cases.) Of course, dlopen(3) can be used to load and initialize more libraries later.
I was reading the difference between dynamic loading and dynamic linking.
(Dynamic loading vs Dynamic linking.) From there I found this useful answer by Jeff Darcy combination of linking and loading
In the third type i.e, Dynamic loading, dynamic linking where we use dlopen() function to get a handle of that library and try to resolve the symbols. The object file is loaded dynamically under program control (i.e. after start), and symbols both in the calling program and in the library are resolved based on the process's possibly-unique memory layout at that time.
Can someone answer to the following questions:
How dlopen() open is different (advantageous) than other method of linking with shared library. For example: I have seen "libssl" is linked with dlopen().
What advantages we will get it. Is this related to only library version control?
What does he mean by "symbols are resolved based on the process's possibly-unique memory layout at that time"
There are related post here and here.
According to my understanding, static linking directly insert code(what code?machine code?) from library into executables. However, dynamic linking only insert reference(pointer?) point to somewhere in the library.
Then I am wondering why we need two separate version of library of same functionality? For example, for intel MKL, we have libmkl_sequential.a and libmkl_sequential.so. And static linking must link static library, dynamic linking must link dynamic library. Why dynamic linking can not just simply point to static library?
What is the real difference between content of .so and .a of same functionaly?
Code which you want to execute needs to be loaded in memory. A function linked statically becomes a part of your program and so they are both loaded together when the program starts.
Why dynamic linking can not just simply point to static library? Static library is a disk file, how would you want to point inside this? There must be a mechanism (loader & binder) which investigates the starting executable program, asks which functions it wants to use, and loads the corresponding libraries into memory.
Yes, the netto code (instructions) in both versions "libmkl_sequential.a" and "libmkl_sequential.so" may be identical, but static and dynamic types of libraries require different auxilliary metainformation dictated by the library format creator.
Recently I've picked up one of my old projects and restarted it, pretty much from scratch.
I've been sick for awhile, so I've had time to crack down hard and implement tons of functionality. However one thing that I feel would be a good idea to implement is module loading. I want to do kernel mode dynamic loading of modules.
The word modules is a bit ambiguous, the correct term would just be to load libraries, such as a miniture implementation of the C library for kernel mode drivers or standard things like the PIT and keyboard which are on IRQ 0 & 1. The method I'm trying to achieve is a bit self-sustaining; in the aspect that the modules my kernel will load, will be used in the kernel itself to get into user mode.
As an example, my kernel uses very few functions from the C library, which I've implemented myself. These functions themselfs are used in the setup of my GDTs, IDTs, IRQs, ISRs etc, etc. I would like to abstract these functions to a library that the kernel can load and use. Which means the kernel itself will require module loading at the very first stage, before anything is setup.
Now, I've thought of a few ways to do this myself, such as adding a structure to this library with a table of function pointers that are assigned the address of the functions in the library itself. Compiling the library as an aout-kludge file, loading the library into the kernel as a void * ( which is okay since I have a working allocator ), and then figuring out the offset of the structure, stepping into the void pointer that much, and recreating the structure in the kernel. This does not sound like it would work, since the table of function pointers need to be assigned, which means there needs to be an initialize function in the library itself. How would that be called, even if I knew the address?
I'm clueless as to how I could implement such a loader, and is it even worth it? I want to abstract as much as I possibly can, my kernel has a modular design. I also do expect to load drivers and other things with this method, I'm just unsure how I would implement it. I tried various methods already, and they all failed. What should I do?
I would recommend you first write a dynamic loader in user space. The techniques needed are very similar, and you may be able to adapt much of the code to kernel space later. Also, don't use a.out and don't make up your own 'table of function pointers' - use a more modern format such as ELF. The compile-time tools already exist, so this will save you a lot of effort; you can just write an appropriate linker script and build straight from a Linux GCC.
As it happens, the Windows kernel does something very similar to what you say - the Windows kernel (ntoskrnl.exe) is a PE executable file linking in routines from various DLLs (PSHED.dll, HAL.dll, KDCOM.dll, CLFS.sys, and Cl.dll on my system). In this case, the NTLDR program loads all files required by ntoskrnl.exe into memory, and a boot stub in ntoskrnl.exe then performs dynamic linking. Later the same dynamic linker can be used to load other drivers as well.
Implementing kernel modules is not a trivial job. It is a little complicated and you will need to read to ELF documentation for coding. I will try to provide you some insight here -
In user-space, executable files required shared libraries to implement some of their functionality or code. So, the code in the executable will reference code in the shared library. This leads to the development of symbols. Symbols represent pointers to data/functions/other & have a name.
CHAR VariableName[20];
For example, in the above code a data symbol will be created with the name 'VariableName'. After the 'invention' of dynamic/shared libraries the symbol table (set of symbols in a binary) has to be loaded to resolve the references from executable file in the libraries. But a lot of debugging symbols & useless symbols are present in the symbol table.
Symbol: Main.c
For example, in the symbol table, even a symbol for the C source file will be present for debugging. But that is not required for resolving references at runtime. Here, the concept of dynamic symbols comes in the play.
Dynamic linking refers to the resolving of references between binaries. The dynamic linker will use the dynamic symbol table (which must be loaded & 'normal' symbol table doesn't have to) to resolve the references that the executable makes in the library.
Now, in the kernel core which is the executable file, there are no references in the shared libraries (kernel modules). But the shared libraries reference in the executable. So, the executable must contain dynamic symbols for resolving the references in the kernel modules. This is contrary to the case in user-space. Thus, if you are using ld,
-pie -T LinkerScript.ld
option should be used for create a dynamic symbol table in the kernel executable.
And you should create a LinkerScript.ld file -
/* File: LinkerScript.ld */
PHDRS {
kernel PT_LOAD FILEHDR;/* This declares a segment in which your code/data is.*/
dynamic PT_DYNAMIC;/* Segment containing the dynamic table (not DST). */
}
SECTIONS {
/* text, data, bss sections must be implemented already */
.dynamic ALIGN(0x1000) : AT(ADDR(.dynamic) - KERNEL_OFFSET)
{
*(.dynamic)
} :dynamic/* add :kernel to text, data, bss*/
}
with the above structure. Make sure, your .text, .data & .bss sections are already present & :kernel is added to the end of the section descriptor.
For more information, read the ELF documentation & LD manual (for linker script insight).