This is a repeat of a question but I could not quickly find an answer to my question. That's why asking it.
Some ELF files contain(executables or shared libs) program headers which explain segments.
They contain a field called virtual address and file offsets and some other fields.
There are also corresponding sections which explain "address in memory" and file offset.
Now I am little confused how sections and segments are related. (For statically compiled executables and for non-statically compiled executables.)
How file offsets are different for statically compiled binaries? Is there any relation between virtual address in program headers and memory address in section headers.
Thanks
A section is the smallest continuous region of the file. So ELF files are subdivided into sections. Sections cannot overlap, that is, no byte can possibly be part of more than one section. But there can be bytes that don't belong to any section ("garbage").
Sections are generally used for linking purposes. They contain different parts of the file that can be rearranged, merged etc. by the linker.
But executable files can contain sections too – to describe the contents of the file, and where each piece of code or data begins. Shared objects use sections too. Those contain symbol tables for dynamic linking & stuff like that.
All sections contained in the ELF file are described in the Section Headers Table, each section having an entry in it.
But in order to make an executable, you need something else: segments. These tell the loader which parts of the file it should load into memory and to what addresses. So segments map into the executable process's memory space. They can contain code as well as data, so segments can be subdivided into sections to achieve that. And I guess that's the answer to your question.
Loadable segments are described in the Program Headers Table.
Long story short:
In executables, you have segments, that can be subdivided further into sections. Segments are loaded into the process's memory. Sections are optional, but can help subdivide the segments further or describe their contents.
In relocatable modules (compiler outputs, .o files) it's the other way around: sections are required, because they describe what's in the file and allow for linking.
As for the memory addresses & stuff:
On modern systems only the virtual addresses matter. A process is being deceived by the operating system that it runs alone in memory, with the entire address space available to it (though not all the address space could be available at the same time due to physical memory limitations). The system maps virtual addresses to physical addresses on the fly, transparently to the process.
Physical addresses are not used, so they can be left as zeros, but could be set to the same addresses just in case.
Related
I was reading about how programs get loaded into memory. I wanted to understand how different sections (like text,data,rodata etc.) of a PE/ELF file are mapped at different places in the virtual address space of a process. I particularly wanted to understand how does the linker know where will a particular section (say for example rodata) will be mapped onto the address space, so that it can resolve correct addresses for all the symbols before creating the executable.
Do operating systems (eg. Windows) have a fixed range of virtual addresses for a process where they load/map a particular section? If not, then how will the linker resolve the correct addresses for the symbols in different sections?
It doesn't. Linker can only propose the executable image be loaded at certain VA, usually 0x00400000 for PE or 0x10000000 for DLL. Virtual adresses of sections (.text, .rodata, .data etc) are aligned by the section alignment (usually 0x00001000) and their proposed VA are therefore 0x00401000, 0x00402000 etc. Linker then fixes adresses of symbols to those assumed VAs.
The default ImageBase address (taken from linker script or linker arguments) is not required by OS loader, but I don't see a reason to change it, its a good habit to see nice rounded addresses during debugging.
In rare cases the Windows loader may find out that part of the proposed address space is occupied, so it will load the image at a different VA and fix position-dependent (absolute) references to their new VA.
Special PE section relocs contains addresses of references to program symbols which need relocation at load-time.
All globally initialised values are stored in .data segment .i.e. initialised data segment and uninitialised values are stored in bss and compiler initialise those uninitiated values to zero automatically in bss. Then why data segment is separated as .data and bss.
whether it has an advantage or not ? or any benefit
The C programming language (it is a specification written in English) does not know about .bss or .data or data segments. Check that by reading n1570 (the latest draft of C11).
In some cases (e.g. embedded computing) you don't have any data segment. For example when you cross-compile for an Arduino, the resulting code gets uploaded to the flash memory of that microcontroller (and data is in RAM, which your program would perhaps explicitly clear).
(most of my answer below is focused on Linux, but you could adapt it to other OSes)
Regarding the data segment on Unix-like systems like Linux, read more the specification of ELF. It is convenient to avoid spending file space in the executable file for something (the .bss) which is all zeros. The kernel code for execve(2) is able to set up an initial virtual address space with some cleared data (which is not mapped to some file segment). See also mmap(2) & elf(5) & ld-linux(8). Try some cat /proc/$$/maps command to understand the virtual address space of your shell. See proc(5).
So the executable contains segments (some of which are all zeros and don't take any disk space), and is constructed by the linker -which also handle relocations- from several object files provided by the compiler from source code files. On Linux, use objdump(1) and readelf(1) (and nm(1)...) to explore and inspect ELF executables and ELF object files.
BTW a cleared data segment don't need to be fetched from disk by the virtual memory subsystem, and that might make things slightly faster. The kernel would just clear a page in RAM when it is paged in.
So the .bss exist to avoid wasting disk space in executables (and to speedup initializing of zeroed data). Obviously, some mutable data is explicitly initialized to non-zero content (and that needs to sit in .data and takes some disk space in the executable). Constant immutable read-only data goes into .rodata (into the text segment, generally shared by several processes running the same program)
You could configure your linker (e.g. with some linker script) to put all the data (even the cleared ones) in some explicit data segment (but doing so is just wasting disk space)...... Historically, Unix have been developed on machines with little but costly disk space (so wasting it was unthinkable at that time, hence the need of .bss; today you care less!)
Read Levine's book Linkers and Loaders for more, and Advanced Linux Programming and Operating Systems : Three Easy Pieces.
An ELF file for executables has a program (segment) header and a section header, which can be seen through readelf -a, here is an example:
The two pictures above are section header and program (segment) header, respectively. It can be seen that a segment header is composed of several section headers, which is used for loading program into the memory.
Is it only necessary for .text, .rodata, .data, .bss sections to be loaded into the memory?
Are all of the other sections in the segment (e.g. .ctors, .dtors .jcr in the 3rd segment) used for aligning?
Sections and segments are two different concepts completely. Sections pertain the the semantics of the data stored there (i.e. what it will be used for) and are actually irrelevant once a program or shared library is linked except for debugging purposes. You could even remove the section headers entirely (or overwrite them with random garbage) and a program would still work.
Segments (i.e. program header load directives) are what the kernel and/or dynamic linker actually look at when loading a program. For example, in your case you have two load directives. The first one causes the first 4k (1 page) of the file to be mapped at address 0x08048000, and indicates that only the first 0x4b8 bytes of this mapping are actually to be used (the rest is alignment). The second causes the first 8k (2 pages) of the file to be mapped at address 0x08049000. The vast majority of that is alignment. The first 0xf14 bytes are not part of the load directive (just alignment) and will be wasted. Beginning at 0x08049f14, 0x108 bytes mapped from the file are actually used, and another 0x10 bytes (to reach the MemSize of 0x118) are zero-filled by the loader (kernel or dynamic linker). This spans up to 0x0804a02c (in the second mapped page). The rest of the second mapped page is unused/wasted (but malloc might be able to recover it for use as part of the heap).
Finally, while the section headers will not be used at all, the contents of many different sections may be used by your program while it's running. Note that the address ranges of .ctors and .dtors lie in the beginning of the second load mapping, so they are mapped and accessible by the program at runtime (the runtime startup/exit code will use them to run global constructors and destructors, if C++ or "GNU C" code with ctor/dtor attribute was used). Also note that .data starts at address 0x0804a00c, in the second mapped page. This allows the first page to be protected read-only after relocations are applied (the RELRO directive in the program header).
I am wondering what load-time relocation actually means on a system with virtual memory support.I was thinking that in a system with virtual memory every executable will have addresses starting from zero and at run-time the addresses will be translated into physical addresses using page tables.Therefore the executable can be loaded anywhere in memory without the need of any relocation. However this article on shared libraries mentions that linker specifies an address in the executable where the executable is to be loaded (Entry-point address).
http://eli.thegreenplace.net/2011/08/25/load-time-relocation-of-shared-libraries/
Also there are many articles on dynamic linking which talk about absolute addresses.
Is my understanding wrong ?
Load-time relocation and virtual memory support are two different concepts. Almost all CPUs and OSes these days have virtual memory support. The only really important point to understand about virtual memory is this: forget physical addresses. That is now a hardware and OS responsibility and, unless you are writing a paging system, you can forget about physical addresses. All addresses that a program uses are virtual addresses. This is a huge advantage and immensely simplifies the programming model. On 32-bit systems, this simply means that each process gets its own 4 GiB memory space, ranging from 0x00000000 to 0xffffffff.
An .exe represents a process. A linker produces .exe from .obj files. While both are binary files, .obj files are not executable because they do not contain the addresses of all the variables and functions. It is the job of the linker to provide these addresses, which it determines by placing these .obj files end-to-end and then computing the exact addresses of all the symbols (functions and variables). Thus, the .exe that is created has every address of functions and variables "hard-coded" into it. But there is still one critical information needed before the .exe can be created. The linker has to have insider knowledge about where in memory the .exe will be loaded. Will it be at address 0x00000000, or at 0xffff0000, or somewhere else? For example, in Windows all .exes are always loaded at an absolute starting address of 0x00400000. This is called the base address. When the linker generates the final addresses of symbols (functions and variables), it computes those from this address onward.
Now, .exes rarely need to be loaded at any other address. But the same is not true for .dlls. .ddls are the same as .exes (both are formatted in the portable executable (PE) file format, which describes the memory layout, for example, where text goes, where data goes, and how to find which one). .dlls have a preferred address, too. This simply means that the linker uses this value when it computes the addresses for symbols inside the .dll. If the .dll is loaded at this address, then we are all set.
But if the .dll cannot be loaded at this address (say it was 0x10000000) because some other .dll had already been loaded at this address, then the loader will find some other space in memory and load the .dll there. However, the global addresses of functions and symbols in the .dll are now incorrect. Thus, the loader has to do a relocation (also called "fixup"), in which it adjusts the addresses of all global symbols and functions to reflect their actual addresses.
In order to do this adjustment, the loader needs to be able to find all such symbols in the .dll. The PE file has a .reloc section that contains the internal offset of all such symbols.
Of course, there are other details, for example, regarding how indirection can be used when the compiler generated the code so that, instead of making direct calls, the calls are indirect and variables are accessed via known memory locations in the header of the .exe.
Finally, the gist is this: You need relocation (of some sort) to adjust addresses in the call and jump as well as variable access instructions when the code does not load at the position (within the 4 GiB address space) it was expected to load. When the OS loads a .exe, it has to pick a suitable place in this 4 GiB address space where it will copy the code and data chunks from this .exe on disk.
I am writing a simple user-space ELF loader under Linux (why? for 'fun'). My loader at the moment is quite simple and is designed to load only statically-linked ELF files containing position-independent code.
Normally, when a program is loaded by the kernel's ELF loader, it is loaded into its own address space. As such, the data segment and code segment can be loaded at the correct virtual address as specified in the ELF segments.
In my case, however, I am requesting addresses from the kernel via mmap, and may or may not get the addresses requested in the ELF segments. This is not a problem for the code segment since it is position independent. However, if the data segment is not loaded at the expected address, code will not be able to properly reference anything stored in the data segment.
Indeed, my loader appears to work fine with a simple assembly executable that does not contain any data. But as soon as I add a data segment and reference it, the executable fails to run correctly or SEGFAULTs.
How, if possible, can I fixup any references to the data segment to point to the correct place? Is there a relocation section stored in the (static) ELF file for this purpose?
If you modify the absolute addresses available in the .got section, (global offset table) your program should work. Make sure to modify the absolute address calculation to cater for the new distance between .text and .data, I'm afraid you need to figure out where this information comes from, for your architecture.
See this: Global Offset Table (Processor-Specific)
Good luck.
I don't see any way you can do that, unless you emulate the kernel-provided virtual address space completely, and run the code inside that virtual space. When you mmap the data section from the file, you are intrinsically relocating it to an unknown address of the virtual address space of your ELF interpreter, and your code will not be able to reference to it in any way.
Glad to be proven wrong. There's something very cool to learn here.