I wrote a small program to add two integers and on using readelf -a executable_name it showed the entry point address in elf header as:
Entry point address: 0x8048330
How does my executable know this address beforehand even before loader loads it in memory?
elf_format.pdf says this member gives the virtual address to which the system first transfers control, thus starting the process. Can anyone please explain what is the meaning of this statement and what is the meaning of virtual address here?
Also let me know, from where the executable file gets the value of 0x8048330 as entry point address. Just for cross check I compiled another program and for that also, the entry point address remains the same value 0x8048330 (offset of .text section being 0x330 in both the cases).
For first question:
the entry point you saw, 0x8048330, is a virtual memory address (in the opposite, is physical memory).
This means your executive doesn't have to know what physical address to map. (after it loads with a loader)
It doesn't even have the access to the physical memory.
To the process of your program, your .text section always starts from 0x8048330, your system (OS and hardware) will then map it (the virtual address) to the physical memory at run-time.
mapping and managing physical memory is a lot of things, you can check on Google for more information.
For the second question
I'm not sure which part confused you so I'll try to cover them all:
Could more than one program have same entry point?
Yes, there could be another program with the same entry point 0x8048330. because this address is virtual, the programs will be mapped to different physical memory at run-time when you try to run them at the same time.
Does the entry always 0x8048330?
Well, Linux executives are start from 0x8048000, but the offset of .text section is related to other sections length. So no, it could be 0x8048034 or anything else.
Why it always start from 0x8048000?
I think it's kind of history thing, the designer of Linux picked this one for some unknown or even random reason. you can refer this thread to see what under that area.
The entry address is set by the link editor, at the time when it creates the
executable. The loader maps the program file at the address(es) specified
by the ELF headers before transferring control to the entry address.
To use a concrete example, consider the following:
% file a.out
a.out: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, \
for GNU/Linux 2.6.15, not stripped
% readelf -e a.out
... snip ...
Elf file type is EXEC (Executable file)
Entry point 0x8048170
There are 6 program headers, starting at offset 52
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x000000 0x08048000 0x08048000 0x7cca6 0x7cca6 R E 0x1000
LOAD 0x07cf98 0x080c5f98 0x080c5f98 0x00788 0x022fc RW 0x1000
... snip ...
The first program header specifies that the contents of the file at
file offset 0 should be mapped to virtual address 0x08048000. The
file and memory sizes for this segment are 0x7cca6 bytes. This
segment is to be mapped in readable and executable but not writable
(it contains the program's code).
The entry point address specified in the ELF header is 0x8048170, which
falls inside the region containing program code.
The book "Linkers and Loaders" by John Levine is a good resource to consult on matters related to link editors and loaders.
About the virtual address question:
Normal userland applications work with virtual addresses which means they don't access directly the memory space. The OS (with the help of some microprocessor's special functions) maps this virtual addresses to physical addresses.
This way, the OS prevents applications from reading/writing into other applications memory or OS reserved memory. Also, this allows the paging of memory (use hard disk as memory) in a transparent way for the application.
Related
I was reading about how programs get loaded into memory. I wanted to understand how different sections (like text,data,rodata etc.) of a PE/ELF file are mapped at different places in the virtual address space of a process. I particularly wanted to understand how does the linker know where will a particular section (say for example rodata) will be mapped onto the address space, so that it can resolve correct addresses for all the symbols before creating the executable.
Do operating systems (eg. Windows) have a fixed range of virtual addresses for a process where they load/map a particular section? If not, then how will the linker resolve the correct addresses for the symbols in different sections?
It doesn't. Linker can only propose the executable image be loaded at certain VA, usually 0x00400000 for PE or 0x10000000 for DLL. Virtual adresses of sections (.text, .rodata, .data etc) are aligned by the section alignment (usually 0x00001000) and their proposed VA are therefore 0x00401000, 0x00402000 etc. Linker then fixes adresses of symbols to those assumed VAs.
The default ImageBase address (taken from linker script or linker arguments) is not required by OS loader, but I don't see a reason to change it, its a good habit to see nice rounded addresses during debugging.
In rare cases the Windows loader may find out that part of the proposed address space is occupied, so it will load the image at a different VA and fix position-dependent (absolute) references to their new VA.
Special PE section relocs contains addresses of references to program symbols which need relocation at load-time.
I'm learning about the layout of executable binaries. My end goal is to analyze a specific executable for things that could be refactored (in its source) to reduce the compiled output size.
I've been using https://www.embeddedrelated.com/showarticle/900.php and https://www.geeksforgeeks.org/memory-layout-of-c-program/ as references for this initial learning.
From what I've learned, a linker script specifies the addresses where sections of compiled binaries are placed. E.g.
> ld --verbose | grep text
PROVIDE (__executable_start = SEGMENT_START("text-segment", 0x400000)); . = SEGMENT_START("text-segment", 0x400000) + SIZEOF_HEADERS;
*(.rela.text .rela.text.* .rela.gnu.linkonce.t.*)
I think this means that the text segments of compiled binaries starts at memory address 0x400000 - true?
What does that value, 0x400000, represent? I'm probably not understanding something properly, but surely that 0x400000 does not represent a physical memory location, does it? E.g. if I were to run two instances of my compiled a.out executable in parallel, they couldn't both simultaneously occupy the space at 0x400000, right?
0x4000000 is not a physical address in the sense how your memory chips see it. This is a virtual address as it's seen from CPU's point of view.
Loader of your program will map a few pages of physical memory to VA 0x400000 and copy the contents of text-segment to it. And yes, another instance of your program could occupy the same physical and virtual block of memory for the text-segment, because text (code) is readable and executable but not writeable. Other segments (data, bss, stack, heap) may have identical VA but each will be mapped to their private protected physical block of memory.
What is 0x400000
I think this means that the text segments of compiled binaries starts at memory address 0x400000 - true?
No, this is well explained in the official documentation at: https://sourceware.org/binutils/docs/ld/Builtin-Functions.html
SEGMENT_START(segment, default)
Return the base address of the named segment. If an explicit value has already been given for this segment (with a command-line ‘-T’ option) then that value will be returned otherwise the value will be default. At present, the ‘-T’ command-line option can only be used to set the base address for the “text”, “data”, and “bss” sections, but you can use SEGMENT_START with any segment name.
Therefore, SEGMENT_START is not setting the address, but rather it is returning it, and 0x4000000 in your case is just the default if that value was not deterministically set by some CLI mechanism mentioned in the documentation (e.g. -Ttext=0x200 as mentioned in man ld)
Physical vs virtual
As you've said, doing things in physical addresses is very uncommon in userland, and would at the very least always require sudo as it would break process separation. Here is an example of userland doing physical address stuff for example: How to access physical addresses from user space in Linux?
Therefore, when the kernel loads an ELF binary with the exec syscalls, all addresses are interpreted as virtual addresses.
Note however that this is just a matter of convention. For example, when I give my Linux kernel ELF binary for QEMU to load into memory to start simulation, or when a bootloader does that in a real system, the ELF addresses would then be treated as physical addresses since there is no page table available at that point.
./hello is a simple echo program in c.
according to objdump file-headers,
$ objdump -f ./hello
./hello: file format elf32-i386
architecture: i386, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x00000430
./hello has start address 0x430
Now loading this binary in gdb.
(gdb) file ./hello
Reading symbols from ./hello...(no debugging symbols found)...done.
(gdb) x/x _start
0x430 <_start>: 0x895eed31
(gdb) break _start
Breakpoint 1 at 0x430
(gdb) run
Starting program: /1/vooks/cdac/ditiss/proj/binaries/temp/hello
Breakpoint 1, 0x00400430 in _start ()
(gdb) x/x _start
0x400430 <_start>: 0x895eed31
(gdb)
in above output before setting breakpoint or running the binary, _start has address 0x430, but after running it, this address changes to 0x400430.
$ readelf -l ./hello | grep LOAD
LOAD 0x000000 0x00000000 0x00000000 0x007b4 0x007b4 R E 0x1000
LOAD 0x000eec 0x00001eec 0x00001eec 0x00130 0x00134 RW 0x1000
How this mapping happens?
Kindly Help.
Basically, after linking, ELF file format provide all necessary information for the loaders to load the program into memory and run it.
Each piece of code and data is placed within an offset inside a section, like data section, text section, etc. and access of specific function or global variable is done by adding the proper offset to the section start address.
Now, ELF file format also include program header table:
An executable or shared object file's program header table is an array
of structures, each describing a segment or other information that the
system needs to prepare the program for execution. An object file
segment contains one or more sections, as described in "Segment
Contents".
Those structures are then used by the OS loader to load the image to memory. The structure:
typedef struct {
Elf32_Word p_type;
Elf32_Off p_offset;
Elf32_Addr p_vaddr;
Elf32_Addr p_paddr;
Elf32_Word p_filesz;
Elf32_Word p_memsz;
Elf32_Word p_flags;
Elf32_Word p_align;
} Elf32_Phdr;
Note the following fields:
p_vaddr
The virtual address at which the first byte of the segment resides in memory
p_offset
The offset from the beginning of the file at which the first byte of
the segment resides.
And p_type
The kind of segment this array element describes or how to interpret the array element's information. Type values and their meanings are specified in Table 7-35.
From Table 7-35, note PT_LOAD:
Specifies a loadable segment, described by p_filesz and p_memsz. The
bytes from the file are mapped to the beginning of the memory segment.
If the segment's memory size (p_memsz) is larger than the file size
(p_filesz), the extra bytes are defined to hold the value 0 and to
follow the segment's initialized area. The file size can not be larger
than the memory size. Loadable segment entries in the program header
table appear in ascending order, sorted on the p_vaddr member.
So, by looking at those fields (and more) the loader can locate the segments (which can contain multiple sections) within the ELF file, and load them (PT_LOAD) into memory at a given virtual address.
Now, can a virtual address of an ELF file segment be changed at runtime (load time)? yes:
The virtual addresses in the program headers might not represent the
actual virtual addresses of the program's memory image. See "Program
Loading (Processor-Specific)".
So, program header contains the segments the OS loader will load into memory (loadable segments, which contains loadable sections), but the virtual addresses the loader puts them can differ from the addresses in the ELF file.
How?
To understand it, lets first read about Base Address
Executable and shared object files have a base address, which is the
lowest virtual address associated with the memory image of the
program's object file. One use of the base address is to relocate the
memory image of the program during dynamic linking.
An executable or shared object file's base address is calculated
during execution from three values: the memory load address, the
maximum page size, and the lowest virtual address of a program's
loadable segment. The virtual addresses in the program headers might
not represent the actual virtual addresses of the program's memory
image. See "Program Loading (Processor-Specific)".
So the practice is the following:
position-independent code. This code enables a segment's virtual
address change from one process to another, without invalidating
execution behavior.
Though the system chooses virtual addresses for individual processes,
it maintains the relative positions of the segments. Because
position-independent code uses relative addressing between segments,
the difference between virtual addresses in memory must match the
difference between virtual addresses in the file.
So by using relative addressing, (PIE- position independent executable) the actual placement can differ from the address in the ELF file.
From PeterCordes's answer:
0x400000 is the Linux default base address for loading PIE
executables with ASLR disabled (like GDB does by default).
So for your specific case (PIE executable in Linux) loader picks this base address.
Of course position independent is just an option. Program can be compiled without it, and than absolute addressing mode takes place, in which there must not be difference between segment address in ELF to the real memory address segment is loaded to:
Executable file segments typically contain absolute
code. For the process to execute correctly, the segments must reside
at the virtual addresses used to create the executable file. The
system uses the p_vaddr values unchanged as virtual addresses.
I would recommend you to take a look at the linux implementation of elf image loading here, and those two SO threads here and here.
Paragraphs takes from Oracle ELF documents (here and here)
You have a PIE executable (Position Independent Executable), so the file only contains an offset relative to the load address, which the OS chooses (and can randomize).
0x400000 is the Linux default base address for loading PIE executables with ASLR disabled (like GDB does by default).
If you compile with -m32 -fno-pie -no-pie hello.c to make a normal position dependent dynamically linked executable that can load from static locations with mov eax, [symname] instead of having to get EIP in a register and use that to do PC-relative addressing without x86-64 RIP-relative addressing modes, objdump -f will say:
./hello-32-nopie: file format elf32-i386
architecture: i386, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x08048380 # hard-coded load address, can't be ASLRed
instead of
architecture: i386, flags 0x00000150: # some different flags set
HAS_SYMS, DYNAMIC, D_PAGED # different ELF type
start address 0x000003e0
In a "regular" position-dependent executable, the linker chooses that base address by default, and does embed it in the executable. The OS's program loader does not get to choose for ELF executables, only for ELF shared objects. A non-PIE executable can't be loaded at any other address, so only their libraries can be ASLRed, not the executable itself. This is why PIE executables were invented.
A non-PIE is allowed to embed absolute addresses without any metadata that would let an OS try to relocate it. Or it's allowed to contain hand-written asm that takes advantage of whatever it wants to about the numeric values of addresses.
A PIE is an ELF shared object with an entry-point. Until PIEs were invented, ELF shared objects were usually only used for shared libraries. See 32-bit absolute addresses no longer allowed in x86-64 Linux? for more about PIEs.
They're quite inefficient for 32-bit code, I'd recommend not making 32-bit PIEs.
A static executable can't be a PIE, so gcc -static will create a non-PIE elf executable; it implies -no-pie. (So will linking with ld directly, because only gcc changed to making PIEs by default, gcc needs to pass -pie to ld to do that.)
So it's easy to understand why you wrote "static vs. dynamic" in your title, if the only dynamic executables you ever looked at were PIEs. But a dynamically linked non-PIE ELF executable is totally normal, and what you should be doing if you care about performance but for some reason want / need to make 32-bit executables.
Until the last couple years or so, normal binaries like /bin/ls in normal Linux distros were non-PIE dynamic executables. For x86-64 code, being PIE only slows them down by maybe 1% I think I've read. Slightly larger code for putting a static address in a register, or for indexing a static array. Nowhere near the amount of overhead that 32-bit code has for PIC/PIE.
When a C program is compiled and the object file(ELF) is created. the object file contains different sections such as bss, data, text and other segments. I understood that these sections of the ELF are part of virtual memory address space. Am I right? Please correct me if I am wrong.
Also, there will be a virtual memory and page table associated with the compiled program. Page table associates the virtual memory address present in ELF to the real physical memory address when loading the program. Is my understanding correct?
I read that in the created ELF file, bss sections just keeps the reference of the uninitialised global variables. Here uninitialised global variable means, the variables that are not intialised during declaration?
Also, I read that the local variables will be allocated space at run time (i.e., in stack). Then how they will be referenced in the object file?
If in the program, there is particular section of code available to allocate memory dynamically. How these variables will be referenced in object file?
I am confused that these different segments of object file (like text, rodata, data, bss, stack and heap) are part of the physical memory (RAM), where all the programs are executed.
But I feel that my understanding is wrong. How are these different segments related to the physical memory when a process or a program is in execution?
1. Correct, the ELF file lays out the absolute or relative locations in the virtual address space of a process that the operating system should copy the ELF file contents into. (The bss is just a location and a size, since its supposed to be all zeros, there is no need to actually have the zeros in the ELF file). Note that locations can be absolute locations (like virtual address 0x100000 or relative locations like 4096 bytes after the end of text.)
2. The virtual memory definition (which is kept in page tables and maps virtual addresses to physical addresses) is not associated with a compiled program, but with a "process" (or "task" or whatever your OS calls it) that represents a running instance of that program. For example, a single ELF file can be loaded into two different processes, at different virtual addresses (if the ELF file is relocatable).
3. The programming language you're using defines which uninitialized state goes in the bss, and which gets explicitly initialized. Note that the bss does not contain "references" to these variables, it is the storage backing those variables.
4. Stack variables are referenced implicitly from the generated code. There is nothing explicit about them (or even the stack) in the ELF file.
5. Like stack references, heap references are implicit in the generated code in the ELF file. (They're all stored in memory created by changing the virtual address space via a call to sbrk or its equivalent.)
The ELF file explains to an OS how to setup a virtual address space for an instance of a program. The different sections describe different needs. For example ".rodata" says I'd like to store read-only data (as opposed to executable code). The ".text" section means executable code. The "bss" is a region used to store state that should be zeroed by the OS. The virtual address space means the program can (optionally) rely on things being where it expects when it starts up. (For example, if it asks for the .bss to be at address 0x4000, then either the OS will refuse to start it, or it will be there.)
Note that these virtual addresses are mapped to physical addresses by the page tables managed by the OS. The instance of the ELF file doesn't need to know any of the details involved in which physical pages are used.
I am not sure if 1, 2 and 3 are correct but I can explain 4 and 5.
4: They are referenced by offset from the top of the stack. When executing a function, the top of the stack is increased to allocate space for local variables. Compiler determines the order of local variables in the stack so the compiler nows what is the offset of the variables from the top of the stack.
Stack in physical memory is positioned upside down. Beginning of stack usually has highest memory address available. As programs runs and allocates space for local variables the address of the top of the stack decrements (and can potentially lead to stack overflow - overlapping with segments on lower addresses :-) )
5: Using pointers - Address of dynamically allocated variable is stored in (local) variable. This corresponds to using pointers in C.
I have found nice explanation here: http://www.ualberta.ca/CNS/RESEARCH/LinuxClusters/mem.html
All the addresses of the different sections (.text, .bss, .data, etc.) you see when you inspect an ELF with the size command:
$ size -A -x my_elf_binary
are virtual addresses. The MMU with the operating system performs the translation from the virtual addresses to the RAM physical addresses.
If you want to know these things, learn about the OS, with source code (www.kernel.org) if possible.
You need to realize that the OS kernel is actually running the CPU and managing the memory resource. And C code is just a light weight script to drive the OS and to run only simple operation with registers.
Virtual memory and Physical memory is about CPU's TLB letting the user space process to use contiguous memory virtually through the power of TLB (using page table) hardware.
So the actual physical memory, mapped to the contiguous virtual memory can be scattered to anywhere on the RAM.
Compiled program doesn't know about this TLB stuff and physical memory address stuff. They are managed in the OS kernel space.
BSS is a section which OS prepares as zero filled memory addresses, because they were not initialized in the c/c++ source code, thus marked as bss by the compiler/linker.
Stack is something prepared only a small amount of memory at first by the OS, and every time function call has been made, address will be pushed down, so that there is more space to place the local variables, and pop when you want to return from the function.
New physical memory will be allocated to the virtual address when the first small amount of memory is full and reached to the bottom, and page fault exception would occur, and the OS kernel will prepare a new physical memory and the user process can continue working.
No magic. In object code, every operation done to the pointer returned from malloc is handled as offsets to the register value returned from malloc function call.
Actually malloc is doing quite complex things. There are various implementations (jemalloc/ptmalloc/dlmalloc/googlemalloc/...) for improving dynamic allocations, but actually they are all getting new memory region from the OS using sbrk or mmap(/dev/zero), which is called anonymous memory.
Just do a man on the command readelf to find out the starting addresses of the different segments of your program.
Regarding the first question you are absolutely right. Since most of today's systems use run-time binding it is only during execution that the actual physical addresses are known. Moreover, it's the compiler and the loader that divide the program into different segments after linking the different libraries during compile and load time. Hence, the virtual addresses.
Coming to the second question it is at the run-time due to runtime binding. The third question is true. All uninitialized global variables and static variables go into BSS. Also note the special case: they go into BSS even if they are initialized to 0.
4.
If you look at a assembler code generated by gcc you can see that memory local variables is allocated in stack through command push or through changing value of the register ESP. Then they are initiated with command mov or something like that.
I am wondering what load-time relocation actually means on a system with virtual memory support.I was thinking that in a system with virtual memory every executable will have addresses starting from zero and at run-time the addresses will be translated into physical addresses using page tables.Therefore the executable can be loaded anywhere in memory without the need of any relocation. However this article on shared libraries mentions that linker specifies an address in the executable where the executable is to be loaded (Entry-point address).
http://eli.thegreenplace.net/2011/08/25/load-time-relocation-of-shared-libraries/
Also there are many articles on dynamic linking which talk about absolute addresses.
Is my understanding wrong ?
Load-time relocation and virtual memory support are two different concepts. Almost all CPUs and OSes these days have virtual memory support. The only really important point to understand about virtual memory is this: forget physical addresses. That is now a hardware and OS responsibility and, unless you are writing a paging system, you can forget about physical addresses. All addresses that a program uses are virtual addresses. This is a huge advantage and immensely simplifies the programming model. On 32-bit systems, this simply means that each process gets its own 4 GiB memory space, ranging from 0x00000000 to 0xffffffff.
An .exe represents a process. A linker produces .exe from .obj files. While both are binary files, .obj files are not executable because they do not contain the addresses of all the variables and functions. It is the job of the linker to provide these addresses, which it determines by placing these .obj files end-to-end and then computing the exact addresses of all the symbols (functions and variables). Thus, the .exe that is created has every address of functions and variables "hard-coded" into it. But there is still one critical information needed before the .exe can be created. The linker has to have insider knowledge about where in memory the .exe will be loaded. Will it be at address 0x00000000, or at 0xffff0000, or somewhere else? For example, in Windows all .exes are always loaded at an absolute starting address of 0x00400000. This is called the base address. When the linker generates the final addresses of symbols (functions and variables), it computes those from this address onward.
Now, .exes rarely need to be loaded at any other address. But the same is not true for .dlls. .ddls are the same as .exes (both are formatted in the portable executable (PE) file format, which describes the memory layout, for example, where text goes, where data goes, and how to find which one). .dlls have a preferred address, too. This simply means that the linker uses this value when it computes the addresses for symbols inside the .dll. If the .dll is loaded at this address, then we are all set.
But if the .dll cannot be loaded at this address (say it was 0x10000000) because some other .dll had already been loaded at this address, then the loader will find some other space in memory and load the .dll there. However, the global addresses of functions and symbols in the .dll are now incorrect. Thus, the loader has to do a relocation (also called "fixup"), in which it adjusts the addresses of all global symbols and functions to reflect their actual addresses.
In order to do this adjustment, the loader needs to be able to find all such symbols in the .dll. The PE file has a .reloc section that contains the internal offset of all such symbols.
Of course, there are other details, for example, regarding how indirection can be used when the compiler generated the code so that, instead of making direct calls, the calls are indirect and variables are accessed via known memory locations in the header of the .exe.
Finally, the gist is this: You need relocation (of some sort) to adjust addresses in the call and jump as well as variable access instructions when the code does not load at the position (within the 4 GiB address space) it was expected to load. When the OS loads a .exe, it has to pick a suitable place in this 4 GiB address space where it will copy the code and data chunks from this .exe on disk.