For this C code:
foobar.c:
static int array[256];
int main() {
return 0;
}
the array is initialized to all 0's, by the C standard. However, when I compile
gcc -S foobar.c
this produces the assembly code foobar.s that I can inspect, and nowhere in foobar.s, is there any initialization of contents of the array.
Hence I reason, that the contents are not initialized, only when an element of the array is inspected, is it initialized, kind of like "copy-on-write" mechanism for fork.
Is my reasoning correct? If so, is this a documented feature, and if so where can I find that documentation?
There's kind of a lot of levels here. This answer addresses Linux in particular, but the same concepts are likely to apply on other systems, possibly with different names.
The compiler requires that the object be "zero initialized". In other words, when a memory read instruction is executed with an address in that range, the value that it reads must be zero. As you say, this is necessary to achieve the behavior dictated by the C standard.
The compiler accomplishes this by asking the assembler to fill the space with zeros, one way or another. It may use the .space or .zero directive which implicitly requests this. It will also place the object in a section with the special name .bss (the reasons for this name are historical). If you look further up in the assembly output, you should see a directive like .bss or .section .bss. The assembler and linker promises that this entire section will be (somehow) initialized to zero. This is documented:
The bss section is used for local common variable storage. You may allocate address space in the bss section, but you may not dictate data to load into it before your program executes. When your program starts running, all the contents of the bss section are zeroed bytes.
Okay, so now what do the assembler and linker do to make it happen? Well, an ELF executable file has a segment header, which specifies how and where code and data from the file should be mapped into the program's memory. (Please note that the use of the word "segment" here has nothing to do with the x86 memory segmentation model or segment registers, and is only vaguely related to the term "segmentation fault".) The size of the segment, and the amount of data to be mapped, are specified separately. If the size is greater, then all remaining bytes are to be initialized to zero. This is also documented in the above-linked man page:
PT_LOAD
The array element specifies a loadable segment,
described by p_filesz and p_memsz. The bytes
from the file are mapped to the beginning of the
memory segment. If the segment's memory size
p_memsz is larger than the file size p_filesz,
the "extra" bytes are defined to hold the value
0 and to follow the segment's initialized area.
So the linker ensures that the ELF executable contains such a segment, and that all objects in the .bss section are in this segment, but not within the part that is mapped to the file.
Once all this is done, then the observable behavior is guaranteed: as above, when an instruction attempts to read from this object before it has been written, the value it reads will be zero.
Now as to how that behavior is ensured at runtime: that is the job of the kernel. It could do it by pre-allocating actual physical memory for that range of virtual addresses, and filling it with zeros. Or by an "allocate on demand" method, like what you describe, by leaving those pages unmapped in the CPU's page tables. Then any access to those pages by the application will cause a page fault, which will be handled by the kernel, which will allocate zero-filled physical memory at that time, and then restart the faulting instruction. This is completely transparent to the application. It just sees that the read instruction got the value zero. If there was a page fault, then it just seems to the application like the read instruction took a long time to execute.
The kernel normally uses the "on demand" method, because it is more efficient in case not all of the "zero initialized" memory is actually used. But this is not going to be documented as guaranteed behavior; it is an implementation detail. An application programmer need not care, and in fact must not care, how it works under the hood. If the Linux kernel maintainers decide tomorrow to switch everything to the pre-allocate method, every application will work exactly as it did before, just maybe a little faster or slower.
Related
Want to get an overview on a few things about how exactly the memory for a variable is allocated.
In C programming,
Taking the context of "auto" variables, which are allocated on the stack section, I have the following question:
Does the compiler generate a logical address for the variables? If yes, then how? Won't the compiler need OS permission to generate or assign such addresses? If no, then is there some sort of indication or instruction that the compiler puts in the code segment asking the OS to allocate memory when running the executable?
Now taking the context of heap allocated variables,
Is the heap of the same size for all programs? If not, then does the executable consist of a header or something that tells the OS how much heap space it needs for dynamic allocation?
I'd be grateful if someone provides the answer or shares any related content/links that explains this.
Stack (most implementations use stack for automatic storage duration objects) and static storage duration objects memory is allocated during the program load and startup.
Does the compiler generate a logical address for the variables? If
yes, then how?
I do not know what is the "logical address" but compilers do "calculate" the references to the automatic storage duration objects. How? Simply compiler knows how far from the stack pointer address the automatic storage duration object is located (offset).
Generally the same applies to the static duration objects and the code, the compiler only calculates the offset from the their sections.
Is the heap of the same size for all programs?
It is implementation defined.
A method typically used in operating systems is that, when a program is starting, there is a piece (or collection) of software used that loads the program. The program loader reads the executable file and sets up memory for the program.
Part of the executable file says what size stack should be allocated for it. Most often, this is set by default when linking the program. (It is 8 MiB for macOS, 2 MiB for Linux, and 1 MiB for Windows.) However, it can be changed by asking the linker to set a different size.
The program loader calls operating system routines to request virtual memory be mapped. It does this for the stack and for other parts of the program, such as the code sections (the parts of memory that contain, mostly, the executable instructions of the program), and the initialized and uninitialized data. When it starts the program, it tells the program where the stack starts by putting that address into a designated register (or similar means).
One of the processor registers is used as a stack pointer; it points to address within the memory allocated for the stack that is the current top of stack. When the compiler arranges to use stack space for objects, it generates instructions that adjust the stack pointer. The addresses for the objects are calculated relative to the stack pointer. If a function needs 128 bytes of data, the compiler generates an instruction that subtracts 128 from the stack pointer. (This may occur in multiple steps, such as “call” and “push” instructions that make some changes to the stack pointer plus an additional “subtract” instruction that finishes the changes.) Then the addresses of all the objects in this stack frame are calculated as offsets from the value of the stack pointer. For example, by taking the stack pointer and adding 40, we calculate the address of the object that has been assigned to be 40 bytes higher than the top of the stack.
(There is some confusion about the wording of directions here because stacks commonly grow from high addresses to low addresses. The program loader may allocate some chunk of memory from, say, address 12300000016 to 12400000016. The stack pointer will start at 12400000016. Subtracting 128 will make it 123FFFF8016. Then 123FFFFA816 is an address that is 40 bytes “higher” than 123FFFF8016 in the address space, but the “top of stack” is below that. That is because the term “top of stack” refers to the model of physically stacking things on top of each other, with the latest thing on top.)
The so-called “heap” is not the same size of all programs. In typical implementations, the memory management routines call system routines to request more virtual memory when they need it.
Note that “heap” is properly a word for a general data structure. Heaps may be used to organize things other than available memory, and the memory management routine keep track of available memory using data structures other than heaps. When referring to memory allocated via the memory management routines, you can call it “dynamically allocated memory.” It may also be shortened to “allocated memory,” but that can be confusing in some situations since all memory that has been reserved for some use is allocated memory.
Some background first
In C programming, Taking the context of "auto" variables, which are allocated on the stack section ...
To understand my answer, you first need to know how the stack works.
Imagine you write the following function:
int myFunction()
{
return function1() + function2() + function3();
}
Unfortunately, you do not use C as programming language but you use a programming language that neither supports local variables nor return values. (This is how most CPUs work internally.)
You may return a value from a function in a global variable:
function1()
{
result = 1234; // instead of: return 1234;
}
And your program may now look the following way if you use a global variable instead of local ones:
int a;
myFunction()
{
function1();
a = result;
function2();
a += result;
function3();
result += a;
}
Unfortunately, one of the three functions (e.g. function3()) may call myFunction() (so the function is called recursively) and the variable a is overwritten when calling function3().
To solve this problem, you may define an array for local variables (myvars[]) and a variable named mypos. In the example, the elements 0...mypos in myvars[] are used; the elements (mypos+1)...(MAX_LOCALS-1) are free:
int myvars[MAX_LOCALS];
int mypos;
...
myFunction()
{
function1();
mypos++;
myvars[mypos] = result;
function2();
myvars[mypos] += result;
function3();
result += myvars[mypos];
mypos--;
}
By changing the value of mypos from 10 to 11 (as an example), your program indicates that the element mypos[11] is now in use and that the functions being called shall store their data in elements mypos[x] with x>=12.
Exactly this is how the stack is working.
Typically, the "variable" mypos is not a variable but a CPU register named "stack pointer". (However, there are a few historic CPUs where an ordinary variable was used for this!)
The actual answers
Does the compiler generate a logical address for the variables?
In the example above, the compiler will perform a mypos+=3 if there are 3 local variables. Let's say they are named a, b and c.
The compiler simply replaces a by myvars[mypos-2], b by myvars[mypos-1] and c by myvars[mypos].
On most CPUs, the stack pointer (named mypos in the example) is not an index into an array but already a pointer (comparable to int * mypos;), so the compiler would replace a by *(mypos-2) instead of myvars[mypos-2] in the example.
For global variables, the compiler simply counts the number of bytes needed for all global variables. In the simplest case, it chooses a range of memory of the same size (e.g. 0x10000...0x10123) and places the variables there.
Won't the compiler need OS permission to generate or assign such addresses?
No.
The "stack" (in the example this is the array myvars[]) is already provided by the OS and the stack pointer (mypos in the example) is also set to a correct value by the OS before the program is started.
Your program knows that the elements myvars[x] with x>mypos can be used.
For global variables, the information about the range used by global variables (e.g. 0x10000...0x10123) is stored in the executable file. The OS must ensure that this memory range can be used by the program. (For example by configuring the MMU accordingly.)
If this is not possible, the OS will simply refuse to start the program with an error message.
... asking the OS to allocate memory when running the executable?
For variables on the stack:
There may be operating systems where this is done.
However, in most cases, the program will simply crash with a "stack overflow" if too much stack is needed. In the example, this would mean: The program crashes if an elements myvars[x] with x>=MAX_LOCALS is accessed.
Now taking the context of heap allocated variables ...
Please first note that global variables are not stored on the heap.
The heap is used for data allocated using malloc() and similar functions (new in C++ for example).
Under many operating systems, malloc() calls an operating system function - so it is actually the operating system that allocates memory on the heap.
... and if there is not enough space, the OS (and malloc()) will return NULL.
Does the compiler generate a logical address for the variables? If yes, then how?
Yes, but they are related to the stack pointer at function entry time (which is normally saved as a constant base pointer, stored in a cpu register) This is because the function can be recursive, and you can have two calls to the function with different instances for that variable (and related to different copies of the base pointer), the compiler assigns the offset to the base pointer for the variable, but the base pointer can be different, depending on the stack contents at function entry time.
Won't the compiler need OS permission to generate or assign such addresses?
Nope, the compiler just generates an executable in the format and form needed for the operating system to manage process' memory. When the program starts, it is given normally three (or more) segments of memory:
text segment. A normally read-only (or execute only) segment that gives no write access to the program. This is normally because the text segment is shared between all programs that are using the same executable at the same time. A program can demand exclusive read-write acces to the text (to allow programs that modify their own executable code) but this happens only rarely. This is normally specified to the compiler and the compiler writes an special flag in the text segment to inform the kernel of this requirement.
Data segment. A read-write segment, that can be grown by means of a system cal (sbrk(2)) This is used for global variables and the heap (while in modern systems, the heap is allocated into a new segment acquired by calling the mmap(2) system call. Sometimes this segment is divided in two. A data segment read-only for constants (so the program receives a signal in case you try to change the value of a constant) and a read-write segment, freely usable by the program. This is where global variables are stored.
Stack segment. A read-write segment, that is allocated for the process to use as the stack segment. It has the capability of growing in one direction as the process starts using it. When the process accesses the data one memory page below the start of the segment, it generates a page fault trap that results in a new page being appended to the segment, so its workings are transparent to the process. This is the memory we are talking about.
the process can ask the kernel explicitly to get a new segment if it wants to (let's say it needs to map some file on memory, or if it has to load a shared executable/library) and on some systems, the read only variables (declared as const) are explititly stored in the text segment or in a specific section called .rodata that demands from the system a special data segment that is read-only. The compiler doesn't normally code this kind of resource itself, it is normally encoded in the program being compiled.
The complete memory is limited by system imposed limits, so if you try to overpass them (around 8Mb of stack space, by default, and depending on the operating system) you will get signalled by the system and your program aborted.
As you see, the process memory is owned by the process, and it can make whatever use it is permitted to. The stack memory is read/write, and allocated on demand, you can use up to 8Mb, but there's no provision to check on the use you do about it.
If no, then is there some sort of indication or instruction that the compiler puts in the code segment asking the OS to allocate memory when running the executable?
The system will know the size of the text segment of the process by the size it has on the executable. The data segment is divided into two parts, normally based on the assumption of what are the global initialized variables and what are the ones defaulting to zero (the memory allocated by the kernel to a process is initialized to zeros for security reasons) so the sum of both the initialized/data and the non initialized data sections are added to know how much memory to assign to the data segment. And the stack segment is assigned initialy just one page of memory, but as the process starts running and filling the stack, it grows as the process generates page faults on the so called next page of the stack segment. As you see, there's no nedd for the compiler to embed in the code any instruction to ask for more memory. Everything is read from the executable file.
The compiler runs as a normal program... it only generates all this information and writes it in a file (the executable file) for the kernel to know the resources needed to run the program. But the compiler's communication with the kernel is just to ask it to open files, write on them, read from source code and struggle it's head to achieve its task. :)
In most POSIX systems, the kernel loads a program in memory by means of the exec*(2) system calls. The kernel reads the executable file pointed to in a parameter of the call and creates the segments above mentioned, based on the parameters passed in the file, checks if another instance of the same program is running in the system to avoid loading the instructions from the file and referencing in this process the segment already open by the other. The data segment contents is initialized to zeros, and the contents of the initialization data are read into the segment (so the first part has the .data section of initialized global variables and the .bss section, which has only a size, is used to calculate the total size of the data segment). Then the stack is normally allocated one or more pages, depending on the initial contents that the exec() calls put in the initial stack. The initial stack is filled with:
a structure of data containing references to the program parameter list that was used on legacy systems to provide the kernel about the command line parameters to show in the ps(1) command output (this is still being generated for legacy purposes, but not used by the kernel for obvious security reasons) Today, a special system call is used to indicate the kernel the command line parameters to be output in the ps(1) output.
a snippet of machine code to use in the return from a system call to allow the execution (in user mode) of any signal handler that should be executed (this is the reason for the requirement that all signal handlers are called when the kernel returns from kernel mode and switches back again to user mode, and not otherwise)
the environment of the process.
the array of pointers to environment strings.
the command line parameters.
the array of char pointers that point to the command line parameters.
the envp array referenct to main().
the argv array reference to the command line parameters.
the argc counter of the number of command line parameters.
Once all these data is pushed to the stack, the program jumps to the start address (fixed by the linker, or by the user by a linker option) and is let to start running.
Before the program jumps to main() the executed code is part of the C runtime, that loads a special shared executable (called /lib/ld.so or similar) that is responsible of searching and loading of all the shared libraries that are linked to the program. Not all the programs have this feature (but almost all of them today are dynamically linked) but IMHO this is out of the scope to this question, as the program has already started and is running.
I have a c program that looks like this
main.c
#include <stdio.h>
#define SOME_VAR 10
static int heap[SOME_VAR];
int main(void) {
printf("%p", heap);
return 0;
}
and outputs this when I run the compiled program a few times
0x58aa7c49060
0x56555644060
0x2f8d1f8e060
0x92f58280060
0x59551c53060
0xd474ed6e060
0x767c4561060
0xf515aeda060
0xbe62367e060
Why does it always end in 060? And is the array stored in heap?
Edit: I am on Linux and I have ASLR on. I compiled the program using gcc
The addresses differ because of ASLR (Address space layout ramdomization). Using this, the binary can be mapped at different locations in the virtual address space.
The variable heap is - in contrast to it's name - not located on the heap, but on the bss. The offset in the address space is therefore constant.
Pages are mapped at page granularity, which is 4096 bytes (hex: 0x1000) on many platforms. This is the reason, why the last three hex digits of the address is the same.
When you did the same with a stack variable, the address could even vary in the last digits on some platforms (namely linux with recent kernels), because the stack is not only mapped somewhere else but also receives a random offset on startup.
If you are using Windows, the reason is PE structure.
Your heap variable is stored in .data section of file and its address is calculated based on start of this section. Each section is loaded in an address independently, but its starting address is multiple of page size. Because you have no other variables, its address is probably start of .data section, so its address will be multiple of chunk size.
For example, this is the table of the compiled Windows version of your code:
The .text section is were your compiled code is and .data contains your heap variable. When your PE is loaded into memory, sections are loaded in different address and which is returned by VirtualAlloc() and will be multiple of page size. But address of each variable is relative to start of section that is now a page size. So you will always see a fixed number on lower digits. Since the relative address of heap from start of section is based on compiler, compile options, etc. you will see different number from same code but different compilers, but every time what will be printed is fixed.
When I compile code, I noticed heap is placed on 0x8B0 bytes after start of .data section. So every time that I run this code, my address end in 0x8B0.
The compiler happened to put heap at offset 0x60 bytes in a data segment it has, possibly because the compiler has some other stuff in the first 0x60 bytes, such as data used by the code that starts the main routine. That is why you see “060”; it is just where it happened to be, and there is no great significance to it.
Address space layout randomization changes the base address(es) used for various parts of program memory, but it always does so in units of 0x1000 bytes (because this avoids causing problems with alignment and other issues). So you see the addresses fluctuate by multiples of 0x1000, but the last three digits do not change.
The definition static int heap[SOME_VAR]; defines heap with static storage duration. Typical C implementations store it in a general data section, not in the heap. The “heap” is a misnomer for memory that is used for dynamic allocation. (It is a misnomer because malloc implementations may use a variety of data structures and algorithms, not limited to heaps. They may even use multiple methods in one implementation.)
I initialize a static variable to 0, but when I see the assembly code, I find that only memory is allocated to the variable. The value is not assigned
And when I initialize the static variable to other numbers, I can find that the memory is assigned a value.
I guess whether GCC thinks the memory should be initialized to 0 by OS before we use the memory.
The GCC option I use is "gcc -m32 -fno-stack-protector -c -o"
When I initialize the static variable to 0, the c code and the assembly code:
static int temp_front=0;
.local temp_front.1909
.comm temp_front.1909,4,4
When I initialize it to other numbers, the code is:
static int temp_front=1;
.align 4
.type temp_front.1909, #object
.size temp_front.1909, 4
temp_front.1909:
.long 1
TL:DR: GCC knows the BSS is guaranteed to be zero-initialized on the platform it's targeting so it puts zero-initialized static data there.
Big picture
The program loader of most modern operating systems gets two different sizes for each part of the program, like the data part. The first size it gets is the size of data stored in the executable file (like a PE/COFF .EXE file on Windows or an ELF executable on Linux), while the second size is the size of the data part in memory while the program is running.
If the data size for the running program is bigger than the amount of data stored in the executable file, the remaining part of the data section is filled with bytes containing zero. In your program, the .comm line tells the linker to reserve 4 bytes without initializing them, so that the OS zero-initializes them on start.
What does gcc do?
gcc (or any other C compiler) allocates zero-initialized variables with static storage duration in the .bss section. Everything allocated in that section will be zero-initialized on program startup. For allocation, it uses the comm directive, and it just specifies the size (4 bytes).
You can see the size of the main section types (code, data, bss) using the size command. If you initialize the variable with one, it is included in a data section, and occupies 4 bytes there. If you initialize it with zero (or not at all), it is instead allocated in the .bss section.
What does ld do?
ld merges all data-type section of all object files (even those from static libraries) into one data section, followed by all .bss-type sections. The executable output contains a simplified view for the operating system's program loader. For ELF files, this is the "program header". You can take a look at it using objdump -p for any format, or readelf for ELF files.
The program headers contain of entries of different type. Among them are a couple of entries with the type PT_LOAD describing the "segments" to be loaded by the operating system. One of these PT_LOAD entries is for the data area (where the .data section is linked). It contains an entry called p_filesz that specifies how many bytes for initialized variables are provided in the ELF file, and an entry called p_memsz telling the loader how much space in the address space should be reserved. The details on which sections get merged into what PT_LOAD entries differ between linkers and depend on command line options, but generally you will find a PT_LOAD entry that describes a region that is both readable and writeable, but not executable, and has a p_filesz value that is smaller than the p_memsz entry (potentially zero if there's only a .bss, no .data section). p_filesz is the size of all read+write data sections, whereas p_memsz is bigger to also provide space for zero-initialized variables.
The amount p_memsz exceeds p_filesz is the sum of all .bss sections linked into the executable. (The values might be off a bit due to alignment to pages or disk blocks)
See chapter 5 in the System V ABI specification, especially pages 5-2 and 5-3 for a description of the program header entries.
What does the operating system do?
The Linux kernel (or another ELF-compliant kernel) iterates over all entries in the program header. For each entry containing the type PT_LOAD it allocates virtual address space. It associates the beginning of that address space with the corresponding region in the executable file, and if the space is writeable, it enables copy-on-write.
If p_memsz exceeds p_filesz, the kernel arranges the remaining address space to be completely zeroed out. So the variable that got allocated in the .bss section by gcc ends up in the "tail" of the read-write PT_LOAD entry in the ELF file, and the kernel provides the zero.
Any whole pages that have no backing data can start out copy-on-write mapped to a shared physical page of zeros.
Why does GCC not assign ...
Most modern OSs will automatically zero-initialize the BSS section.
Using such an OS an "uninitialized" variable is identical to a variable that is initialized to zero.
However, there is one difference: The data of uninitialized variables are not stored in the resulting object and executable files; the data of initialized variables is.
This means that "real" zero-initialized variables may lead to a larger file size compared to uninitialized variables.
For this reason the compiler prefers using "uninitialized" variables if variables are really zero-initialized.
The GCC option I use is ...
Of course there are also operating systems which do not automatically initialize "uninitialized" memory to zero.
As far as I remember Windows 95 is an example for this.
If you want to compile for such an operating system, you may use the GCC command line option -fno-zero-initialized-in-bss. This command line option forces GCC to "really" zero-initialize variables that are zero-initialized.
I just compiled your code with that command line option; the output looks like this:
.data
.align 4
.type temp_front, #object
.size temp_front, 4
temp_front:
.zero 4
There's no point even in Windows 95 to make zero-initialisation in code of every compiled module. May be the Win95 program loader (or even MS-DOS) does not initialize the bss section, but the "ctr0" init module (linked in every comppiled C/C++ program, and that will finally call main() or the DllEntry point, can do that directly in a fast operation for the whole BSS section, whose size is already on the program header and that can also be determined in a static preinitialized variable whose value is computed by the linker, and there's no need to change the way each module is compiled with gcc.
However there are more difficulties about automatic variables (local variables allocated on the stack): the compiler does not know if the variable will be initialized if its first use is by reference in a call parameter (to a non-inlined function, which may be in another module compiled separately or linked from an external library or DLL), supposed to fill it.
GCC only knows when the variable is explicitly assigned in the function itself, but if it gets used by reference only, GCC can now fircibly preinitialize it to zero to prevent it to keep a sensitive value left on the stack. In that case this adds some zero-fill code in the compiled function preamble for these local variables, and this helps prevent some data leaks (generally such leak is unlikely when the varaible is a simple type, but when it is a whole structure, many fields may be left in random state by the subcall.
C11 indicates that such code assuming initialization of auto variables has "undefined" behavior. But GCC will help close the security risk: this is allowed by C11 because this forced zeroing is better to leaving random value and both behaviors are conforming to the "undefined" behavior: zero is as well acceptable as a randomly leaked value.
Some secure functions also avoid leaving senstive data when returning, they explicitly clear the variables they no longer need to avoid expose them after these function return (and notably when they return from a privilege code to an unprivileged one): this is a good practice, but it is independant of the forced initilization of auto variables used by references in subcalls before they were initialized. And GCC is smart enough to not forcibly initialize these auto varaibles when there's explicit code that assign them an explicit value. So the impact is minimal. This feature may be disabled in GCC for those apps that want microoptimizations in terms of performance, but in both cases this does not add to the BSS size, and the image size just grows by only <0.1% for the Linux kernel only because of the few bytes of code compiled in a few functions that benefit of this security fix.
And this has no effect on "uninitialized" static variables, that GCC puts in the BSS section, cleared by the program loader of the OS, or by the small program's crt0 init module.
When a C program is compiled and the object file(ELF) is created. the object file contains different sections such as bss, data, text and other segments. I understood that these sections of the ELF are part of virtual memory address space. Am I right? Please correct me if I am wrong.
Also, there will be a virtual memory and page table associated with the compiled program. Page table associates the virtual memory address present in ELF to the real physical memory address when loading the program. Is my understanding correct?
I read that in the created ELF file, bss sections just keeps the reference of the uninitialised global variables. Here uninitialised global variable means, the variables that are not intialised during declaration?
Also, I read that the local variables will be allocated space at run time (i.e., in stack). Then how they will be referenced in the object file?
If in the program, there is particular section of code available to allocate memory dynamically. How these variables will be referenced in object file?
I am confused that these different segments of object file (like text, rodata, data, bss, stack and heap) are part of the physical memory (RAM), where all the programs are executed.
But I feel that my understanding is wrong. How are these different segments related to the physical memory when a process or a program is in execution?
1. Correct, the ELF file lays out the absolute or relative locations in the virtual address space of a process that the operating system should copy the ELF file contents into. (The bss is just a location and a size, since its supposed to be all zeros, there is no need to actually have the zeros in the ELF file). Note that locations can be absolute locations (like virtual address 0x100000 or relative locations like 4096 bytes after the end of text.)
2. The virtual memory definition (which is kept in page tables and maps virtual addresses to physical addresses) is not associated with a compiled program, but with a "process" (or "task" or whatever your OS calls it) that represents a running instance of that program. For example, a single ELF file can be loaded into two different processes, at different virtual addresses (if the ELF file is relocatable).
3. The programming language you're using defines which uninitialized state goes in the bss, and which gets explicitly initialized. Note that the bss does not contain "references" to these variables, it is the storage backing those variables.
4. Stack variables are referenced implicitly from the generated code. There is nothing explicit about them (or even the stack) in the ELF file.
5. Like stack references, heap references are implicit in the generated code in the ELF file. (They're all stored in memory created by changing the virtual address space via a call to sbrk or its equivalent.)
The ELF file explains to an OS how to setup a virtual address space for an instance of a program. The different sections describe different needs. For example ".rodata" says I'd like to store read-only data (as opposed to executable code). The ".text" section means executable code. The "bss" is a region used to store state that should be zeroed by the OS. The virtual address space means the program can (optionally) rely on things being where it expects when it starts up. (For example, if it asks for the .bss to be at address 0x4000, then either the OS will refuse to start it, or it will be there.)
Note that these virtual addresses are mapped to physical addresses by the page tables managed by the OS. The instance of the ELF file doesn't need to know any of the details involved in which physical pages are used.
I am not sure if 1, 2 and 3 are correct but I can explain 4 and 5.
4: They are referenced by offset from the top of the stack. When executing a function, the top of the stack is increased to allocate space for local variables. Compiler determines the order of local variables in the stack so the compiler nows what is the offset of the variables from the top of the stack.
Stack in physical memory is positioned upside down. Beginning of stack usually has highest memory address available. As programs runs and allocates space for local variables the address of the top of the stack decrements (and can potentially lead to stack overflow - overlapping with segments on lower addresses :-) )
5: Using pointers - Address of dynamically allocated variable is stored in (local) variable. This corresponds to using pointers in C.
I have found nice explanation here: http://www.ualberta.ca/CNS/RESEARCH/LinuxClusters/mem.html
All the addresses of the different sections (.text, .bss, .data, etc.) you see when you inspect an ELF with the size command:
$ size -A -x my_elf_binary
are virtual addresses. The MMU with the operating system performs the translation from the virtual addresses to the RAM physical addresses.
If you want to know these things, learn about the OS, with source code (www.kernel.org) if possible.
You need to realize that the OS kernel is actually running the CPU and managing the memory resource. And C code is just a light weight script to drive the OS and to run only simple operation with registers.
Virtual memory and Physical memory is about CPU's TLB letting the user space process to use contiguous memory virtually through the power of TLB (using page table) hardware.
So the actual physical memory, mapped to the contiguous virtual memory can be scattered to anywhere on the RAM.
Compiled program doesn't know about this TLB stuff and physical memory address stuff. They are managed in the OS kernel space.
BSS is a section which OS prepares as zero filled memory addresses, because they were not initialized in the c/c++ source code, thus marked as bss by the compiler/linker.
Stack is something prepared only a small amount of memory at first by the OS, and every time function call has been made, address will be pushed down, so that there is more space to place the local variables, and pop when you want to return from the function.
New physical memory will be allocated to the virtual address when the first small amount of memory is full and reached to the bottom, and page fault exception would occur, and the OS kernel will prepare a new physical memory and the user process can continue working.
No magic. In object code, every operation done to the pointer returned from malloc is handled as offsets to the register value returned from malloc function call.
Actually malloc is doing quite complex things. There are various implementations (jemalloc/ptmalloc/dlmalloc/googlemalloc/...) for improving dynamic allocations, but actually they are all getting new memory region from the OS using sbrk or mmap(/dev/zero), which is called anonymous memory.
Just do a man on the command readelf to find out the starting addresses of the different segments of your program.
Regarding the first question you are absolutely right. Since most of today's systems use run-time binding it is only during execution that the actual physical addresses are known. Moreover, it's the compiler and the loader that divide the program into different segments after linking the different libraries during compile and load time. Hence, the virtual addresses.
Coming to the second question it is at the run-time due to runtime binding. The third question is true. All uninitialized global variables and static variables go into BSS. Also note the special case: they go into BSS even if they are initialized to 0.
4.
If you look at a assembler code generated by gcc you can see that memory local variables is allocated in stack through command push or through changing value of the register ESP. Then they are initiated with command mov or something like that.
I struck a little problem when learning. I know that uninitialized global variables in C are assigned to the .bss section in the executable ELF file. But what happens to them when I start to use them?
I.e. do they get a place on the heap or somewhere else?
I tried to find out by printing the address of the (still uninitialized) global variable with
printf("%x",&glbl);
which always return the same value 0x80495bc... Why?
When the OS loads your program, it allocates enough storage from your program's address space to store everything in the .bss section and zeros all of that memory. When you assign or read from or take the address of the variable, you're manipulating that memory that was allocated to provide storage for the .bss section.
The global variables always get static memory, if they're uninitialized they don't have space in the binary, but they do get it in memory when the binary is loaded to the process memory space.
The BSS is a placeholder defined in your executable (or ELF) format. So it does not take up disk space, but only specifies what memory region should be allocated by the linker or loader.
The exact operation depends on the operating system. Since you refer to ELF, I assume it is for use in an embedded system. If you build for ROMmable code, your linker cmd file will map the BSS to a static address region.
In case you build for an operating system (i.e. Linux), the loader from the operating system will perform a relocation pass, in which it maps all locations marked as relative in the excecutable format to physical or logical locations in memory.
Because you mention always seeing the same value, this indicates that the process is repeatable for your system. Expect to see changes when you change linker files (i.e. address regions), link order (i.e. modules will get assigned space in a different order) or operating system.
Wether or not you use the BSS values, the address will remain the same for the process you run.
That BSS section is given a memory block in the process address space just like the code and stack sections (and any other ELF may have). Once there, they don't go anywhere. The loader arranges things then calls the process entry point.