I am trying to understand the compilation process of a C program. The pre-processed program was given to the compiler (to create obj file). The compiler will check for compilation errors. But somewhere I read that code segment, data segment will be created by the compiler and places the corresponding entries in to those segments. Is this correct?
How will the compiler create the segments in the memory? Since we haven't started running the program. Can anyone please let me know what are the exact things performed by the compiler?
As you mentioned, the text and data segments (and technically the BSS) are generated by the compiler. The text contains program code, the data contains global and static data. Those are all part of your binary image on disk.
The stack and the heap are not created by the compiler, but rather allocated at runtime -- they only exist in memory while the process is still alive.
This is quite simple.
So code segment is for instructions and data segment is for global and static variables.
It's obvious then, that in the end the compiler knows the size of both the code segment and data segment and this exactly the amount of memory required to load your program/library.
It's not actually memory allocation - this will happen at runtime.
But the point is that processor's instruction pointer should not get out of code segment. And this makes the length of code block quite important.
The compiler does not load the program. It only creates the executable file.
The text section and data section is created by the compiler and placed at the right places but only in the executable file. The executable is really just made up of descriptions and instructions to the runtime loader to tell it where to place code and data at run time.
Related
Consider the following picture showing a RAM within which is stored a very simple program divided into instruction block and data block. The example is very similar to the ones found in the book "Code" by Charles Petzold:
As you see there's an instruction block and a data block. In the book this RAM is put inside a rudimentary computer within which you have to put manually both data and instructions by using some switches (just like the old altair 8800). In order for the machine to start executing instructions you had to set the initial address of instruction block and then the machine started executing one instruction after the other sequentially. Basically all this program does is loading the value 1 into accumulator, then add 5 to it, store the result in the address 000Ch (h stands for hex) and finally it stops executing with Halt instruction.
Now when I try to connect the knowledge I got from this book to the way a C source code is compiled I get a bit confused. Specifically the phase in which there's some separation between code segment and data segment. Consider this simple source code:
#include <stdlib.h>
#include <stdio.h>
int test=10;
int main(){
test ++;
return 0;
}
Now my idea is that the compiler should tell the computer to execute machine instructions like this :
int test=10; -> STORE [addressX],10
int main()
{
test++; -> LOAD A,[addressX]
-> INR A
-> STORE [addressX],A
return 0;
}
According to the definition of Wikipedia the data segment "contains initialized static variables, i.e. global variables and local static variables which have a defined value and can be modified".
In my simple example the variable test is a global variable.
However my idea is that before the variable is put inside the data segment of the RAM some sort of machine instructions like STORE must have been invoked. Otherwise how can the global variable be stored inside RAM?
Can someone explain in detail what is really happening and how the simple source code I showed here is really divided into text segment and data segment. What is exactly the text segment for this example? And what about the data segment?
I hope you understood what my doubt is and be able to answer as clear as possible. I appreciate if you could also address me to some good and in deep (with example and not abstract) resources to understand what's really going on when dealing with code, data, stack and heap segments.
I can explain how things work roughly on Windows.
First of all, the given information in the book does not apply that much to modern nowadays OSes. Most OS (such as Windows, Linux, etc.) has an executable file format that describes how the code and data are stored within the file, how they can be mapped into RAM, where to start to execute the code so on. On Windows, the format is called Portable Executable. PE format consists of zero or more sections to store the code and other data. Sections contain some important information such as how the OS will find the data of the section in the file, how to map this data to the memory, what kind of protection method will be used for this data in the memory. Sections can also have a name such as .text, .data, .bss, .idata, .rdata giving a clue about what kind of data the section contains.
When you compile and link your code with MSVC on Windows, you have a portable executable file for your program. This PE file will have one or more sections. For your example, it may have a .text for code, a .data for initialized data, and a .idata section for your imports from other modules. .text section has the compiled machine code, .data section has the data of value 10 for the variable test. When you execute the file, the OS loader will try to load, parse and map it into the memory created for its process.
So, you don't need a STORE instruction to store and initialize the data in RAM. All data in your program is located at the corresponding section and will be mapped into memory by the loader.
When we write a particular code in C, that code gets allocated to either data memory or code memory. When are those memory initialised, at run time or compile time. Any possible explanation of why they are initialised that way?
Most of .code comes from the binary image (as you might expect), the loader does make some changes before the program runs (writing the actual addresses of imported functions to an import table for example). On a system with hardware level memory control the pages that .code is loaded to will be marked read-only (and executable if the hardware gives that level of control).
.data also comes from the program binary, but those pages are marked read-write and non-executable (except for .rodata which is read-only)
.bss does not come from the binary, it simply gets allocated and initialized to 0 (this is the stack lives.
After finishing the load the real program entry point runs (not main), this sets up the environment, runs initializers calls main runs destructors and any final tear-down code the platform needs.
I just learned about different memory segments like: Text, Data, Stack and Heap. My question is:
1- Where the boundaries between these sections are defined? Is it in Compiler or OS?
2- How the compiler or OS know which addresses belong to each section? Should we define it anywhere?
This answer is from the point of view of a more special-purpose embedded system rather than a more general-purpose computing platform running an OS such as Linux.
Where the boundaries between these sections are defined? Is it in Compiler or OS?
Neither the compiler nor the OS do this. It's the linker that determines where the memory sections are located. The compiler generates object files from the source code. The linker uses the linker script file to locate the object files in memory. The linker script (or linker directive) file is a file that is a part of the project and identifies the type, size and address of the various memory types such as ROM and RAM. The linker program uses the information from the linker script file to know where each memory starts. Then the linker locates each type of memory from an object file into an appropriate memory section. For example, code goes in the .text section which is usually located in ROM. Variables go in the .data or .bss section which are located in RAM. The stack and heap also go in RAM. As the linker fills one section it learns the size of that section and can then know where to start the next section. For example, the .bss section may start where the .data section ended.
The size of the stack and heap may be specified in the linker script file or as project options in the IDE.
IDEs for embedded systems typically provide a generic linker script file automatically when you create a project. The generic linker file is suitable for many projects so you may never have to customize it. But as you customize your target hardware and application further you may find that you also need to customize the linker script file. For example, if you add an external ROM or RAM to the board then you'll need to add information about that memory to the linker script so that the linker knows how to locate stuff there.
The linker can generate a map file which describes how each section was located in memory. The map file may not be generated by default and you may need to turn on a build option if you want to review it.
How the compiler or OS know which addresses belong to each section?
Well I don't believe the compiler or OS actually know this information, at least not in the sense that you could query them for the information. The compiler has finished its job before the memory sections are located by the linker so the compiler doesn't know the information. The OS, well how do I explain this? An embedded application may not even use an OS. The OS is just some code that provides services for an application. The OS doesn't know and doesn't care where the boundaries of memory sections are. All that information is already baked into the executable code by the time the OS is running.
Should we define it anywhere?
Look at the linker script (or linker directive) file and read the linker manual. The linker script is input to the linker and provides the rough outlines of memory. The linker locates everything in memory and determines the extent of each section.
For your Query :-
Where the boundaries between these sections are defined? Is it in Compiler or OS?
Answer is OS.
There is no universally common addressing scheme for the layout of the .text segment (executable code), .data segment (variables) and other program segments. However, the layout of the program itself is well-formed according to the system (OS) that will execute the program.
How the compiler or OS know which addresses belong to each section? Should we define it anywhere?
I divided your this question into 3 questions :-
About the text (code) and data sections and their limitation?
Text and Data are prepared by the compiler. The requirement for the compiler is to make sure that they are accessible and pack them in the lower portion of address space. The accessible address space will be limited by the hardware, e.g. if the instruction pointer register is 32-bit, then text address space would be 4 GiB.
About Heap Section and limit? Is it the total available RAM memory?
After text and data, the area above that is the heap. With virtual memory, the heap can practically grow up close to the max address space.
Do the stack and the heap have a static size limit?
The final segment in the process address space is the stack. The stack takes the end segment of the address space and it starts from the end and grows down.
Because the heap grows up and the stack grows down, they basically limit each other. Also, because both type of segments are writeable, it wasn't always a violation for one of them to cross the boundary, so you could have buffer or stack overflow. Now there are mechanism to stop them from happening.
There is a set limit for heap (stack) for each process to start with. This limit can be changed at runtime (using brk()/sbrk()). Basically what happens is when the process needs more heap space and it has run out of allocated space, the standard library will issue the call to the OS. The OS will allocate a page, which usually will be manage by user library for the program to use. I.e. if the program wants 1 KiB, the OS will give additional 4 KiB and the library will give 1 KiB to the program and have 3 KiB left for use when the program ask for more next time.
Most of the time the layout will be Text, Data, Heap (grows up), unallocated space and finally Stack (grows down). They all share the same address space.
The sections are defined by a format which is loosely tied to the OS. For example on Linux you have ELF and on Mac OS you have Mach-O.
You do not define the sections explicitly as a programmer, in 99.9% of cases. The compiler knows what to put where.
I want to dump all global and static variables to a file and load them back on the next program invocation. A solution I thought of is to dump the .data segment to a file. But .data segment on a 32bit machine spans over 2^32 address space (4GB). In which part of this address space the variables reside? How do I know which part of the .data segment I should dump?
And when loading the dumped file, I guess that since the variables are referenced by offset in the data segment, it will be safe to just memcpy the whole dump to the alleged starting point of the "variables area". Please correct me if I am wrong.
EDIT
A good start is this question.
Your problem is how to find the beginning and the end of the data segment. I am not sure how to do this, but I could give you a couple of ideas.
If all your data are relatively self-contained, (they are declared within the same module, not in separate modules,) you might be able to declare them within some kind of structure, so the beginning will be the address of the structure, and the end will be some variable that you will declare right after the structure. If I remember well, MASM had a "RECORD" directive or something like that which you could use to group variables together.
Alternatively, you may be able to declare two additional modules, one with a variable called "beginning" and another with a variable called "end", and make sure that the first gets linked before anything else, and the second gets linked after everything else. This way, these variables might actually end up marking the beginning and the end of the data segment. But I am not sure about this, I am just giving you a pointer.
One thing to remember is that your data will inevitably contain pointers, so saving and loading all your data will only work if the OS under which you are running can guarantee that your program will always be loaded in the same address. If not, forget it. But if you can have this guarantee, then yes, loading the data should work. You should not even need a memcpy, just set the buffer for the read operation to be the beginning of the data segment.
The state of an entire program can be very complicated, and will not only involve variables but values in registers. You'll almost certainly be better off keeping track of what data you want to store and then storing it to a file yourself. This can be relatively painless with the right setup and encapsulation. Then when you resume the application, read in the program state and resume.
Assuming you are using gnu tools (gcc, binutils) if you look at the linker scripts the embedded folks use like the gba developers and microcontroller developers using roms (yagarto or devkit-arm for example). In the linker script they surround the segments of interest with variables that they can use elsewhere in their code. For rom based software for example you specify the data segment with a ram AT rom or rom AT ram in the linker script meaning link as if the data segment is in ram at this address space, but also link the data itself into rom at this address space, the boot code then copies the .data segment from the rom to the ram using these variables. I dont see why you couldnt do the same thing to have the compiler/linker tools tell you where stuff is then runtime use those variables to grab the data from memory and save it somewhere to hybernate or shut down and then restore that data from wherever. The variables you use to perform the restore of course should not be part of the .data segment or you trash the variables you are using to restore the segment.
In response to your header question, on Windows, the location and size of the data and bss segments can be obtained from the in-memory PE header. How that is laid out and how to parse it is documented in this specification:
http://msdn.microsoft.com/en-us/windows/hardware/gg463119
I do not believe that there is a guarantee that with every execution you will have the sam sequence of variables, hence the offsets may point to the wrong content.
How do I inspect in what parts of my memory my heap, stack etc lie? I am currently looking at a program in C, and in looking at the .elf file I can see what memory addresses the program is using, but I don't know if it's in the heap or stack.
That's quite hard to know from a static analysis of the compiled code itself. You should be able to see any static initialized data areas, and also static uninitialized (BSS) sections, but exactly how those are loaded with respect to stack, heap and so on is down to the platform's executable loader.
If you are working in embedded platform , you should probably use some linker scripts(lcf files) along with building the program, then you can identify in detail all the sections(stack,heap,intvec,bss,text,code) ,its placement in the memory (whether in L1 cache,L2 cache or DDR) and its starting/ending address while loading into the board.
The thing is that, please have a look into the linker manual(you can find it in the compiler installation directory) for proper understanding of the keywords in the lcf.
Also there is one more way to analyse the sections, you can create the "map file" for your project and go through it.It will list all sections in the program and its addresses.
you could try using ollydbg, which is a free debugger. the one drawback to this is it shows everything in assembly form, but it will show you what's in your stack, heap, and even what is in your registers. I'm not sure if this is what you are looking for.