I'm trying to understand how C allocates memory to global variables.
I'm working on a simple Kernel. So far it can't do much more than print to screen and enable interrupts. I'm now working on a basic physical memory manager.
My memory manager is a bitmap that sets a 1 or 0 if memory is allocated or available. I need to add the memory that my Kernel is using to the bitmap as 'allocated', so nothing overwrites it.
I can easily find out the start of the Kernel, as it's statically loaded to 0x100000. Figuring out the length shouldn't be too difficult either. The part I'm not sure about is where global variables are put in memory?
Let's say my Kernel is 12K, I can then allocate these 3x 4K blocks of memory to it for protection. Do I need to allocate more to cover the variables it uses? Or are the variables part of that 12K?
Thank you for your help, I hope I am making enough sense.
have a look at
http://www.geeksforgeeks.org/archives/14268
your globals mostly are in the BSS
As the previous answer says, most variables are stored in the .bss section but they can also be stored in the .data or .rodata section depending on if you defined the global variables as static or const. After compiling you can use readelf -S kernel.bin to see exactly how much space each section will utilize. For the .bss section the memory is only occupied when the binary is loaded in memory and does not take any space on disk. This means that your compiled kernel binary will be smaller than the actual size it will later use when brought into memory (by grub usually).
A simple way to figure out exactly how much data your kernel will use besides using readelf is to place the .bss section inside the .data section within your linker script. The size of the kernel binary will then be the same size both on disk as in memory (or actually it will be a bit smaller in memory since not all sections are copied by grub) but then at least you know the minimum amount of memory you need to allocate.
I'd recommend using a custom linker script (assuming you use gcc): it makes the layout of kernel sections explicit and customizable (to read more about linker scripts, read info ld). You can see an example of my OS's linker script here.
To see the default linker script use -v/--verbose option of ld.
Mostly global variables are located in .data.* and .rodata.* sections, variables initialized with 0 go in .bss.
Related
I just learned about different memory segments like: Text, Data, Stack and Heap. My question is:
1- Where the boundaries between these sections are defined? Is it in Compiler or OS?
2- How the compiler or OS know which addresses belong to each section? Should we define it anywhere?
This answer is from the point of view of a more special-purpose embedded system rather than a more general-purpose computing platform running an OS such as Linux.
Where the boundaries between these sections are defined? Is it in Compiler or OS?
Neither the compiler nor the OS do this. It's the linker that determines where the memory sections are located. The compiler generates object files from the source code. The linker uses the linker script file to locate the object files in memory. The linker script (or linker directive) file is a file that is a part of the project and identifies the type, size and address of the various memory types such as ROM and RAM. The linker program uses the information from the linker script file to know where each memory starts. Then the linker locates each type of memory from an object file into an appropriate memory section. For example, code goes in the .text section which is usually located in ROM. Variables go in the .data or .bss section which are located in RAM. The stack and heap also go in RAM. As the linker fills one section it learns the size of that section and can then know where to start the next section. For example, the .bss section may start where the .data section ended.
The size of the stack and heap may be specified in the linker script file or as project options in the IDE.
IDEs for embedded systems typically provide a generic linker script file automatically when you create a project. The generic linker file is suitable for many projects so you may never have to customize it. But as you customize your target hardware and application further you may find that you also need to customize the linker script file. For example, if you add an external ROM or RAM to the board then you'll need to add information about that memory to the linker script so that the linker knows how to locate stuff there.
The linker can generate a map file which describes how each section was located in memory. The map file may not be generated by default and you may need to turn on a build option if you want to review it.
How the compiler or OS know which addresses belong to each section?
Well I don't believe the compiler or OS actually know this information, at least not in the sense that you could query them for the information. The compiler has finished its job before the memory sections are located by the linker so the compiler doesn't know the information. The OS, well how do I explain this? An embedded application may not even use an OS. The OS is just some code that provides services for an application. The OS doesn't know and doesn't care where the boundaries of memory sections are. All that information is already baked into the executable code by the time the OS is running.
Should we define it anywhere?
Look at the linker script (or linker directive) file and read the linker manual. The linker script is input to the linker and provides the rough outlines of memory. The linker locates everything in memory and determines the extent of each section.
For your Query :-
Where the boundaries between these sections are defined? Is it in Compiler or OS?
Answer is OS.
There is no universally common addressing scheme for the layout of the .text segment (executable code), .data segment (variables) and other program segments. However, the layout of the program itself is well-formed according to the system (OS) that will execute the program.
How the compiler or OS know which addresses belong to each section? Should we define it anywhere?
I divided your this question into 3 questions :-
About the text (code) and data sections and their limitation?
Text and Data are prepared by the compiler. The requirement for the compiler is to make sure that they are accessible and pack them in the lower portion of address space. The accessible address space will be limited by the hardware, e.g. if the instruction pointer register is 32-bit, then text address space would be 4 GiB.
About Heap Section and limit? Is it the total available RAM memory?
After text and data, the area above that is the heap. With virtual memory, the heap can practically grow up close to the max address space.
Do the stack and the heap have a static size limit?
The final segment in the process address space is the stack. The stack takes the end segment of the address space and it starts from the end and grows down.
Because the heap grows up and the stack grows down, they basically limit each other. Also, because both type of segments are writeable, it wasn't always a violation for one of them to cross the boundary, so you could have buffer or stack overflow. Now there are mechanism to stop them from happening.
There is a set limit for heap (stack) for each process to start with. This limit can be changed at runtime (using brk()/sbrk()). Basically what happens is when the process needs more heap space and it has run out of allocated space, the standard library will issue the call to the OS. The OS will allocate a page, which usually will be manage by user library for the program to use. I.e. if the program wants 1 KiB, the OS will give additional 4 KiB and the library will give 1 KiB to the program and have 3 KiB left for use when the program ask for more next time.
Most of the time the layout will be Text, Data, Heap (grows up), unallocated space and finally Stack (grows down). They all share the same address space.
The sections are defined by a format which is loosely tied to the OS. For example on Linux you have ELF and on Mac OS you have Mach-O.
You do not define the sections explicitly as a programmer, in 99.9% of cases. The compiler knows what to put where.
In a buffer overflow attack, it's possible to run code from the BSS section (assuming the user disabled some security protections). How is code running there different than code running in the text section? Does it make sense to push things onto the stack while running code from the BSS section? If not, how can functions be called from there?
I'm using linux x86.
As much as i am aware, your premise of the BSS segment containing executable instructions is flawed.
The BSS segment is used to hold only static variables that haven't been assigned values for example:
static char *test_var;
The text segment is the segment that contains the executable instructions and not the BSS segment.
For more clarity refer to:
http://en.wikipedia.org/wiki/.bss
http://en.wikipedia.org/wiki/Code_segment
Also, you might want to look at Virtual Memory layout. The link http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory/ illustrates this very well with diagrams etc.
However, if you want to see which segments of an executable are marked as executable, use this tool called readelf on an executable as shown below:
readelf -l ./test
Yes, you are correct. Provided that the memory segment or selector that holds the BSS is not marked non-executable you can easily execute code from it if:
You know where it is in memory
You have a way to control the EIP to redirect execution here
You have some input (file, actual input, network or environment) that will end up in a statically allocated variable.
Simply inject your code into #3 and you're off to the races.
By the way.. I would not expect BSS to be marked executable, but don't despair. This by no means indicates that some other selector doesn't point at exactly the same memory and is marked executable. This means that you could approach it through BSS to inject code since that will be read/write and then through some other selector to execute.
For example, I find a fair number of examples where CS is pointing to precisely the same memory as DS, but CS is read-only and executable while DS is readwrite and non-executable. Make sense?
I am generating mips disassembly in order to simulating it. I need to have big data to work on it but I don't want to have big assembly files so I wanted to work on a big uninitialized array (and then possibly initialize it in my simulator...). So I need this array to be global. And global variables seem to be put on the .bss section to be initialized when the page is actually accessed.
The problem is in my binary the array is in the .bss section, but is explicitly filled with zero...This is not the behaviour expected if I understood correctly what I have found on internet...Is there a way for saying to the compiler (or linker, or loader...I don't understand well which one do what for that) to not really put zero in this array ?
Or alternatively, can we have an option while compiling, or a C instruction for saying we don't want this array for being initialized with 0 ? (I tried to change the array section with attribute but it is still initialized with 0).
By the way, I am generating my disassembly file with objdump, and it normally skip blocks of zeroes, but I really need the other blocks of zeroes to be disassembled, so I using the "-z" option.
What I really don't understand is that everywhere I looked, it was said that .bss section didn't really put zero in the binary file...
The data for the .bss section isn't stored in the compiled object files because, well, there is no data—the compiler puts variables in that segment precisely because they should be zero-initialized.
When the OS loads the executable, it just looks at the size of the .bss segment, allocates that much memory, and zero-initializes it for you. By not storing that data in the executable file, it reduces loading times.
If you want data to be initialized with certain data, then give it an initializer in your code. The compiler will then put it in the .data segment (initialized data) instead of .bss (uninitialized data). When the OS then loads the executable, it will allocate the memory for the data and then copy it in from the executable. This takes extra I/O, but your data is explicitly initialized how you want it.
Alternatively, you could leave the data stay in the .bss segment and then initialize it yourself at runtime. If the data is quick and easy to generate at runtime, it might be faster to recompute it at startup rather then read it off of disk. But those situations are probably rare.
I suspect that using the -z option is causing objdump to show you zeroes for the .bss, even though the zeroes are not actually in your binary. Try using od -t x4 to get a simple hexadecimal dump of what is really in the binary. If od shows you blocks of zeroes, then they really are in the binary.
I want to dump all global and static variables to a file and load them back on the next program invocation. A solution I thought of is to dump the .data segment to a file. But .data segment on a 32bit machine spans over 2^32 address space (4GB). In which part of this address space the variables reside? How do I know which part of the .data segment I should dump?
And when loading the dumped file, I guess that since the variables are referenced by offset in the data segment, it will be safe to just memcpy the whole dump to the alleged starting point of the "variables area". Please correct me if I am wrong.
EDIT
A good start is this question.
Your problem is how to find the beginning and the end of the data segment. I am not sure how to do this, but I could give you a couple of ideas.
If all your data are relatively self-contained, (they are declared within the same module, not in separate modules,) you might be able to declare them within some kind of structure, so the beginning will be the address of the structure, and the end will be some variable that you will declare right after the structure. If I remember well, MASM had a "RECORD" directive or something like that which you could use to group variables together.
Alternatively, you may be able to declare two additional modules, one with a variable called "beginning" and another with a variable called "end", and make sure that the first gets linked before anything else, and the second gets linked after everything else. This way, these variables might actually end up marking the beginning and the end of the data segment. But I am not sure about this, I am just giving you a pointer.
One thing to remember is that your data will inevitably contain pointers, so saving and loading all your data will only work if the OS under which you are running can guarantee that your program will always be loaded in the same address. If not, forget it. But if you can have this guarantee, then yes, loading the data should work. You should not even need a memcpy, just set the buffer for the read operation to be the beginning of the data segment.
The state of an entire program can be very complicated, and will not only involve variables but values in registers. You'll almost certainly be better off keeping track of what data you want to store and then storing it to a file yourself. This can be relatively painless with the right setup and encapsulation. Then when you resume the application, read in the program state and resume.
Assuming you are using gnu tools (gcc, binutils) if you look at the linker scripts the embedded folks use like the gba developers and microcontroller developers using roms (yagarto or devkit-arm for example). In the linker script they surround the segments of interest with variables that they can use elsewhere in their code. For rom based software for example you specify the data segment with a ram AT rom or rom AT ram in the linker script meaning link as if the data segment is in ram at this address space, but also link the data itself into rom at this address space, the boot code then copies the .data segment from the rom to the ram using these variables. I dont see why you couldnt do the same thing to have the compiler/linker tools tell you where stuff is then runtime use those variables to grab the data from memory and save it somewhere to hybernate or shut down and then restore that data from wherever. The variables you use to perform the restore of course should not be part of the .data segment or you trash the variables you are using to restore the segment.
In response to your header question, on Windows, the location and size of the data and bss segments can be obtained from the in-memory PE header. How that is laid out and how to parse it is documented in this specification:
http://msdn.microsoft.com/en-us/windows/hardware/gg463119
I do not believe that there is a guarantee that with every execution you will have the sam sequence of variables, hence the offsets may point to the wrong content.
I struck a little problem when learning. I know that uninitialized global variables in C are assigned to the .bss section in the executable ELF file. But what happens to them when I start to use them?
I.e. do they get a place on the heap or somewhere else?
I tried to find out by printing the address of the (still uninitialized) global variable with
printf("%x",&glbl);
which always return the same value 0x80495bc... Why?
When the OS loads your program, it allocates enough storage from your program's address space to store everything in the .bss section and zeros all of that memory. When you assign or read from or take the address of the variable, you're manipulating that memory that was allocated to provide storage for the .bss section.
The global variables always get static memory, if they're uninitialized they don't have space in the binary, but they do get it in memory when the binary is loaded to the process memory space.
The BSS is a placeholder defined in your executable (or ELF) format. So it does not take up disk space, but only specifies what memory region should be allocated by the linker or loader.
The exact operation depends on the operating system. Since you refer to ELF, I assume it is for use in an embedded system. If you build for ROMmable code, your linker cmd file will map the BSS to a static address region.
In case you build for an operating system (i.e. Linux), the loader from the operating system will perform a relocation pass, in which it maps all locations marked as relative in the excecutable format to physical or logical locations in memory.
Because you mention always seeing the same value, this indicates that the process is repeatable for your system. Expect to see changes when you change linker files (i.e. address regions), link order (i.e. modules will get assigned space in a different order) or operating system.
Wether or not you use the BSS values, the address will remain the same for the process you run.
That BSS section is given a memory block in the process address space just like the code and stack sections (and any other ELF may have). Once there, they don't go anywhere. The loader arranges things then calls the process entry point.