I wrote a simple code trying to find out if we can read and print the memory in code segment:
#include <stdio.h>
void main() {
int *code_ptr = 0x4;
printf("code_ptr = %x\n", code_ptr);
printf("*code_ptr = %x\n", *code_ptr);
}
My system is x86_64 + Ubuntu 19.04 (Disco Dingo). And the program failed with the following output:
code_ptr = 4
Segmentation fault (core dumped)
IIUC, in Linux, the code segment and data segment share the same base address. And if that's true, this program will read the memory in code segement, and I was expecting that there won't be any crash since 0x04 should be in the range of data segment (which starts at the beginning). And this should pass the paging check since the mapped memory for the code segment is read-only and we only read the memory here.
So did I miss anything or is there any other mechanisms that prevent us from reading from this %ds:0x4?
I think your key misunderstanding is that you're assuming the 8086 hardware feature called "the data segment" is the same as the executable image subdivision also called "the data segment." Xenix may have used that hardware feature that way, but no modern x86 Unix does. On a modern Unix, %ds:0 always points to linear address zero, not to the beginning of the executable's data segment. (And similarly %cs:0 points to linear address zero, not to the executable's text segment.)
All of an executable's segments will be loaded into linear address space somewhere well above linear address 0, and on current-generation OSes the load addresses will be randomized on each run.
There's no standard way to get a pointer to the beginning of the executable's code or data segment. On GNU systems you can use dl_iterate_phdr, and other OSes may have similar functionality under a different name.
Related
For this C code:
foobar.c:
static int array[256];
int main() {
return 0;
}
the array is initialized to all 0's, by the C standard. However, when I compile
gcc -S foobar.c
this produces the assembly code foobar.s that I can inspect, and nowhere in foobar.s, is there any initialization of contents of the array.
Hence I reason, that the contents are not initialized, only when an element of the array is inspected, is it initialized, kind of like "copy-on-write" mechanism for fork.
Is my reasoning correct? If so, is this a documented feature, and if so where can I find that documentation?
There's kind of a lot of levels here. This answer addresses Linux in particular, but the same concepts are likely to apply on other systems, possibly with different names.
The compiler requires that the object be "zero initialized". In other words, when a memory read instruction is executed with an address in that range, the value that it reads must be zero. As you say, this is necessary to achieve the behavior dictated by the C standard.
The compiler accomplishes this by asking the assembler to fill the space with zeros, one way or another. It may use the .space or .zero directive which implicitly requests this. It will also place the object in a section with the special name .bss (the reasons for this name are historical). If you look further up in the assembly output, you should see a directive like .bss or .section .bss. The assembler and linker promises that this entire section will be (somehow) initialized to zero. This is documented:
The bss section is used for local common variable storage. You may allocate address space in the bss section, but you may not dictate data to load into it before your program executes. When your program starts running, all the contents of the bss section are zeroed bytes.
Okay, so now what do the assembler and linker do to make it happen? Well, an ELF executable file has a segment header, which specifies how and where code and data from the file should be mapped into the program's memory. (Please note that the use of the word "segment" here has nothing to do with the x86 memory segmentation model or segment registers, and is only vaguely related to the term "segmentation fault".) The size of the segment, and the amount of data to be mapped, are specified separately. If the size is greater, then all remaining bytes are to be initialized to zero. This is also documented in the above-linked man page:
PT_LOAD
The array element specifies a loadable segment,
described by p_filesz and p_memsz. The bytes
from the file are mapped to the beginning of the
memory segment. If the segment's memory size
p_memsz is larger than the file size p_filesz,
the "extra" bytes are defined to hold the value
0 and to follow the segment's initialized area.
So the linker ensures that the ELF executable contains such a segment, and that all objects in the .bss section are in this segment, but not within the part that is mapped to the file.
Once all this is done, then the observable behavior is guaranteed: as above, when an instruction attempts to read from this object before it has been written, the value it reads will be zero.
Now as to how that behavior is ensured at runtime: that is the job of the kernel. It could do it by pre-allocating actual physical memory for that range of virtual addresses, and filling it with zeros. Or by an "allocate on demand" method, like what you describe, by leaving those pages unmapped in the CPU's page tables. Then any access to those pages by the application will cause a page fault, which will be handled by the kernel, which will allocate zero-filled physical memory at that time, and then restart the faulting instruction. This is completely transparent to the application. It just sees that the read instruction got the value zero. If there was a page fault, then it just seems to the application like the read instruction took a long time to execute.
The kernel normally uses the "on demand" method, because it is more efficient in case not all of the "zero initialized" memory is actually used. But this is not going to be documented as guaranteed behavior; it is an implementation detail. An application programmer need not care, and in fact must not care, how it works under the hood. If the Linux kernel maintainers decide tomorrow to switch everything to the pre-allocate method, every application will work exactly as it did before, just maybe a little faster or slower.
This question already has an answer here:
Why do virtual memory addresses for linux binaries start at 0x8048000?
(1 answer)
Closed 8 years ago.
I am looking into to the memory layout of a given process. I notice that the starting memory location of each process is not 0. On this website, TEXT starts at 0x08048000. One reason can be to distinguish the address with the NULL pointer. I am just wondering if there is any another good reasons? Thanks.
The null pointer doesn't actually have to be 0. It's guaranteed in the C standard that when a 0 value is given in the context of a pointer it's treated as NULL by the compiler.
But the 0 that you use in your source code is just syntactic sugar that has no relation to the actual physical address the null-pointer value is "pointing" to.
For further details see:
Why is NULL/0 an illegal memory location for an object?
Why is address zero used for the null pointer?
An application on your operating system has its unique address space, which it sees as a continuous block of memory (the memory isn't physically continuous, it's just "the impression" the operating system gives to every program).
For the most part, each process's virtual memory space is laid out in a similar and predictable manner (this is the memory layout in a Linux process, 32-bit mode):
(image from Anatomy of a Program in Memory)
Look at the text segment (the default .text base on x86 is 0x08048000, chosen by the default linker script for static binding).
Why the magical 0x08048000? Likely because Linux borrowed that address from the System V i386 ABI.
... and why then did System V use 0x08048000?
The value was chosen to accommodate the stack below the .text section,
growing downward. The 0x48000 bytes could be mapped by the same page
table already required by the .text section (thus saving a page table
in most cases), while the remaining 0x08000000 would allow more room
for stack-hungry applications.
Is there anything below 0x08048000? There could be nothing (it's only 128M), but you can pretty much map anything you desire there, using the mmap() system call.
See also:
What's the memory before 0x08048000 used for in 32 bit machine?
Reorganizing the address space
mmap
I think this sums it up:
Each process has its own set of page tables, but there is a catch. Once virtual addresses are enabled, they apply to all software running in the machine, including the kernel itself. Thus a portion of the virtual address space must be reserved to the kernel.
So while the process gets it's own address space. Without allocating a block to the kernel, it would not be able to address kernel code and data.
This is always the first block of memory it appears and so includes address 0. The user mode space starts beyond this, and so that is where both the stack and heap reside.
Distinguishing from NULL pointer
Even if the user mode space started at address 0, there would not be any data allocated to the address 0 as that will be in the stack or the heap which themselves do not start at the beginning of the user area. Therefore NULL (with the value of 0) could be used still and is not a reason for this layout.
However one benefit related to the NULL and the first block being kernel memory is any attempt to read/write to NULL throws a Segmentation Fault.
A loader loads a binary in segments into memory: text (constants), data, code. There is no need to start from 0, and as C is has the problem from bugs accessing around null, like in a[i] that is even dangerous. This allows (on some processors) to intercept segmentation faults.
It would be the C runtime introducing a linear address space from 0. That might be imaginable where C is the operating system's implementation language. But serves no purpose; to have the heap start from 0. The memory model is one of segments. A code segment might be protected against modification by some processors.
And in segments allocation happens in C runtime managed memory blocks.
I might add, that physical 0 and upwards is often used by the operating system itself.
So I'm working with the linux 0.11 kernel on a virtual machine, and I need to write a program that analyses executable files that are ran on that kernel. The files are in the format a.out. What I want to know is, how does the operating system decide where to load the file in (virtual?) memory? Is it decided by something called "base address", and if so, how come I can't seem to find any mention of it in the a.out header?
//where is base address?
struct exec {
unsigned long a_magic; /* Use macros N_MAGIC, etc for access */
unsigned a_text; /* length of text, in bytes */
unsigned a_data; /* length of data, in bytes */
unsigned a_bss; /* length of uninitialized data area for file, in bytes */
unsigned a_syms; /* length of symbol table data in file, in bytes */
unsigned a_entry; /* start address */
unsigned a_trsize; /* length of relocation info for text, in bytes */
unsigned a_drsize; /* length of relocation info for data, in bytes */
};
I tried looking for documentations about the format, but the only information I found just explains what each of these fields are, what values a_magic can have, etc.
I need to know about it because the program needs to print out file and line numbers when given an address in memory of an instruction in the executable, and the debug symbols only have their addresses as offsets (e.g. relative to the start of text section, etc).
Also, out of curiosity, I know that in C, "(void*)0" is NULL, which you can't dereference. How then would you get the content of memory address 0?
As you see, I know very little about linux kernel and operating systems in general, so please start from the basics...
I appreciate any help you can give, thanks.
The base address is the a_entry field.
Also, out of curiosity, I know that in C, "(void*)0" is NULL, which you can't dereference. How then would you get the content of memory address 0?
Any system that puts memory usable by a C program at address zero would have to make it work, somehow. While one can imagine possible ways to do this, I don't know of anyone who bothers. Virtual address zero is, for all intents and purposes, never used.
The operating system can load the application at any location it chooses and then relocate the embedded addresses to be relative to that point. This relocation information is recorded in the a.out file. The base address depends on the architecture and other details and is often non-zero.
If you look at a linker map file, you should see a symbol that is either at the beginning of the memory image, or at a fixed offset from it. At runtime, subtract this value from the actual addresses you note for debugging to get to the relative address of the position you are interested in.
Note, the symbols will not be present in the executable if your linker script strips them.
Also, out of curiosity, I know that in C, "(void*)0" is NULL, which you can't dereference. How then would you get the content of memory address 0?
Actually you can dereference NULL, but the results are not defined. For convenience, most operating systems trap the access to help you debug pointer problems.
Also, memory location with address 0 in a process space is different from the memory location with address 0 in the 'hardware space'. The 'pagination' support in the CPU and operating system are 'decoupling' the physical memory from the virtual memory. It could happen that a virtual page be mapped at address 0, although there you usually have interrupt vectors and other special device memory and not real RAM anyway.
I have a couple of doubts, I remember some where that it is not possible for me to manually put a variable in a particular location in memory, but then I came across this code
#include<stdio.h>
void main()
{
int *x;
x=0x200;
printf("Number is %lu",x); // Checkpoint1
scanf("%d",x);
printf("%d",*x);
}
Is it that we can not put it in a particular location, or we should not put it in a particular location since we will not know if it's a valid location or not?
Also, in this code, till the first checkopoint, I get output to be 512.
And then after that Seg Fault.
Can someone explain why? Is 0x200 not a valid memory location?
In the general case - the behavior you will get is undefined - everything can happen.
In linux for example, the first 1GB is reserved for kernel, so if you try to access it - you will get a seg fault because you are trying to access a kernel memory in user mode.
No idea how it works in windows.
Reference for linux claim:
Currently the 32 bit x86 architecture is the most popular type of
computer. In this architecture, traditionally the Linux kernel has
split the 4GB of virtual memory address space into 3GB for user
programs and 1GB for the kernel.
Adding to what #amit wrote:
In windows it is the same. In general it is the same for all protected-mode operating systems. Since DOS etc. are no longer around it is the same with all systems except kernel-mode (km-drivers) and embedded systems.
The operating system manages which memory-pages you are allowed to write to and places markers that will make the cpu automatically raise access-violations if some other page is written to.
Up until the "checkpoint", you haven't accessed memory location 0x200, so everything works fine.
There I'd a local variable x in the function main. It is of type "pointer to int". x is assigned the value 0x200, and then that value is printed. But the target of x hasn't been accessed, so up to this point it doesn't matter whether x holds a valid memory address or not.
Then scanf tries to write to the memory address you passed in, which is the 0x200 stored in x. Then you get a seg fault, which is certainly sac possible result of trying to write to an arbitrary memory address.
So what are your doubts? What makes you think that this might work, when you come across this code that clearly doesn't?
Writing to a particular memory address might work under certain conditions, but is extremely unlikely to in general. Under all modern OSes, normal programs do not have control over their memory layout. The OS decides where initial things like the program's code, stack, and globals go. The OS will probably also be using some memory space, and it is not required to tell you what it's using. Instead you ask for memory (either by making variables or by calling memory allocation routines), and you use that.
So writing to particular addresses is very very likely to get either memory that hasn't been allocated, or memory that is being used for some other purpose. Neither of those is good, even if you do manage to hit an address that is actually writable. What if you clobber sundry some piece of data used by one of your program's other variables? Or some other part of your program clobbers the value you just wrote?
You should never be choosing a particular hard-coded memory address, you should be using an address of something you know is a variable, or an address you got from something like malloc.
I wanted know how the kernel is providing memory for simple C program .
For example :
#include<stdio.h>
#include<malloc.h>
int my_global = 10 ;
main()
{
char *str ;
static int val ;
str = ( char *) malloc ( 100 ) ;
scanf ( "%s" , str ) ;
printf( " val:%s\n",str ) ;
free(str) ;
return 1 ;
}
See, In this program I have used static , global and malloc for allocating dynamic memory
So , how the memory lay out will be ... ?
Any one give me url , which will have have details information about this process..
Very basically, in C programs built to target ELF (Executable and Linkable Format) such as those built on linux there is a standard memory layout that is created. Similar layouts probably exist for other architectures, but I don't know enough to tell you more about them.
The Layout:
There are some global data sections that are initialized at low memory addresses in memory (such as sections for the currently executing code, global data, and any strings that are created with "..." inside your C code).
Below that there is a heap of open memory that can be used. The size of this heap increases automatically as calls to malloc and free move what is called the "program break" to higher addresses in memory.
Starting at a high address in memory, the stack grows towards lower addresses. The stack contains memory for any locally allocated variables, such as those at the top of functions or within a scope ({ ... }).
More Info:
There is a good description of a running ELF program here and more details on the format itself on the Wikipedia article. If you want an example of how a compiler goes about translating C code into assembly you might look at GCC, their Internals Manual has some interesting stuff in it; the most relevant sections are probably those in chapter 17, especially 17.10, 17.19 and 17.21. Finally, Intel has a lot of information about memory layout in its IA-32 Architectures Software Developer’s Manual. It describes how Intel processors handle memory segmentation and the creation of stacks and the like. There's no detail about ELF, but it's possible to see where the two match up. The most useful bits are probably section 3.3 of Volume 1: Basic Architecture, and chapter 3 of Volume 3A: System Programming Guide, Part 1.
I hope this helps anyone diving into the internals of running C programs, good luck.
There's a brief discussion at wikipedia.
A slightly longer introduction is here.
More details available here, but I'm not sure it's presented very well.
All static and global variables are stored in the Data segment, all automatic and temporary variables are stored on the stack, and all dynamic variable are stored on the heap.
All function parameters are stored on the stack and there is a different stack frame for each function call this is how recursion function works.
For more on this, see this site.
In practical words, when you run any C-program, its executable image is loaded into RAM of computer in an organized manner which is called process address space or Memory layout of C program.
http://www.firmcodes.com/memory-layout-c-program-2/
all the static and global uninitialized variables goes into bss(Block started by Symbol).
all Initialized GLobal/Local/static variable further divide as
read only
const int x=10;
& read/write
char Str[]="StackOverFlow"
The stack segment is area where local variables are stored. By saying local variable means that all those variables which are declared in every function including main( ) in your C program.
Text segment contain executable instructions of your C program, its also called code segment. This is the machine language representation of the program steps to be carried out, including all functions making up the program, both user defined and system. The text segment is sharable so that only a single copy needs to be in memory for different executing programs, such as text editors, shells, and so on. Usually, text segment is read-only, to prevent a program from accidentally modifying its instructions.
one more region in the memory layout of a program is Unmapped or reserved segment contain command line arguments and other program related data like lower address-higher address of executable image, etc.