I mean the physical memory, the RAM.
In C you can access any memory address, so how does the operating system then prevent your program from changing memory address which is not in your program's memory space?
Does it set specific memory adresses as begin and end for each program, if so how does it know how much is needed.
Your operating system kernel works closely with memory management (MMU) hardware, when the hardware and OS both support this, to make it impossible to access memory you have been disallowed access to.
Generally speaking, this also means the addresses you access are not physical addresses but rather are virtual addresses, and hardware performs the appropriate translation in order to perform the access.
This is what is called a memory protection. It may be implemented using different methods. I'd recommend you start with a Wikipedia article on this subject — http://en.wikipedia.org/wiki/Memory_protection
Actually, your program is allocated virtual memory, and that's what you work with. The OS gives you a part of the RAM, you can't access other processes' memory (unless it's shared memory, look it up).
It depends on the architecture, on some it's not even possible to prevent a program from crashing the system, but generally the platform provides some means to protect memory and separate address space of different processes.
This has to do with a thing called 'paging', which is provided by the CPU itself. In old operating systems, you had 'real mode', where you could directly access memory addresses. In contrast, paging gives you 'virtual memory', so that you are not accessing the raw memory itself, but rather, what appears to your program to be the entire memory map.
The operating system does "memory management" often coupled with TLB's (Translation Lookaside Buffers) and Virtual Memory, which translate any address to pages, which the operation system can tag readable or executable in the current processes context.
The minimum requirement for a processors MMU or memory management unit is in current context restrict the accessable memory to a range which can be only set in processors registers in supervisor mode (as opposed to user mode).
The logical address is generated by the CPU which is mapped to the physical address by the memory mapping unit. Unlike the physical address space the logical address is not restricted by memory size and you just get to work with the logical address space. The address binding is done by MMU. So you never deal with the physical address directly.
Most computers (and all PCs since the 386) have something called the Memory Management Unit (or MMU). It's job is to translate local addresses used by a program into the physical addresses needed to fetch real bytes from real memory. It's the operating system's job to program the MMU.
As a result of this, programs can be loaded into any region of memory and appear, from that program's point of view while executing, to be be any any other address. It's common to find that the code for all programs appear (locally) to be at the same address and their data always appears (locally) to be at the same address even though physically they will be in different locations. With each memory access, the MMU transparently translates from the local address space to the physical one.
If a program trys to access a memory address that has not been mapped into its local address space, the hardware generates an exception and typically gets flagged as a "segmentation violation", followed by the forcible termination of the program. This protects from accessing the memory of other processes.
But that doesn't have to be the case! On systems with "virtual memory" and current resource demands on RAM that exceed the amount of physical memory, some pages (just blocks of memory of a common size, often on the order of 4-8kB) can be written out to disk and given as RAM to a program trying to allocate and use new memory. Later on, when that page is needed by whatever program owns it, the memory access causes an exception and the OS swaps out some other memory page and re-loads the needed one from disk. The program that "page-faulted" gets delayed while this happens but otherwise notices nothing.
There are lots of other tricks the MMU/OS can do as well, like sharing memory between processes, making a disk file appear to be direct-memory-accessible, setting some pages as "NX" so they can't be treated as executable code, using arbitrary sections of the logical memory space regardless of how much and at what address the physical ram uses, and more.
Related
This question already has answers here:
What are the differences between virtual memory and physical memory?
(6 answers)
Closed 3 years ago.
What is virtual memory and, how it differs from physical memory (RAM)? It says that physical memory is stored on sth on motherboard, while virtual memory is stored on disk.
Somewhere it also says that virtual spaces are used only when the physical memory is filled, which confused me a lot.
Then, why Windows uses virtual memory? Is it because the RAMs are small-spaced and not designed for big storage, so use the virtual to store more bigger-sized things?
The next thing is about the address. Since virtuals are on disk, they shouldn't share the address of physicals. So they have independent addresses. Is that right?
And,
When writing memory of another process, why recommend using VirtualAlloc instead of HeapAlloc?
Is it true that virtual memory is process-dependent and the physical memory shared through processes?
"Virtual memory" means there is a valid address space, which does not map to any particular physical memory or storage, hence virtual. In context of modern common operating systems, each process has its own virtual memory space, with overlapping virtual memory addresses.
This address space is divided into pages for easier management (example size 4 KB). Each valid page can be in 3 different states:
not stored physically (assumed to be all 0). If process writes to this kind of page, it needs to be given a page of physical memory (by OS, see below) so value can be stored.
Mapped to physical memory, meaning some page-size area in computers RAM stores the contents, and they can be directly used by the process.
Swapped out to disk (might be a swap file), in order to free physical RAM pages (done automatically by the operating system). If the process accesses the page (read or write), it needs to be loaded to page in RAM first (see above).
Only when virtual memory page is mapped to physical RAM page, is there something there. In other cases, if process accesses that page, there is a CPU exception, which transfers control to operating system. OS then needs to either map that virtual memory page to RAM (possibly needing to free some RAM first by swapping current data out to swap file, or terminating some application if out of all memory) and load the right data into it, or it can terminate the application (address was not in valid range, or is read-only but process tries to write).
Same page of memory can also be mapped to several places at once, for example with shared memory, so same data can be accessed by several processes at once (virtual address is probably different, so can't share pointer variables).
Another special case of virtual memory use is mapping a regular file on disk to virtual memory (same thing which happens with swap file, but now controlled by normal application process). Then OS takes care of actually reading bytes (in page-sized chunks) from disk and writing changes back, the process can just access the memory like any memory.
Every modern multi-tasking general purpose operating system uses virtual memory, because the CPUs they run support it, and because it solves a big bunch of problems, for example memory fragmentation, transparently using swapping to disk, memory protection... They could be solved differently, but virtual memory is the way today.
Physical memory is shared between processes the same way as computer power supply is shared, or CPU is shared. It is part of the physical computer. A normal process never handles actual physical memory addresses, all that it sees is virtual memory, which may be mapped to different physical locations.
The contents of virtual memory are not normally shared, except when they are (when using shared memory for example).
Not sure you mean by "When collecting memory for other process", so can't answer that.
Virtual memory can essentially be thought of as a per process virtual address that's mapped to a physical address. In the case of x86 there is a register CR3 that points to the translation table for that process. When allocating new memory the OS will allocate physical memory, which may not even be contiguous, and then set a free contiguous virtual region to point to that physical memory. Whenever the CPU accesses any virtual memory it uses this translation table in CR3 to convert it to the actual physical address.
More Info
https://en.m.wikipedia.org/wiki/Control_register#CR3
https://en.m.wikipedia.org/wiki/Page_table
To quote Wikipedia:
In computing, virtual memory (also virtual storage) is a memory management technique that provides an "idealized abstraction of the storage resources that are actually available on a given machine" which "creates the illusion to users of a very large (main) memory."
Because virtual memory is an illusory memory (so, non-existent), some other computer resources rather than RAM is used. In this case, the resource used is the disk, because it has a lot of space, more than RAM, where the OS can run its VM stuff.
Somewhere it also says that virtual spaces are used only when the physical memory is filled, which confused me a lot.
It shouldn't. VM uses the disk and I/O with the disk is much slower than I/O with the RAM. This is why physical memory is preferred nowadays and VM is used when physical memory is not enough.
Then, why Windows uses virtual memory? Is it because the RAMs are small-spaced and not designed for big storage, so use the virtual to store more bigger-sized things?
This is one of the main reasons, yes. In the past (the 70s), computer memory was very expensive so workarounds had to be conceived.
In C, when you get the address of a variable is that address an address that really exist in the RAM of the computer or just an address in a fake memory in the C compiler (if that's how it really works)? Can you explain in layman’s terms?
Yes and no. When you take the address of a variable, and perform some operations on it (assuming the compiler doesn't optimize it out), it will correspond to an address in ram. However because of virtual memory, the address used in your program is almost certainly not the address of the variable in physical ram. The kernel remaps what virtual addresses (what your program sees) refer to which physical addresses (what the memory sees), so that different processes can be loaded into memory at the same time, yet not be able to access each others' memory. Additionally, your process's memory can be paged out, or written to disk if it has not been used recently and/or something else needs more memory, and reloaded into a completely different address, yet the virtual address will remain the same.
So yes, when you access a pointer, that address corresponds to an address in memory. But that address doesn't correspond to the actual address in ram, and the address it corresponds to can change over time.
The sort answer is "neither".
In general terms, the address of a variable in memory is in the context of a running program's address space.
What differs is how the program's address space is mapped to hardware by the host system.
With modern hardware that has a memory management unit (MMU), and operating systems (or their device drivers) that use the MMU, a program's address space is mapped to physical memory, which may consist of RAM or virtual memory, such as a swap file on a hard drive. The operating system uses the MMU to isolate programs from each other (so two processes cannot access each other's address space) and also uses the MMU to support swapping of data between RAM and swap. The running process cannot generally tell where its data is in physical memory, because the operating system and MMU specifically prevent it from doing so. Over time, the operating system and MMU may migrate memory used a program to different areas of RAM or to swap, but the program cannot detect this, since the operating system and MMU take care of mapping an address in the program (which never changes as far as the program is concerned) to the actual address. This covers most modern versions of windows, unix, and various realtime operating systems. (Those systems also typically provide means of programatically accessing physical memory, but only for programs that are running with higher privileges or for kernel mode drivers).
Older hardware did not have an MMU, so operating systems were not able to give programs separate address spaces. On such systems, the address as seen by a program had a one-to-one correspondence to a location in physical memory.
Somewhere in between was hardware that had separate areas of physical memory (e.g. provided by distinct banks of memory chips). On those systems, with support of special drivers, a host system could implement a partial mapping between addresses in a program's address space, and locations in particular areas of physical memory. This is why some target systems, and compilers that support them, support more than one pointer type (e.g. with names like near, far, and huge) as a compiler extension. In those cases, a pointer could refer to a location in a particular area of memory, and there may be some mapping of values, for each pointer type, from the value of a pointer seen by a program to the actual location within a corresponding area of physical memory.
The C compiler does not become a part of executable program it builds (otherwise, to install any built program, it would be necessary to also install and execute the compiler used to build it, or the program would not run). Typically, a compiler is no longer running when a program is executed (or, at least, a program cannot rely on it being present). A program therefore cannot access addresses within the compiler's address space.
In an interpreted environment (e.g. C code is interpreted by another program - the interpreter) the interpreter acts as an intermediary between the program and the hardware, and handles mapping between a program's address space, the interpreter's address space, and physical memory. C interpreters are relatively rare in practice, compared with toolchains that use compilers and linkers.
On ancient OSes, the MMU isn't present on the target processor, or not used (even if the processor allows it).
In that case, physical addresses are used, which is simpler to understand but also annoying because when you're debugging an assembly program or trying to decode a traceback, you have to know where the program was loaded or the post-mortem traceback is useless.
Without MMU, you can do very hacky & simple things. Shared memory can be coded in a few lines, you can inspect the whole memory very easily, etc...
On modern OSes, relying on MMU processor capability and address translation, executables are running in a virtual memory, which isn't an issue since they cannot access other executables memory anyway.
The good side is that if you're running/debugging the same executable many times, you always get the same addresses. Useful on long debugging sessions where you have to restart the debugger many times.
Also, some languages/compilers (like GNAT Ada compiler) provide a traceback with addresses when the program does something illegal. Using addr2line on the executable, you're able to get the exact traceback even after the process has ended and memory has been released.
The exception I know of is Windows shared libraries (DLL) which are almost never loaded at the same address, since this address is potentially shared between several executables. In those cases, for instance, a post-mortem traceback will be useless because the declared symbol address has an offset from the actual traceback address.
In case of multi-process environment where multiple processes runs at same time, linker can not decide address of the variables at compile time.
Reason is simple, if you assign dedicated address to the variables then you limit the number of processes that can run on your system.
So they assign a virtual address to the variables and those addresses translated to the physical addresses during run-time with the help of OS and processor.
One example of such system is linux running on x86 CPU.
In other cases where only one process/application runs on a processor then linker can assign actual physical address to variables.
example: embedded systems performing dedicated tasks, such as Oven.
I have gone through below link and it says that on most Operating Systems , pointers store the virtual address rather than physical address but I am unable to get the benefit of storing virtual address in a pointer .
when at the end we can modify the contents of a particular memory location directly through a pointer so what is the issue whether it is a virtual address or a physical address ?
Also during the time the code executes , most of the time the data segment will also remain in memory ,so we are dealing with physical memory location only so how the virtual address is useful ?
C pointers and the physical address
Security issues (as noted before) aside, there is another big advantage:
Globals and functions (and your stack) can always be found at fixed addresses (so the assembler can hardcode them), independently of where the instance of your program is loaded.
If you'd really like your code to run from any address, you have to make it position independent (with gcc you'd use the -fPIC argument). This question might be an interresting read wrt the -fPIC and virtual addressing: GCC -fPIC option
Actually, pointers normally hold LOGICAL ADDRESSES.
There are several reasons for using logical addressing, including:
Ease of memory management. The OS never has to allocate contiguous physical page frames to a process.
Security. Each process has access to its own logical address space and cannot mess with another processes address space.
Virtual Memory. Logical address translation is a prerequisite for implementing virtual memory.
Page protection. Aids security by limiting access to system pages to higher processor modes. Helps error (and virus) trapping by limiting the types of access to pages (e.g., not allowing writes to or executes of data pages).
This is not a complete list.
The same virtual address can point to different physical addresses in different moments.
If your physical memory is full, your data is swapped out from the memory to your HDD. When your program wants to access this data again - it is currently not stored in memory - the data is swapped in back to memory, but it often will be a different location as before.
The Page Table, which stores the assignment of virtual to physical addresses, is updated with the new physical address. So your virtual address remains the same, while the physical address may change.
I understand if I try to print the address of an element of an array it would be an address from virtual memory not from real memory (physical memory) i.e DRAM.
printf ("Address of A[5] and A[6] are %u and %u", &A[5], &A[6]);
I found addresses were consecutive (assuming elements are chars). In reality they may not be consecutive at least not in the DRAM. I want to know the real addresses. How do I get that?
I need to know this for either Windows or Linux.
You can't get the physical address for a virtual address from user code; only the lowest levels of the kernel deal with physical addresses, and you'd have to intercept things there.
Note that the physical address for a virtual address may not be constant while the program runs — the page might be paged out from one physical address and paged back in to a different physical address. And if you make a system call, this remapping could happen between the time when the kernel identifies the physical address and when the function call completes because the program requesting the information was unscheduled and partially paged out and then paged in again.
The simple answer is that, in general, for user processes or threads in a multiprocessing OS such as Windows or Linux, it is not possible to find the address even of of a static variable in the processor's memory address space, let alone the DRAM address.
There are a number of reasons for this:
The memory allocated to a process is virtual memory. The OS can remap this process memory from time-to-time from one physical address range to another, and there is no way to detect this remaping in the user process. That is, the physical address of a variable can change during the lifetime of a process.
There is no interface from userspace to kernel space that would allow a userspace process to walk through the kernel's process table and page cache in order to find the physical address of the process. In Linux you can write a kernel module or driver that can do this.
The DRAM is often mapped to the procesor address space through a memory management unit (MMU) and memory cache. Although the MMU maping of DRAM to the processor address space is usually done only once, during system boot, the processor's use of the cache can mean that values written to a variable might not be written through to the DRAM in all cases.
There are OS-specific ways to "pin" a block of allocated memory to a static physical location. This is often done by device drivers that use DMA. However, this requires a level of privilege not available to userspace processes, and, even if you have the physical address of such a block, there is no pragma or directive in the commonly used linkers that you could use to allocate the BSS for a process at such a physical address.
Even inside the Linux kernel, virtual to physical address translation is not possible in the general case, and requires knowledge about the means that were used to allocate the memory to which a particular virtual address refers.
Here is a link to an article called Translating Virtual to Physical Address on Windows: Physical Addresses that gives you a hint as to the extreme ends to which you must go to get physical addresses on Windows.
AFAIK, string literals are stored in read only memory in case of C language.
where is this actually present on the hardware.
as per my knowledge heap is on RAM.correct me if i am wrong.
how different is heap from read only memory?
is it OS dependant?
It is usually done with hardware assistance.
The virtual memory subsystem of the hardware can be told to mark a page as read-only. When an application tries to write a read-only page, the hardware generates a fault that the OS catches. The OS can tell that the app tried to write a read-only page and end your program.
So the OS/loader makes sure the pages the string literals are in are marked as read-only.
The heap and read-only memory are orthogonal issues.
It's OS and hardware dependent. The spec says they can be placed in read-only memory, rather than they have to be. If you're writing C for a simple embedded device, then the strings get blown into the rom and runtime memory is allocated from the RAM; these are physically separate (Harvard). If it's a typical unix-like computer, then there is virtual memory subsystem which converts logical addresses to physical addresses in pages and can mark some pages read-only and some executable, but the memory itself can be either data or instructions (Von Neumann).
Usually that's ordinary process virtual memory with write protection set.
That's implementation-dependent, but processors usually use special metadata blocks for controlling access to memory regions and the operating system can set those accordingly. So the string literals and all other unchangeable stuff is loaded into a region that is set protection on. So when the program tries to modify that memory a special unit inside the processor checks whther that write is allowed and if it is not wllowed issues a hardware interrupt that is handled by the operating system.
From an hw point of view RAM is RAM and it can be R/W. The "read-only" memory is an attribute put by software; on some architecture (almost all we're used to) there is hardware support to make a portion of memory not writable (in the sense that when you try to access an address inside that memory, an "exception" occurs).