I have gone through below link and it says that on most Operating Systems , pointers store the virtual address rather than physical address but I am unable to get the benefit of storing virtual address in a pointer .
when at the end we can modify the contents of a particular memory location directly through a pointer so what is the issue whether it is a virtual address or a physical address ?
Also during the time the code executes , most of the time the data segment will also remain in memory ,so we are dealing with physical memory location only so how the virtual address is useful ?
C pointers and the physical address
Security issues (as noted before) aside, there is another big advantage:
Globals and functions (and your stack) can always be found at fixed addresses (so the assembler can hardcode them), independently of where the instance of your program is loaded.
If you'd really like your code to run from any address, you have to make it position independent (with gcc you'd use the -fPIC argument). This question might be an interresting read wrt the -fPIC and virtual addressing: GCC -fPIC option
Actually, pointers normally hold LOGICAL ADDRESSES.
There are several reasons for using logical addressing, including:
Ease of memory management. The OS never has to allocate contiguous physical page frames to a process.
Security. Each process has access to its own logical address space and cannot mess with another processes address space.
Virtual Memory. Logical address translation is a prerequisite for implementing virtual memory.
Page protection. Aids security by limiting access to system pages to higher processor modes. Helps error (and virus) trapping by limiting the types of access to pages (e.g., not allowing writes to or executes of data pages).
This is not a complete list.
The same virtual address can point to different physical addresses in different moments.
If your physical memory is full, your data is swapped out from the memory to your HDD. When your program wants to access this data again - it is currently not stored in memory - the data is swapped in back to memory, but it often will be a different location as before.
The Page Table, which stores the assignment of virtual to physical addresses, is updated with the new physical address. So your virtual address remains the same, while the physical address may change.
Related
This question already has answers here:
What are the differences between virtual memory and physical memory?
(6 answers)
Closed 3 years ago.
What is virtual memory and, how it differs from physical memory (RAM)? It says that physical memory is stored on sth on motherboard, while virtual memory is stored on disk.
Somewhere it also says that virtual spaces are used only when the physical memory is filled, which confused me a lot.
Then, why Windows uses virtual memory? Is it because the RAMs are small-spaced and not designed for big storage, so use the virtual to store more bigger-sized things?
The next thing is about the address. Since virtuals are on disk, they shouldn't share the address of physicals. So they have independent addresses. Is that right?
And,
When writing memory of another process, why recommend using VirtualAlloc instead of HeapAlloc?
Is it true that virtual memory is process-dependent and the physical memory shared through processes?
"Virtual memory" means there is a valid address space, which does not map to any particular physical memory or storage, hence virtual. In context of modern common operating systems, each process has its own virtual memory space, with overlapping virtual memory addresses.
This address space is divided into pages for easier management (example size 4 KB). Each valid page can be in 3 different states:
not stored physically (assumed to be all 0). If process writes to this kind of page, it needs to be given a page of physical memory (by OS, see below) so value can be stored.
Mapped to physical memory, meaning some page-size area in computers RAM stores the contents, and they can be directly used by the process.
Swapped out to disk (might be a swap file), in order to free physical RAM pages (done automatically by the operating system). If the process accesses the page (read or write), it needs to be loaded to page in RAM first (see above).
Only when virtual memory page is mapped to physical RAM page, is there something there. In other cases, if process accesses that page, there is a CPU exception, which transfers control to operating system. OS then needs to either map that virtual memory page to RAM (possibly needing to free some RAM first by swapping current data out to swap file, or terminating some application if out of all memory) and load the right data into it, or it can terminate the application (address was not in valid range, or is read-only but process tries to write).
Same page of memory can also be mapped to several places at once, for example with shared memory, so same data can be accessed by several processes at once (virtual address is probably different, so can't share pointer variables).
Another special case of virtual memory use is mapping a regular file on disk to virtual memory (same thing which happens with swap file, but now controlled by normal application process). Then OS takes care of actually reading bytes (in page-sized chunks) from disk and writing changes back, the process can just access the memory like any memory.
Every modern multi-tasking general purpose operating system uses virtual memory, because the CPUs they run support it, and because it solves a big bunch of problems, for example memory fragmentation, transparently using swapping to disk, memory protection... They could be solved differently, but virtual memory is the way today.
Physical memory is shared between processes the same way as computer power supply is shared, or CPU is shared. It is part of the physical computer. A normal process never handles actual physical memory addresses, all that it sees is virtual memory, which may be mapped to different physical locations.
The contents of virtual memory are not normally shared, except when they are (when using shared memory for example).
Not sure you mean by "When collecting memory for other process", so can't answer that.
Virtual memory can essentially be thought of as a per process virtual address that's mapped to a physical address. In the case of x86 there is a register CR3 that points to the translation table for that process. When allocating new memory the OS will allocate physical memory, which may not even be contiguous, and then set a free contiguous virtual region to point to that physical memory. Whenever the CPU accesses any virtual memory it uses this translation table in CR3 to convert it to the actual physical address.
More Info
https://en.m.wikipedia.org/wiki/Control_register#CR3
https://en.m.wikipedia.org/wiki/Page_table
To quote Wikipedia:
In computing, virtual memory (also virtual storage) is a memory management technique that provides an "idealized abstraction of the storage resources that are actually available on a given machine" which "creates the illusion to users of a very large (main) memory."
Because virtual memory is an illusory memory (so, non-existent), some other computer resources rather than RAM is used. In this case, the resource used is the disk, because it has a lot of space, more than RAM, where the OS can run its VM stuff.
Somewhere it also says that virtual spaces are used only when the physical memory is filled, which confused me a lot.
It shouldn't. VM uses the disk and I/O with the disk is much slower than I/O with the RAM. This is why physical memory is preferred nowadays and VM is used when physical memory is not enough.
Then, why Windows uses virtual memory? Is it because the RAMs are small-spaced and not designed for big storage, so use the virtual to store more bigger-sized things?
This is one of the main reasons, yes. In the past (the 70s), computer memory was very expensive so workarounds had to be conceived.
I am working on Video HAL Application & there I am getting Camera frame CallBack from HAL Layer. During programming I found that memcpy copying data from physical address gets crashed while it is ok by copying data from virtual address. I searched for such a information about memcpy but found no where & even not on its man page.
so, my question is does memcpy required physical address or virtual address? Anywhere mentioned this type of information about memcpy?
memcpy is implemented in C or optimized in assembler. As such, it doesn't care about what type of address it gets. It just loads the addresses in the CPU registers and executes mov instructions.
It is the operating system and memory hardware architecture that are responsible for mapping any logical (virtual) address to a physical address.
Note also that with modern OS/memory architectures, each process gets its own address space. Passing addresses between address spaces will not work.
In these cases, the OS will probably provide functionality to exchange memory objects (shared or otherwise) between processes.
As Paul Ogilvie correctly explained, memcpy deals with user space addresses. As such they are virtual addresses, not necessarily physical addresses.
Yet there is a possibility for very large areas with very specific alignment characteristics to optimize memcpy by requesting the OS to remap some of the destination virtual addresses as duplicates of the physical memory mapped to the source addresses. These pages would get the COPY_ON_WRITE attribute to ensure that the program only modifies pages in the appropriate array, if and when it writes to either one. The generic implementation of memcpy in the GlibC does this (see glibc-2.3.5/sysdeps/generic/memcpy.c). But this is transparent for the programmer who still provides addresses in user space.
I understand if I try to print the address of an element of an array it would be an address from virtual memory not from real memory (physical memory) i.e DRAM.
printf ("Address of A[5] and A[6] are %u and %u", &A[5], &A[6]);
I found addresses were consecutive (assuming elements are chars). In reality they may not be consecutive at least not in the DRAM. I want to know the real addresses. How do I get that?
I need to know this for either Windows or Linux.
You can't get the physical address for a virtual address from user code; only the lowest levels of the kernel deal with physical addresses, and you'd have to intercept things there.
Note that the physical address for a virtual address may not be constant while the program runs — the page might be paged out from one physical address and paged back in to a different physical address. And if you make a system call, this remapping could happen between the time when the kernel identifies the physical address and when the function call completes because the program requesting the information was unscheduled and partially paged out and then paged in again.
The simple answer is that, in general, for user processes or threads in a multiprocessing OS such as Windows or Linux, it is not possible to find the address even of of a static variable in the processor's memory address space, let alone the DRAM address.
There are a number of reasons for this:
The memory allocated to a process is virtual memory. The OS can remap this process memory from time-to-time from one physical address range to another, and there is no way to detect this remaping in the user process. That is, the physical address of a variable can change during the lifetime of a process.
There is no interface from userspace to kernel space that would allow a userspace process to walk through the kernel's process table and page cache in order to find the physical address of the process. In Linux you can write a kernel module or driver that can do this.
The DRAM is often mapped to the procesor address space through a memory management unit (MMU) and memory cache. Although the MMU maping of DRAM to the processor address space is usually done only once, during system boot, the processor's use of the cache can mean that values written to a variable might not be written through to the DRAM in all cases.
There are OS-specific ways to "pin" a block of allocated memory to a static physical location. This is often done by device drivers that use DMA. However, this requires a level of privilege not available to userspace processes, and, even if you have the physical address of such a block, there is no pragma or directive in the commonly used linkers that you could use to allocate the BSS for a process at such a physical address.
Even inside the Linux kernel, virtual to physical address translation is not possible in the general case, and requires knowledge about the means that were used to allocate the memory to which a particular virtual address refers.
Here is a link to an article called Translating Virtual to Physical Address on Windows: Physical Addresses that gives you a hint as to the extreme ends to which you must go to get physical addresses on Windows.
We have address translation table to translate virtual address (VA) of a process to its corresponding physical address in RAM, but if the table does not have any entry for a VA , it results in page fault and kernal goes to backing store (often a hard drive) and fetch the corresponding data and update the RAM and address translation table. So my question is how does the OS come to know what is the address corresponding to a VA in backing store ? Does it have a separate translation table for that?
A process starts by allocating virtual memory. That eventually will cause a page fault when the program starts actually addressing the virtual memory address. The OS knows that the memory access is valid. Since it was allocated explicitly.
So no harm done, the OS simply maps the VM address to a physical address.
If the page fault is for an address that was not previously requested to be a valid VM address then the processor will discover that there is no page table entry for the address. And will instead raise an GP fault, an AccessViolation or segfault in your program. Kaboom, program over.
There is no direct correlation, at least not in the way that you suppose.
The operating system divides virtual and phsyical RAM as well as swap space (backing store) and mapped files into pages, most commonly 4096 bytes.
When your code accesses a certain address, this is always a virtual address within a page that is either valid-in-core, valid-not-accessed, valid-out-of-core, or invalid. The OS may have other properties (such as "has been written to") in its books, but they're irrelevant for us here.
If the page is in-core, then it has a physical address, otherwise it does not. When swapped out and in again, the same identical page could in theory very well land in a different physical region of memory. Similarly, the page after some other page in memory (virtual or physical) could be before that page in the swap file or in a memory-mapped file. There's no guarantee for that.
Thus, there is no such thing as translating a virtual address to a physical address in backing store. There is only a translation from a virtual address to a page which may temporarily have a physical address. In the easiest case, "translating" means dividing by 4096, but of course other schemes are possible.
Further, every time your code accesses a memory location, the virtual address must be translated to a physical one. There exists dedicated logic inside a CPU to do this translation fully automatically (for a very small subset of "hot" pages, often as few as 64), or in a hardware-assisted way, which usually involves a lookup in a more or less complicated hierarchical structure.
This is also a fault, but it's one that you don't see. The only faults that you get to see are the ones when the OS doesn't have a valid page (or can't supply it for some reason), and thus can't assign a physical address to the to-be-translated virtual one.
When your program asks for memory, the OS remembers that certain pages are valid, but they do not exist yet because you have never accessed them.
The first time you access a page, a fault happens and obviously its address is nowhere in the translation tables (how could it be, it doesn't exist!). Thus the OS looks into its books and (assuming the page is valid) it either loads the page from disk or assigns the address of a zero page otherwise.
Not rarely, the OS will cheat and all zero pages are the same write-protected zero page until you actually write to it (at which point, a fault occurs and you are secretly redirected to a different physical memory area, one which you can write to, too.
Otherwise, that is if you haven't reserved memory, the OS sends a signal (or an equivalent, Windows calls it "exception") which will terminate your process unless handled.
For a multitude of reasons, the OS may later decide to remove one or several pages from your working set. This normally does not immediately remove them, but maked them candidates for being swapped (for non-mapped data) or discarded (for mapped data) in case more memory is needed. When you access an address in one of these pages again, it is either re-added to your working set (likely pushing another one out) or reloaded from disk.
In either case, all the OS needs to know is how to translate your virtual address to a page identifier of some sort (e.g. "page frame number"), and whether the page is resident (and at what address).
I think your question answer is issue about interrupt table.
Page fault is a kind of software interrupt, and operating system must have some solution to that interrupt.And the solution code is already in the os kernel, and that piece of code address is right at the interrupt table.So the page fault happen, os will go to that piece of code to get the unmapped page into the physical memory.
This is OS-specific, but many implementations share logic with memory-mapped file features (so that anonymous pages actually are memory-mapped views of the pagefile, flagged so that the content can be discards at unmapping instead of flushed).
For Windows, much of this is documented here, on the CreateFileMapping page
I mean the physical memory, the RAM.
In C you can access any memory address, so how does the operating system then prevent your program from changing memory address which is not in your program's memory space?
Does it set specific memory adresses as begin and end for each program, if so how does it know how much is needed.
Your operating system kernel works closely with memory management (MMU) hardware, when the hardware and OS both support this, to make it impossible to access memory you have been disallowed access to.
Generally speaking, this also means the addresses you access are not physical addresses but rather are virtual addresses, and hardware performs the appropriate translation in order to perform the access.
This is what is called a memory protection. It may be implemented using different methods. I'd recommend you start with a Wikipedia article on this subject — http://en.wikipedia.org/wiki/Memory_protection
Actually, your program is allocated virtual memory, and that's what you work with. The OS gives you a part of the RAM, you can't access other processes' memory (unless it's shared memory, look it up).
It depends on the architecture, on some it's not even possible to prevent a program from crashing the system, but generally the platform provides some means to protect memory and separate address space of different processes.
This has to do with a thing called 'paging', which is provided by the CPU itself. In old operating systems, you had 'real mode', where you could directly access memory addresses. In contrast, paging gives you 'virtual memory', so that you are not accessing the raw memory itself, but rather, what appears to your program to be the entire memory map.
The operating system does "memory management" often coupled with TLB's (Translation Lookaside Buffers) and Virtual Memory, which translate any address to pages, which the operation system can tag readable or executable in the current processes context.
The minimum requirement for a processors MMU or memory management unit is in current context restrict the accessable memory to a range which can be only set in processors registers in supervisor mode (as opposed to user mode).
The logical address is generated by the CPU which is mapped to the physical address by the memory mapping unit. Unlike the physical address space the logical address is not restricted by memory size and you just get to work with the logical address space. The address binding is done by MMU. So you never deal with the physical address directly.
Most computers (and all PCs since the 386) have something called the Memory Management Unit (or MMU). It's job is to translate local addresses used by a program into the physical addresses needed to fetch real bytes from real memory. It's the operating system's job to program the MMU.
As a result of this, programs can be loaded into any region of memory and appear, from that program's point of view while executing, to be be any any other address. It's common to find that the code for all programs appear (locally) to be at the same address and their data always appears (locally) to be at the same address even though physically they will be in different locations. With each memory access, the MMU transparently translates from the local address space to the physical one.
If a program trys to access a memory address that has not been mapped into its local address space, the hardware generates an exception and typically gets flagged as a "segmentation violation", followed by the forcible termination of the program. This protects from accessing the memory of other processes.
But that doesn't have to be the case! On systems with "virtual memory" and current resource demands on RAM that exceed the amount of physical memory, some pages (just blocks of memory of a common size, often on the order of 4-8kB) can be written out to disk and given as RAM to a program trying to allocate and use new memory. Later on, when that page is needed by whatever program owns it, the memory access causes an exception and the OS swaps out some other memory page and re-loads the needed one from disk. The program that "page-faulted" gets delayed while this happens but otherwise notices nothing.
There are lots of other tricks the MMU/OS can do as well, like sharing memory between processes, making a disk file appear to be direct-memory-accessible, setting some pages as "NX" so they can't be treated as executable code, using arbitrary sections of the logical memory space regardless of how much and at what address the physical ram uses, and more.