Architecture: X86, Linux
I have gone through several threads related to kmalloc() and vmalloc() and I know the basic differences between them but still have some doubts. I need your help to understand the fundamentals.
So As we know that in 32-bit systems the virtual address space is divided between user and kernel. The high 1GB memory addresses are allocated for kernel and the lower 3GB to user space.
Please correct me i am wrong here:
So out of 1GB 896MB is 1:1 mapped (physically contiguous) to the kernel logical address and 128MB is used to access the high memory addresses means accessing the physical(RAM) beyond 896 MB by mapping it into the 128MB using vmalloc()? (lets consider we have 2GB of RAM)
And what would happen if we have only 1GB of physical RAM in point 1. The 896MB is 1:1 mapping between kernel logical address and physical addresses achieved through kmalloc(). Then what would be the use of 128MB, why do we need vmalloc() anymore? So lets say if we have 1GB of RAM which entirely can be accessed through kernel logical address so does that mean that it is possible only when the amount of memory requested by a thread/process can be allocated in physically contiguous addresses. And if there is no physical contiguous memory available, in that case memory allocated will not be physically contiguous, hence we need to use vmalloc() to map it in 128MB of space and access it through there?
What happens when we use 64-bit system? Is it true that both kmalloc() and vmalloc() can have enormous amount of address space and can access entire RAM?
Related
Considering a 32bit x86 Linux system with 4 GB of RAM memory, So as described in books as well as on many forums that the Memory mapping would be as follows:
Kernel logical address - upto 896 MB - Which is one to one mapped and can be allocated using kmalloc().
Kernel virtual address - 128MB (above 896MB - kernel logical address) - allocated using vmalloc() and allocates Virtually contiguous but physically(scattered within RAM) non-contiguous memory pages.
Few points that I am not able to fully understand and need clarity on.
My understanding is that When kmalloc() is used to allocate memory, It always comes from the 0 to 896MB within the RAM and not beyond that.
When we use vmalloc() to allocate memory, Does that memory allocated anywhere from 896MB to 4GB range within the RAM ? or it is allocated only from 896MB upto 1GB range within RAM?
When we say that kernel has only 1GB of virtual address space, Does that meant that the kernel can not access the RAM beyond 1GB ? If it can then how it is done ? Does the 128MB of kernel virtual address space are used for this purpose ?
Please help.
In theory there are 3 different "memory managers". One manages physical RAM (mostly keeping track of pages of free physical RAM), one manages virtual space (what is mapped into each virtual address space where, working with fixed size pieces - the page size), and the third manages "heap" (allowing a larger area of the virtual address space to be split up into arbitrary sized pieces).
Originally; the Linux kernel tried to use its kernel "heap" to manage all 3 of these very different things. By linearly mapping "all RAM" into kernel space they bypass the need for managing the kernel's virtual memory and end up with a simple relationship between virtual addresses in kernel space and physical addresses (e.g. "physical = virtual - base"), and by allocating "heap" you also allocate physical memory.
This was fine originally because at the time computers rarely had more than 128 MiB of RAM (and Linus didn't expect the kernel to exist for very long, as GNU were planning to switch to Hurd "soon"), and kernel space was significantly larger than "all RAM". As the amount of RAM increased it became a problem -"all RAM" became larger than kernel space, so "use heap to manage 3 very different things" couldn't work.
Of course once it became a problem a lot of the kernel's code dependent on "kmalloc to allocate physical memory", making it too hard to fix the problem. Instead, they split physical memory into 2 zones - one zone that would be managed by "kmalloc" and another zone that is managed by "vmalloc"; then changed pieces of the kernel to use "vmalloc" instead of "kmalloc" where it's easy make those changes.
My understanding is that When kmalloc() is used to allocate memory, It always comes from the 0 to 896MB within the RAM and not beyond that.
Yes; this is the first zone of physical memory, which fits into the kernel space mapping that "kmalloc" uses.
When we use vmalloc() to allocate memory, Does that memory allocated anywhere from 896MB to 4GB range within the RAM ? or it is allocated only from 896MB upto 1GB range within RAM?
It would be allocated from any RAM that is not in the first zone (anywhere in "896MB or higher" range).
When we say that kernel has only 1GB of virtual address space, Does that meant that the kernel can not access the RAM beyond 1GB ? If it can then how it is done ? Does the 128MB of kernel virtual address space are used for this purpose ?
Of the kernel's 1 GiB of virtual space; some (896MB) will be the linear mapping of the physical address space, some will be memory mapped (PCI) devices, and some will be set aside as an area where dynamic mappings can be made. For "vmalloc" the kernel would allocate physical pages of RAM and then map them into the "dynamic mapping area" (and return a pointer to where it was mapped that has nothing to do with its physical address and breaks the "physical = virtual - base" relationship).
Note 1: Exact sizes/limits are variable - e.g. kernel can be compiled for "2 GiB / 2 GiB split" where kernel space is 2 GiB (instead of "3 GiB / 1 GiB split"); and the size of the "kmalloc zone" probably depends on various factors (how much space PCI devices need, how much RAM there is, etc) and may be something other than 896MB.
Note 2: Since introducing "vmalloc" to work around the original problem; computers switched to 64 bit (where "all memory" can/does fit in kernel space again), and "vmalloc" became unnecessary (and probably just falls through to "kmalloc"). However a lot of other changes have occurred (introduction of NUMA, encrypted RAM, non-volatile RAM, ..; plus more security vulnerabilities than any single person can keep track of) so the original design flaw has reached a temporary "bad idea, but still technically not broken if we keep adding work-arounds for security vulnerabilities" stage (until RAM and non-volatile RAM sizes inevitably increase and "vmalloc" becomes needed again at some point in the future - probably in about 30 years).
This question already has answers here:
What are the differences between virtual memory and physical memory?
(6 answers)
Closed 3 years ago.
What is virtual memory and, how it differs from physical memory (RAM)? It says that physical memory is stored on sth on motherboard, while virtual memory is stored on disk.
Somewhere it also says that virtual spaces are used only when the physical memory is filled, which confused me a lot.
Then, why Windows uses virtual memory? Is it because the RAMs are small-spaced and not designed for big storage, so use the virtual to store more bigger-sized things?
The next thing is about the address. Since virtuals are on disk, they shouldn't share the address of physicals. So they have independent addresses. Is that right?
And,
When writing memory of another process, why recommend using VirtualAlloc instead of HeapAlloc?
Is it true that virtual memory is process-dependent and the physical memory shared through processes?
"Virtual memory" means there is a valid address space, which does not map to any particular physical memory or storage, hence virtual. In context of modern common operating systems, each process has its own virtual memory space, with overlapping virtual memory addresses.
This address space is divided into pages for easier management (example size 4 KB). Each valid page can be in 3 different states:
not stored physically (assumed to be all 0). If process writes to this kind of page, it needs to be given a page of physical memory (by OS, see below) so value can be stored.
Mapped to physical memory, meaning some page-size area in computers RAM stores the contents, and they can be directly used by the process.
Swapped out to disk (might be a swap file), in order to free physical RAM pages (done automatically by the operating system). If the process accesses the page (read or write), it needs to be loaded to page in RAM first (see above).
Only when virtual memory page is mapped to physical RAM page, is there something there. In other cases, if process accesses that page, there is a CPU exception, which transfers control to operating system. OS then needs to either map that virtual memory page to RAM (possibly needing to free some RAM first by swapping current data out to swap file, or terminating some application if out of all memory) and load the right data into it, or it can terminate the application (address was not in valid range, or is read-only but process tries to write).
Same page of memory can also be mapped to several places at once, for example with shared memory, so same data can be accessed by several processes at once (virtual address is probably different, so can't share pointer variables).
Another special case of virtual memory use is mapping a regular file on disk to virtual memory (same thing which happens with swap file, but now controlled by normal application process). Then OS takes care of actually reading bytes (in page-sized chunks) from disk and writing changes back, the process can just access the memory like any memory.
Every modern multi-tasking general purpose operating system uses virtual memory, because the CPUs they run support it, and because it solves a big bunch of problems, for example memory fragmentation, transparently using swapping to disk, memory protection... They could be solved differently, but virtual memory is the way today.
Physical memory is shared between processes the same way as computer power supply is shared, or CPU is shared. It is part of the physical computer. A normal process never handles actual physical memory addresses, all that it sees is virtual memory, which may be mapped to different physical locations.
The contents of virtual memory are not normally shared, except when they are (when using shared memory for example).
Not sure you mean by "When collecting memory for other process", so can't answer that.
Virtual memory can essentially be thought of as a per process virtual address that's mapped to a physical address. In the case of x86 there is a register CR3 that points to the translation table for that process. When allocating new memory the OS will allocate physical memory, which may not even be contiguous, and then set a free contiguous virtual region to point to that physical memory. Whenever the CPU accesses any virtual memory it uses this translation table in CR3 to convert it to the actual physical address.
More Info
https://en.m.wikipedia.org/wiki/Control_register#CR3
https://en.m.wikipedia.org/wiki/Page_table
To quote Wikipedia:
In computing, virtual memory (also virtual storage) is a memory management technique that provides an "idealized abstraction of the storage resources that are actually available on a given machine" which "creates the illusion to users of a very large (main) memory."
Because virtual memory is an illusory memory (so, non-existent), some other computer resources rather than RAM is used. In this case, the resource used is the disk, because it has a lot of space, more than RAM, where the OS can run its VM stuff.
Somewhere it also says that virtual spaces are used only when the physical memory is filled, which confused me a lot.
It shouldn't. VM uses the disk and I/O with the disk is much slower than I/O with the RAM. This is why physical memory is preferred nowadays and VM is used when physical memory is not enough.
Then, why Windows uses virtual memory? Is it because the RAMs are small-spaced and not designed for big storage, so use the virtual to store more bigger-sized things?
This is one of the main reasons, yes. In the past (the 70s), computer memory was very expensive so workarounds had to be conceived.
I came across the following paragraph in an article about malloc.
The heap is a continuous (in term of virtual addresses) space of
memory with three bounds:a starting point, a maximum limit (managed
through sys/ressource.h’s functions getrlimit(2) and setrlimit(2)) and
an end point called the break. The break marks the end of the mapped
memory space, that is, the part of the virtual address space that has
correspondence into real memory.
I would like to better understand the concept of mapped region and unmapped region.
If memory addresses are 64 bits long, as in many modern computers, you have 18446744073709551616 possible memory addresses. (It depends on the processor architecture how many bits can actually be used, but addresses are stored using 64 bits.) That is more than 17 billion gigabytes, which is probably more memory than your computer actually has. So only some of those 17 billion gigabytes correspond to actual memory. For the rest of the addresses, the memory simply doesn't exist. There is no correspondence between the memory address and a memory location. Those addresses are, therefore, unmapped.
That is the simple explanation. In reality, it's a bit more complicated. The memory addresses of your program are not the actual memory addresses of the memory chips, the physical memory, in your computer. Instead, it is virtual memory. Each process has its own memory space, that is, its own 18446744073709551616 addresses, and the memory addresses that a process uses are translated to physical memory addresses by the computer hardware. So one process may have stored some data at memory address 4711, which is actually stored in a real physical memory chip over here, and another process may have also stored some data at memory address 4711, but that is a completely different place, stored in a real physical memory chip over there. The process-internal virtual memory addresses are translated, or mapped, to actual physical memory, but not all of them. The rest, again, are unmapped.
That is, of course, also a simplified explanation. You can use more virtual memory than the amount of physical memory in your computer. This is done by paging, that is, taking some chunks (called pages) of memory not being used right now, and storing them on disk until they are needed again. (This is also called "swapping", even if that term originally meant writing all the memory of a process to disk, not just parts of it.)
And to complicate it even further, some modern operating systems such as Linux and MacOS X (but, I am told, not Windows) overcommit when they allocate memory. This means that they allocate more memory addresses than can be stored on the computer, even using the disk. For example, my computer here with 32 gigabytes of physical memory, and just 4 gigabytes available for paging out data to disk, can't possibly allow for more than 36 gigabyes of actual, usable, virtual memory. But malloc happily allocates more than one hundred thousand gigabytes. It is not until I actually try to store things in all that memory that it is connected to physical memory or disk. But it was part of my virtual memory space, so I would call that too mapped memory, even though it wasn't mapped to anything.
The mapped region in the heap means the virtual memory area that can be mapped together with physical memory. And the unmapped region indicates the unused virtual memory space which does not point to any physical memory.
The boundary between mapped region for the heap and unmapped region is the system break point. As the malloc() is used to request some memory, the system break point would be moved to enlarge the mapped region. Linux system offeres brk() and sbrk() methods to increase and decrease the virtual memory address of the system break point.
I'm on 32bit machine. From what I understand, User space's address ranges from 0x00000000 to 0xbfffffff, and kernel's ranges from 0xc0000000 to 0xffffffff.
But when I used pmap to see a process's memory allocation, I see that the library are loaded in around 0xf7777777. Please see the attached screenshot. Does it mean those libraries are loaded in kernel space? And when I used mmap(), I got the address from 0xe0000000. So, mmap() got memory from kernel space?
I'm on 32bit machine. From what I understand, User space's address
ranges from 0x00000000 to 0xbfffffff, and kernel's ranges from
0xc0000000 to 0xffffffff.
Not exactly. Kernel memory space starts at 0xC0000000, but it doesn't have to fill the entire GB. In fact, it fills up to virtual address 0xF7FFFFFF. This covers 896MB of physical memory. Virtual addresses 0xF8000000 and up are used as a 128MB window for the kernel to map any region of physical memory beyond the 896MB limit.
All user processes share the same memory map for virtual addresses 0xC0000000 and beyond, so if the kernel does not use its entire GB of virtual space, it may reuse part of it to map commonly used shared libraries, so every process can see them.
In a 32-bit machine each process gets a 4GB virtual space. In this case one can worry that we might face trouble due to fragmentation. But in the case of a 64-bit machine we theoretically have a huge addressable virtual memory, so why is memory fragmentation still an issue (if it is) in a 64-bit machine?
Each virtual address that you try to access is mapped by the operating system to physical memory. Physical memory is allocated in pages (e.g. 4K in size). If you manage to allocate a byte at offset 1000000*n and do it for n from 1 to 1000000 (I think you could do that with mmap), then the OS will have to back that with a million pages of physical memory, which is something like 4G. That physical memory will not be available for anything else. If you had allocated the bytes contiguously, you'd only need about 1M of physical memory (256 pages) for your million bytes.
You can get in a similar bad situation if you allocate 4G for legitimate reasons, and then deallocate parts of it, keeping a bit of every page allocated. The OS will not be able to actually reuse the freed memory for anything else because there are no physical pages that are fully free. So that's a fragmentation problem.
In theory, you could imagine that virtual addresses 1000000 and 2000000 would map to the same page of physical memory, avoiding the fragmentation. But in practice, and for good reasons, the virtual memory mapping is done on a page by page basis. You can read more about it here: http://en.wikipedia.org/wiki/Page_table.
Because all that memory is "wasted" consider an application where you have a lot of internal fragmentation. That process requires more pages in memory because the working set is now scattered in memory and that means its memory footprint is much higher. If this application is contending for physical slots in RAM (machines still really only have about 4 - 8 GB of RAM for a typical home setup) then it causes more page swapping. Generally you want to reduce your applications memory footprint to avoid memory pressure and contention with other applications.
There are cases though where it doesn't really matter, it won't kill you to use an extra megabyte here or there but it all adds up in larger applications. It depends on the situation as to whether or not it is important to have as little fragmentation as possible depending on what you're coding or what the aim of your project is.