Cannot Exhaust Physical Memory in 32 bit Linux - c

Question

So I've got an interesting OS based problem for you. I've spent the last few hours conversing with anyone I know who's experienced with C programming, and nobody seems to be able to come up with a definitive answer as to why this behaviour is occurring.
I have a program that is intentionally designed to cause an extreme memory leak, (as an example of what happens when you don't free memory after allocating it). On 64 bit operating systems, (Windows, Linux, etc), it does what it should do. It fills physical ram, then fills the swap space of the OS. In Linux, the process is then terminated by the OS. In Windows however, it is not, and it continues running. The eventual result is a system crash.
Here's the code:
#include <stdlib.h>
#include <stdio.h>
void main()
{
while(1)
{
int *a;
a = (int*)calloc(65536, 4);
}
}
However, if you compile and run this code on a 32 bit Linux distribution, it has no effect on physical memory usage at all. It uses approximately 1% of my 4 GB of allocated RAM, and it never rises after that. I don't have a legitimate copy of 32 Bit Windows to test on, so I can't be certain this occurs on 32 bit Windows as well.
Can somebody please explain why the use of calloc will fill the physical ram of a 64 bit Linux OS, but not a 32 bit Linux OS?

Answer 1

The malloc and calloc functions do not technically allocate memory, despite their name. They actually allocate portions of your program's address space with OS-level read/write permissions. This is a subtle difference and is not relevant most of the time.
This program, as written, only consumes address space. Eventually, calloc will start returning NULL but the program will continue running.
#include <stdlib.h>
// Note main should be int.
int main() {
while (1) {
// Note calloc should not be cast.
int *a = calloc(65536, sizeof(int));
}
}
If you write to the addresses returned from calloc, it will force the kernel to allocate memory to back those addresses.
#include <stdlib.h>
#include <string.h>
int main() {
size_t size = 65536 * 4;
while (1) {
// Allocates address space.
void *p = calloc(size, 1);
// Forces the address space to have allocated memory behind it.
memset(p, 0, size);
}
}
It's not enough to write to a single location in the block returned from calloc because the granularity for allocating actual memory is 4 KiB (the page size... 4 KiB is the most common). So you can get by with just writing to each page.
What about the 64-bit case?
There is some bookkeeping overhead for allocating address space. On a 64-bit system, you get something like 40 or 48 bits of address space, of which about half can be allocated to the program, which comes to at least 8 TiB. On a 32-bit system this comes to 2 GiB or so (depending on kernel configuration).
So on a 64-bit system, you can allocate ~8 TiB, and a 32-bit system you can allocate ~2 GiB, and the overhead is what causes the problems. There is typically a small amount of overhead for each call to malloc or calloc.
See also Why malloc+memset is slower than calloc?