According to the answer to this question:
Difference between malloc and calloc?
Isak Savo explains that:
calloc does indeed touch the memory (it writes zeroes on it) and thus you'll be sure the OS is backing the allocation with actual RAM (or swap). This is also why it is slower than malloc (not only does it have to zero it, the OS must also find a suitable memory area by possibly swapping out other processes)
So, I decided to try it myself:
#include <stdlib.h>
#include <stdio.h>
#define ONE_MB = 1048576
int main() {
int *p = calloc(ONE_MB, sizeof(int));
int n;
for(n = 0; n != EOF; n = getchar()) ; /* Gives me time to inspect the process */
free(p);
return 0;
}
After executing this application, Windows's Task Manager would tell me that only 352 KB were being used out of RAM.
It appears that the 1MB block I allocated is not being backed with RAM by the OS.
On the other hand, however, if I would call malloc and initialize the array manually:
#include <stdlib.h>
#include <stdio.h>
#define ONE_MB = 1048576
int main() {
int *p = malloc(sizeof(int) * ONE_MB);
int n;
/* Manual Initialization */
for(n = 0; n < ONE_MB; n++)
memory[n] = n;
for(n = 0; n != EOF; n = getchar()) ; /* Gives me time to inspect the process */
free(p);
return 0;
}
Task Manager would show me that there is actually 4.452KB of RAM being used by the application.
Was Isak incorrect about his argument? If so, what does calloc do then? Doesn't it zero the whole memory block, and therefore "touches" it, just as I did?
If that's the case, why isn't RAM being used in the first sample?
He was wrong in the point, that it is much slower because of it has to write 0 in the block first.
Any smart coded OS prepares such blocks for such purposes (where calloc() isn't the only case such blocks are used for)
and if you call calloc() it just assigns such a block of zeroed memory to your process instead of a uninitialized one as it >could< do by calling malloc().
So it handles such blocks of memory the same way. and if the compiler/OS decides you don't ever/yet need the full 1MB it also dont' gives you a full 1MB block of the zeroed memory.
In How far he was right:
If you heavily call calloc() and also use the memory, the OS could go out of zeroed memory which was probably prepared in some idle time.
This would be causing indeed the system to get a bit slower, as than the os is forced by a call to calloc() to write 0's in the block first.
But at all: There is no regulation about whether malloc/calloc have to allocate the memory on the call or just as you are using the memory. So your special example depends on the OS treatment.
Related
i'm doing some tests allocating and deallocating memory. This is the code i'm using:
#include <stdlib.h>
#include <stdio.h>
#define WAVE_SIZE 100000000
int main(int argc,char* argv[]){
int i;
int **p;
printf("%d allocs...\n",WAVE_SIZE);
// Malloc
printf("Allocating memory...\n");
p = (int**)malloc(WAVE_SIZE*sizeof(int*));
for(i = 0;i < WAVE_SIZE;i++)
p[i] = (int*)malloc(sizeof(int));
// Break
printf("Press a key to continue...\n");
scanf("%*s");
// Dealloc
printf("Deallocating memory...\n");
for(i = 0;i < WAVE_SIZE;i++)
free(p[i]);
free(p);
// Break
printf("Press a key to continue...\n");
scanf("%*s");
return 0;
}
During breaks i check the total memory used by the process and i don't see what i expect.
Until the first pause i see memory consumption increasing. However, at second pause I do not see it being released.
Is this a OS thing? What happens if my machine's load is high, i don't have free memory and another process try to alloc?
free does not necessarily release the memory back to the operating system. Most often it just puts back that memory area in a list of free blocks. These free blocks could be reused for the next calls to malloc.
When memory is allocated and later free'd, that memory still stays with the process but is marked as free so it can be allocated again. This is because otherwise the operating system have to alter the virtual memory mapping of the process each time you call malloc or free, which takes time.
free() merely tells the allocator that your program no long needs this block. The allocator may cache it for further allocation, or it may return it to the system by change the brk pointer. It all depends on the implementation.
When the system memory will be actually released depends on the OS.
Foe example in Windows, even if you close a program, the memory is not released at the same time. It is made in order to reuse it on the next program start.
The following code when tested, gives output as
1
0
0
2
0
which is amazing because ptr[3], ptr[4] did not have any memory allocation. Although they stored value in them and prints it. I tried the same code for few larger i's in ptr[i] which again compiled successfully and gives result but for very large value of i, of the order of 100000,program get crashed. if calloc() allocates such a large memory on single call then it is not worth effective. So how calloc() works? Where is this discrepancy?
#include <stdio.h>
void main() {
int * ptr = (int *)calloc(3,sizeof(int));//allocates memory to 3 integer
int i = 0;
*ptr = 1;
*(ptr+3) = 2;//although memory is not allocated but get initialized
for( i =0 ; i<5 ; ++i){
printf("%d\n",*ptr++);
}
}
After that i tried this code which continuously runs without any output
#include <stdio.h>
void main() {
int * ptr = (int *)calloc(3,sizeof(int));
int i = 0;
*ptr = 1;
*(ptr+3) = 2;
//free(ptr+2);
for( ; ptr!=NULL ;)
{
//printf("%d\n",*ptr++);
i++;
}
printf("%d",i);
}
You are confounding two kinds of memory allocation: the kernel's and libc's.
When you ask malloc or calloc for 5 bytes, it does not turn around and ask the kernel for 5 bytes. That would take forever. Instead, the libc heap system obtains larger blocks of memory from the kernel, and subdivides it.
Therefore, when you ask for a small amount of memory, there is usually plenty of more accessible memory right after it that has not been allocated yet by libc, and you can access it. Of course, accessing it is an instant recipe for bugs.
The only time that referencing off the end will get a SIGnal is if you happen to be at the very end of the region acquired from the kernel.
I recommend that you try running your test case under valgrind for additional insight.
The code that you have has undefined behavior. However, you do not get a crash because malloc and calloc indeed often allocate more memory than you ask.
One way to tell how much memory you've got is to call realloc with increasing size, until the pointer that you get back is different from the original. Although the standard does not guarantee that this trick is going to work, very often it would produce a good result.
Here is how you can run this experiment:
int *ptr = calloc(1,sizeof(int));
// Prevent expansion of the original block
int *block = calloc(1, sizeof(int));
int *tmp;
int k = 1;
do {
tmp = realloc(ptr, k*sizeof(int));
k++;
} while (tmp == ptr);
printf("%d\n", k-1);
This prints 4 on my system and on ideone (demo on ideone). This means that when I requested 4 bytes (i.e. one sizeof(int) from calloc, I got enough space for 16 bytes (i.e. 4*sizeof(int)). This does not mean that I can freely write up to fourints after requesting memory for a singleint`, though: writing past the boundary of the requested memory is still undefined behavior.
calloc is allocating memory for 3 int's in the given snippet. Actually you are accessing unallocated memory. Accessing unallocated memory invokes undefined behavior.
calloc allocates only the amount of memory that you asked, which in your case in for 3 int variables
but it doesnt create a bound on the pointer that it has created (in your case ptr). so you can access the unallocated memory just by incrementing the pointer. thats exactly whats happening in your case..
I would like to ask you a question. I have the following code:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define XXX 1024*1024
int main()
{
int *p;
unsigned long x=0;
while (1)
{
//p = (int *) calloc (1,XXX);
p = (int *) malloc (XXX);
memset (p,0,XXX);
x++;
printf ("%lu MB allocated.\n",x);
sleep (1);
}
return 0;
}
If I run this code, everything runs as normal. Every second, a new MB is allocated in the memory. The problem I encounter is if I uncomment the calloc() line and comment the malloc() and memset() lines. From what I know, calloc() should initialize all bytes to zero in the allocated memory; the same thing that malloc() and memset() do.
When I run the code with calloc() (without the malloc() and memset()), an initial 1 MB is allocated (as it is normal), and then after some seconds (~10) another MB is allocated.
Why is this behaviour?
Thanks in advance!
From what I know, calloc() should initialize all bytes to zero in the allocated memory.
This is partly true based on my understanding of the calloc call.
It reserves the space but doesn't initialise all memory to zero. It will often or generally initialise one section to zero, and point all others to that; when memory is then modified or accessed within this block, it will initialise it to zero before using. It means that a calloc call of very large size doesn't set all of that memory to zero multiple times, but only when it actually needs to.
tl;dr: it's an OS theory trick where kernels will cheat. There's a longer description here: https://stackoverflow.com/a/2688522/2441252.
i'm doing some tests allocating and deallocating memory. This is the code i'm using:
#include <stdlib.h>
#include <stdio.h>
#define WAVE_SIZE 100000000
int main(int argc,char* argv[]){
int i;
int **p;
printf("%d allocs...\n",WAVE_SIZE);
// Malloc
printf("Allocating memory...\n");
p = (int**)malloc(WAVE_SIZE*sizeof(int*));
for(i = 0;i < WAVE_SIZE;i++)
p[i] = (int*)malloc(sizeof(int));
// Break
printf("Press a key to continue...\n");
scanf("%*s");
// Dealloc
printf("Deallocating memory...\n");
for(i = 0;i < WAVE_SIZE;i++)
free(p[i]);
free(p);
// Break
printf("Press a key to continue...\n");
scanf("%*s");
return 0;
}
During breaks i check the total memory used by the process and i don't see what i expect.
Until the first pause i see memory consumption increasing. However, at second pause I do not see it being released.
Is this a OS thing? What happens if my machine's load is high, i don't have free memory and another process try to alloc?
free does not necessarily release the memory back to the operating system. Most often it just puts back that memory area in a list of free blocks. These free blocks could be reused for the next calls to malloc.
When memory is allocated and later free'd, that memory still stays with the process but is marked as free so it can be allocated again. This is because otherwise the operating system have to alter the virtual memory mapping of the process each time you call malloc or free, which takes time.
free() merely tells the allocator that your program no long needs this block. The allocator may cache it for further allocation, or it may return it to the system by change the brk pointer. It all depends on the implementation.
When the system memory will be actually released depends on the OS.
Foe example in Windows, even if you close a program, the memory is not released at the same time. It is made in order to reuse it on the next program start.
This is a followup to the question I just asked here.
I've created a simple program to help myself understand memory allocation, malloc() and free(). Notice the commented out free line. I created an intentional memory leak so I can watch the Windows reported "Mem Usage" bloat slowly to 1GB. But then I found something stranger. If I comment out the loop just above the free line, so that I don't initialize my storage block with random ints, it appears that the space doesn't actually get "claimed" by the program, from the OS. Why is this?
Sure, I haven't initialized it the block, but I have claimed it, so shouldn't the OS still see that the program is using 1GB, whether or not that GB is initialized?
#include <stdio.h>
#include <stdlib.h>
void alloc_one_meg() {
int *pmeg = (int *) malloc(250000*sizeof(int));
int *p = pmeg;
int i;
// for (i=0; i<250000; i++) /* removing this loop causes memory to not be used? */
// *p++ = rand();
// free((void *)pmeg); /* removing this line causes memory leak! */
}
main()
{
int i;
for (i=0; i<1000; i++) {
alloc_one_meg();
}
}
Allocated memory can be in two states in Windows: Reserved, and Commited (see the documentation of VirtualAlloc about MEM_RESERVE: "Reserves a range of the process's virtual address space without allocating any actual physical storage in memory or in the paging file on disk.").
If you allocate memory but do not use it, it remains in the Reserved state, and the OS doesn't count that as used memory. When you try to use it (whether it is only on write, or on both read and write, I do not know, you might want to do a test to find out), it turns into Commited memory, and the OS counts it as used.
Also, the memory allocated by malloc will not be full of 0's (actually it may happen to be, but it's not guaranteed), because you have not initialised it.
it could be compiler optimization : the memory is not used at all, so a possible optimization is to not allocate this memory depening on compiler and on optimization options.
i tested your code, the line : free((void *)pmeg); does not cause any memory leak for me.