I would like to ask you a question. I have the following code:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define XXX 1024*1024
int main()
{
int *p;
unsigned long x=0;
while (1)
{
//p = (int *) calloc (1,XXX);
p = (int *) malloc (XXX);
memset (p,0,XXX);
x++;
printf ("%lu MB allocated.\n",x);
sleep (1);
}
return 0;
}
If I run this code, everything runs as normal. Every second, a new MB is allocated in the memory. The problem I encounter is if I uncomment the calloc() line and comment the malloc() and memset() lines. From what I know, calloc() should initialize all bytes to zero in the allocated memory; the same thing that malloc() and memset() do.
When I run the code with calloc() (without the malloc() and memset()), an initial 1 MB is allocated (as it is normal), and then after some seconds (~10) another MB is allocated.
Why is this behaviour?
Thanks in advance!
From what I know, calloc() should initialize all bytes to zero in the allocated memory.
This is partly true based on my understanding of the calloc call.
It reserves the space but doesn't initialise all memory to zero. It will often or generally initialise one section to zero, and point all others to that; when memory is then modified or accessed within this block, it will initialise it to zero before using. It means that a calloc call of very large size doesn't set all of that memory to zero multiple times, but only when it actually needs to.
tl;dr: it's an OS theory trick where kernels will cheat. There's a longer description here: https://stackoverflow.com/a/2688522/2441252.
Related
I have a programs like this,
Program 1
#include <stdio.h>
#include <string.h>
#define ARRAY_SIZE 0x320204
int main ()
{
unsigned long long_array[ARRAY_SIZE];
memset (long_array, 0, ARRAY_SIZE);
return 0;
}
Program 2
#include <stdio.h>
#include <string.h>
#define ARRAY_SIZE 0x320204
int main ()
{
unsigned long long_array[ARRAY_SIZE] = {0};
//memset (long_array, 0, ARRAY_SIZE);
return 0;
}
Both Programs are giving Segmentation Fault.
Please clarify how to initialize this long size array. Thanks in Advance.
The segmentation fault is because you try to allocate on the stack 0x320204 longs, which may be too much. In decimal, this is 3277316 * sizeof(unsigned long), which on my machine is 3277316 * 8 = 26218528 bytes, i.e. around 25 Mbytes.
The second way of initializing it is the most common, it will zero all elements. The first method will only zero the first ARRAY_SIZE bytes, but remember that sizeof(unsigned long) is usually 8 bytes, so you need to zero sizeof(unsigned long) * ARRAY_SIZE bytes.
What you could do is use calloc, which is only one call, clearer to a lot of people, though not really a big deal, and uses the heap, which is the biggest benefit. Once you want to make really big array (or objects, or whatever), the heap is the place for you. Eventually, using the stack for this kind of stuff isn't practical, and will likely cause a stackoverflow error. Plus, you can pass this pointer returned from calloc to anything with having to worry what will happen to that memory allocated. This is unlike the stack, where it will probably be overwritten once that function returns, something that has happened to me plenty of times, creating bugs that are super hard to fix. There are downsides, however, to calloc like the fact that its pretty slow, but just as fast as memsetin this case. But if you don't really NEED all the bits to be set to 0, you could use malloc (much faster and more popular). The second downside to calloc or malloc is that you must call free on the pointers, otherwise you leak memory. Here's how to use calloc:
#include <stdlib.h> // or "#include <cstdlib>" in C++
/* Allocate space for array, set all bytes to 0,
and return pointer. Then assign to long_array.
*/
unsigned long* long_array = (unsigned long*) calloc(ARRAY_SIZE);
// When you're done with this variable:
free(long_array);
Easy as that. Both calloc and malloc are made for this kind of stuff. I hope this helped!
Note: Though these links are to a C++ website, you can still use these functions in pure C by including <stdlib.h>
According to the answer to this question:
Difference between malloc and calloc?
Isak Savo explains that:
calloc does indeed touch the memory (it writes zeroes on it) and thus you'll be sure the OS is backing the allocation with actual RAM (or swap). This is also why it is slower than malloc (not only does it have to zero it, the OS must also find a suitable memory area by possibly swapping out other processes)
So, I decided to try it myself:
#include <stdlib.h>
#include <stdio.h>
#define ONE_MB = 1048576
int main() {
int *p = calloc(ONE_MB, sizeof(int));
int n;
for(n = 0; n != EOF; n = getchar()) ; /* Gives me time to inspect the process */
free(p);
return 0;
}
After executing this application, Windows's Task Manager would tell me that only 352 KB were being used out of RAM.
It appears that the 1MB block I allocated is not being backed with RAM by the OS.
On the other hand, however, if I would call malloc and initialize the array manually:
#include <stdlib.h>
#include <stdio.h>
#define ONE_MB = 1048576
int main() {
int *p = malloc(sizeof(int) * ONE_MB);
int n;
/* Manual Initialization */
for(n = 0; n < ONE_MB; n++)
memory[n] = n;
for(n = 0; n != EOF; n = getchar()) ; /* Gives me time to inspect the process */
free(p);
return 0;
}
Task Manager would show me that there is actually 4.452KB of RAM being used by the application.
Was Isak incorrect about his argument? If so, what does calloc do then? Doesn't it zero the whole memory block, and therefore "touches" it, just as I did?
If that's the case, why isn't RAM being used in the first sample?
He was wrong in the point, that it is much slower because of it has to write 0 in the block first.
Any smart coded OS prepares such blocks for such purposes (where calloc() isn't the only case such blocks are used for)
and if you call calloc() it just assigns such a block of zeroed memory to your process instead of a uninitialized one as it >could< do by calling malloc().
So it handles such blocks of memory the same way. and if the compiler/OS decides you don't ever/yet need the full 1MB it also dont' gives you a full 1MB block of the zeroed memory.
In How far he was right:
If you heavily call calloc() and also use the memory, the OS could go out of zeroed memory which was probably prepared in some idle time.
This would be causing indeed the system to get a bit slower, as than the os is forced by a call to calloc() to write 0's in the block first.
But at all: There is no regulation about whether malloc/calloc have to allocate the memory on the call or just as you are using the memory. So your special example depends on the OS treatment.
I wish to free blocks of memory which I don't have pointers to. In my program, I call malloc sequentially, hoping that the memory created by malloc(1), malloc(4), malloc(5) is continuous. Then I free these memory when I only have the pointer to malloc(5). But I can't think of how this can be done; I cannot simply create a pointer that reference to the address of ptr[-5] and then free 5 bytes of memory? How can this be done?
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
int main(){
malloc(1);
malloc(4);
char* ptr = malloc(5);
free(ptr);
}
You cannot do what you want to do. You should not even try to do what you want to do.
Even if you work out exactly what malloc() is doing, your program would then be relying on undefined behavior. The behavior could change when a new version of the C library arrives, and your program would almost certainly fail if you compiled it using a different toolchain (switch from GNU C to Microsoft C or whatever).
Any time you allocate memory, you need to keep track of the pointer. If your program doesn't even know about the memory, there is no way to free it.
Keep track of your memory allocations. If you are designing data structures to be dynamically allocated, your design should include features to track them, such as keeping a list of addresses in a linked list or something.
If this seems like a lot of work, maybe consider using a managed language like C# or Java or Python or whatever.
free(void*)
[deallocate] A block of memory previously allocated by a call to malloc, calloc or realloc is deallocated, making it available again for further allocations.
If ptr does not point to a block of memory allocated with the above functions, it causes undefined behavior.
- http://www.cplusplus.com/reference/cstdlib/free/
There is no way.
But I can't think of how this can be done
That's because it is not possible. The blocks that you get back from malloc can come in truly arbitrary order. The only way to free a dynamically allocated block of memory is to keep a pointer to it accessible to your program. Anything else is undefined behavior.
Note: Implementations of malloc perform "bookkeeping" to figure out what kind of block you are releasing. While it is not impossible to hack into their implementation, there is no way of doing it in a standard-compliant, portable way.
You cannot create a [-5]...thing for a variety of reasons but the from a practical standpoint you have to remember that memory allocated with malloc() is coming off of the heap and not the stack so to "count" to it from somewhere else is difficult (since multiple calls to malloc are not guaranteed to be sequential).
What happens when a pointer loses its association to memory (or goes out of scope) without being freed is called a memory leak and without exhaustive techniques not readily available in C (Java's mark/sweep garbage collection for example, or mallocing the entire memory and scanning it or something) it is not possible to reclaim this memory.
So you cannot free memory in C when a pointer is not known.
First of all - as it seems you do not understand how malloc works - passing continuous numbers to malloc, won't make it allocate an array. malloc is defined as follows:
void* malloc (size_t size);
While an integer can be converted to size_t, it's still the number of bytes allocated, not the element number. If you want to allocate an array, do it as follows:
int* myDynamicArray = malloc(sizeof(int)*numberOfElements);
Then, you can access the elements by doing:
int i;
for(i=0;i<numberOfElements;i++)
printf("%d",myDynamicArray[i]);
Then, like others pointed out - you can deallocate the memory by calling the free function. free is defined as follows:
void free (void* ptr);
And you simply call it by doing:
free(myDynamicArray);
This is by no means an endorsement of what you have done, but it is possible assuming you know that the blocks were allocated continuously.
For example:
int main(){
char* ptr1=malloc(1);
char* ptr2=malloc(4);
char* ptr3=malloc(5);
// Verify that the memory is in fact continuous.
assert(ptr3==(ptr2+4));
assert(ptr3==(ptr1+5));
free(ptr3); // Frees 5 bytes at ptr3
free(ptr3-4); // Frees 4 bytes at ptr2
free(ptr3-5); // Frees 1 byte at ptr1
}
So, you if you have a pointer and know for a fact that you allocated a set of continuous bytes before it, you can simply offset the pointer with pointer arithmetic. It is highly dangerous and not recommended, but it is possible.
Edit:
I ran a test program and on my architecture, it allocated in 32 byte chunks, so ptr1+32==ptr2, and ptr2+32=ptr3. It did this for any chunks less than or equal to 24 bytes. So if I allocated 24 or less, then each ptr would be 32 bytes greater than the previous. If I allocated 25 or more, then it allocated an additional 16 bytes, making the total 48.
So, in my architecture, you'd need to be much more creative in how you generate your pointers using pointer arithmetic since it will not work as expected.
Here is an example program that works for all sizes of ptr1, ptr2, and ptr3 on my architecture.
#define ROUNDUP(number, multiple) (((number + multiple -1)/multiple)*multiple)
#define OFFSET(size) ((size < 24) ? 32 : ROUNDUP(size+8,16))
int main(int argc, char* argv[]){
char* ptr1, *ptr2, *ptr3;
int s1=atoi(argv[1]);
int s2=atoi(argv[2]);
int s3=atoi(argv[3]);
ptr1=(char*)malloc(s1);
ptr2=(char*)malloc(s2);
ptr3=(char*)malloc(s3);
fprintf(stdout, "%p %p %p\n", ptr1, ptr2, ptr3);
assert(ptr3==(ptr2+OFFSET(s2)));
assert(ptr2==(ptr1+OFFSET(s1)));
// Try to construct ptr2 from ptr3.
free(ptr3);
free(ptr3-OFFSET(s2));
free(ptr3-OFFSET(s2)-OFFSET(s1));
}
The following code when tested, gives output as
1
0
0
2
0
which is amazing because ptr[3], ptr[4] did not have any memory allocation. Although they stored value in them and prints it. I tried the same code for few larger i's in ptr[i] which again compiled successfully and gives result but for very large value of i, of the order of 100000,program get crashed. if calloc() allocates such a large memory on single call then it is not worth effective. So how calloc() works? Where is this discrepancy?
#include <stdio.h>
void main() {
int * ptr = (int *)calloc(3,sizeof(int));//allocates memory to 3 integer
int i = 0;
*ptr = 1;
*(ptr+3) = 2;//although memory is not allocated but get initialized
for( i =0 ; i<5 ; ++i){
printf("%d\n",*ptr++);
}
}
After that i tried this code which continuously runs without any output
#include <stdio.h>
void main() {
int * ptr = (int *)calloc(3,sizeof(int));
int i = 0;
*ptr = 1;
*(ptr+3) = 2;
//free(ptr+2);
for( ; ptr!=NULL ;)
{
//printf("%d\n",*ptr++);
i++;
}
printf("%d",i);
}
You are confounding two kinds of memory allocation: the kernel's and libc's.
When you ask malloc or calloc for 5 bytes, it does not turn around and ask the kernel for 5 bytes. That would take forever. Instead, the libc heap system obtains larger blocks of memory from the kernel, and subdivides it.
Therefore, when you ask for a small amount of memory, there is usually plenty of more accessible memory right after it that has not been allocated yet by libc, and you can access it. Of course, accessing it is an instant recipe for bugs.
The only time that referencing off the end will get a SIGnal is if you happen to be at the very end of the region acquired from the kernel.
I recommend that you try running your test case under valgrind for additional insight.
The code that you have has undefined behavior. However, you do not get a crash because malloc and calloc indeed often allocate more memory than you ask.
One way to tell how much memory you've got is to call realloc with increasing size, until the pointer that you get back is different from the original. Although the standard does not guarantee that this trick is going to work, very often it would produce a good result.
Here is how you can run this experiment:
int *ptr = calloc(1,sizeof(int));
// Prevent expansion of the original block
int *block = calloc(1, sizeof(int));
int *tmp;
int k = 1;
do {
tmp = realloc(ptr, k*sizeof(int));
k++;
} while (tmp == ptr);
printf("%d\n", k-1);
This prints 4 on my system and on ideone (demo on ideone). This means that when I requested 4 bytes (i.e. one sizeof(int) from calloc, I got enough space for 16 bytes (i.e. 4*sizeof(int)). This does not mean that I can freely write up to fourints after requesting memory for a singleint`, though: writing past the boundary of the requested memory is still undefined behavior.
calloc is allocating memory for 3 int's in the given snippet. Actually you are accessing unallocated memory. Accessing unallocated memory invokes undefined behavior.
calloc allocates only the amount of memory that you asked, which in your case in for 3 int variables
but it doesnt create a bound on the pointer that it has created (in your case ptr). so you can access the unallocated memory just by incrementing the pointer. thats exactly whats happening in your case..
I'm working on a homework assignment that requires me to write a program that leaks memory, keep track of how much memory it is leaking until it crashes.
My general thoughts for the program would be to continuously reassign a malloc pointer.
Here's my code so far:
char *oldMemory = malloc(125000); //1MB of memory.
char *newMemory = malloc(125000);
oldMemory = newMemory;
Is there are way to put this in a loop and repeatedly orphan a
certain amount of memory until the program can no longer allocate any memory and crashes?
How can I keep track of how much memory was leaked before the program crashed?
Thanks for your time and expertise!
Yes.
Count the size of the leaked allocations.
Don't forget to print the size leaked on each iteration - so you see the result even if the program crashes. The program should not crash if you test for failed allocations before accessing it.
Hence:
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
enum { ALLOCSIZE = 125000 };
int main(void)
{
long long size = 0;
char *space = malloc(ALLOCSIZE);
while (space != 0)
{
size += ALLOCSIZE;
printf("OK %lld\n", size);
memset(space, '\0', ALLOCSIZE);
}
return(0);
}
The Linux OOM might confuse things; it allows over-commitment of memory. You'd have to access the allocated memory before leaking it - hence the memset() (or you could use calloc() instead of malloc()).
when you allocate some block of memory in C you should be responsible to free it using
free() function.
if you want to make sure that your program won't crash due to some memory leaks.
you can use assert() function when asking for more memory or using the allocated memory.
and you should check on the returned pointer returned from malloc , if it's NULL so the allocation failed. >> Be careful in linux kernels malloc() won't return NULL ever,due to "memory overcommit"