We know that the heap is an area of demand-zero memory that begins immediately after the uninitialized data area and grows upward (toward higher addresses). By demand-zero, it means the first time the CPU touches a virtual page in heap area, the corresponding physical page will be all zeros.
If that is the case, then why there is a function calloc used to initialize the allocated memory to zero? Why do demand-zero pages need to be initialized zero again if they will be zero already when accessed?
Because after you've used the space and released it with free(), it might be allocated again. If you don't use calloc(), there's no guarantee that the memory will be zeroed on the second time it is used. (Calling free() does not zero the space.)
calloc does not necessarily have to initialize the memory to zero by itself. The description of calloc says that:
The space is initialized to all bits zero.
but it does not say that it is calloc that does this, just that the memory is initialized to zero by some mechanism. This is unlike malloc:
The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate.
calloc guarantees that the memory is zeroed, and malloc does not. If the contents of the block are copy-on-write zero pages, then calloc may know not to zero it again and is faster than malloc + memset, as memset would not know that the memory was already zeroed (unless the compiler optimizes malloc + memset(..., 0, ...) to calloc); on the other hand if the block is reused, then calloc needs to zero it, even if the caller would not care about zeroing, therefore a malloc would be faster than calloc if no zeroing is needed, because then calloc would indeed do effectively malloc + memset
In short, it's more portable to set the memory to zero explicitly if that's what your application needs; and faster to leave it alone if it doesn't. You don't have to use calloc() to get zero'd memory -- you could just use malloc() and zero it yourself (e.g.,using memset). But calloc() will give you zero'd memory more quickly, if it can take advantage of a platform-specific feature to get it.
the 'heap' memory will contain what ever trash was in memory when your program was loaded, (just like the stack)
It is up to your application to zero memory. There are a few ways to do that.
calloc()
a loop in your code
memset()
This is in part historical, and in part as a useful feature
We had heap with uninitialised memory (which was not zero), so more quick to get memory. Then this was discovered that this is a security problem, so much later, you get memory cleared to zero (and never memory set by other processes). Do not assume all systems will do it (especially on small embedded CPU, where CPU and bus access are expensive (time and power).
calloc is very handy, when you allocate arrays (as you see, the signature is done for arrays). Often on arrays, you want to initialize values to zero. A loop is very slow, and static had already the initialization to zero. We have two possibilities: initialized memory with calloc or uninitialised memory with malloc.
Note: malloc doesn't guarantee to give you all zeros: it could give you already allocated (from your process) and freed memory. Just now new memory given by kernel is zeroed (e.g. with brk/sbrk, which is sometime called by malloc/calloc, in case of lack of free memory in existing heap memory). These are two different allocation of memory.
Related
I'm new to C and heap memory, below is my understanding about dynamic memory allocation, please correct me if I'm wrong:
Fact 1-When the first time calling malloc, malloc will call sbrk internally to move/grow the brk pointer(program break).
Fact 2-after calling malloc and free a couple of times, there could be free blocks between two allocated blocks, so if we call malloc again with a required size less than the size of free blocks, then this time malloc will not call sbrk internally, instead, malloc just modify one existing free block's structure(setting allocated bit ...etc) and return the address of this block.
There is nothing in the standard that says malloc() must move/grow the brk pointer to allocate memory. In fact, nowadays malloc() will often use mmap(..., MAP_ANONYMOUS, ...) internally to obtain memory.
It is likely that if there is a gap after repeated malloc() and free() calls, that the next malloc() might be able to allocate memory in the gap, such that no calls to sbrk() or mmap() are necessary. But again, there is no guarantee that this will actually happen.
I'm trying to figure out how many bytes in a block are taken up by the boundary tags. I have been told that when trying to malloc an adjacent block of memory, a "jump" will appear in assembly code, and I can use that to determine the size of the boundary tag. I've tried this:
int* arr = malloc(8);
arr++;
arr = malloc(8);
But there isn't any jump in assembly code. Am I "trying to malloc an adjacent block of memory"?
EDIT: I think he means a jump will appear between address value. I use the beginning of the second block of memory subtract the payload size of the first block. But I'm still confused, how could I malloc an adjacent block of memory?
Unless you're writing an actual memory allocator, you can't actually allocate two consecutive chunks of memory. If you want to see some pretty gnarly code which does this, have a look at the Illumos malloc https://github.com/illumos/illumos-gate/blob/master/usr/src/lib/libc/port/gen/malloc.c.
If you want to see how Illumos (and Solaris) handle the redzone between allocated blocks, you should trawl through https://github.com/illumos/illumos-gate/tree/master/usr/src/lib/libumem/common.
The memory consumed by malloc(3) requires, for proper management of the actually used memory, of some structures that must be dynamically allocated also. For this reason, many allocators just do allocate the space required for the management data adjacent to the block space dedicated to the user. This makes that normally two consecutive junks of memory allocated by malloc(2) show some gap in their addresses.
There are other reasons to see gaps, one fundamental is that malloc normally gives you aligned memory addresses, so it is warranted that the data you store on that memory will be properly aligned.
And of course, there can be implementations (normally when heap allocation should be more robust in respect to buffer overruns) that the memory dedicated to storage of management data is completely unrelated and apart off the final given memory. In this case you could observe no gaps between memory allocations on some cases.
Anyway, your code has serious bugs, let's see:
int* arr = malloc(8);
You had better here to acquire just the memory you need, using the sizeof operator, as in int *arr = malloc(sizeof *arr); instead.
arr++;
this statement is useless, as you are going to overwrite the value of arr (the pointer) with new assignment statement after it from malloc(), so it is of no use to increment the pointer value. You also are somewhat losing the returned value of the previous malloc() (which is essential in order to return the allocated memory, but read below).
arr = malloc(8);
Until here, you had the chance to --arr decrementing the value of arr in order to be capable of free(3) that block. But this statement overwrites the value stored in arr so the previous pointer value is overwritten by the new pointer. Memory you acquired on the first malloc has no way to be accessed again. This is what is commonly known as a memory leak, and is normally a serious error (very difficult to catch) on long run programs (like servers or system daemons). The program allocates a bunch of memory in the inner part of a loop, that is not returned back with a call to free(3), so the program begins growing and growing until it exhausts all the available memory.
A final note, I don't understand what did you mean with malloc adjacent block of memory. Did you believe that incrementing the pointer would make malloc() to give you a special block of memory?
First, malloc has no idea of what are you going to do with the pointer it gives to you.
But also, it doesn't know anything about the variable contents of the pointer you are assigning to (you can even not store it in a variable, and pass it as another parameter to another functions) So there's no possibility for malloc to know that you have incremented the pointer value, or even the pointer location, from its body.
I cannot guess your interpretation. It would be nice to know what has made you to think that you can control how malloc(3) selects the block of memory to give to you. You have no control on the internals of malloc() You just specify the amount of continous memory you want, and mallocs provides it, giving you a pointer pointing to the start of that block. You cannot assume that the next time you call malloc (with the same or different amount of memory) it will give you an adjacent block. It just can be completely unrelated (above or below in memory) to the previous given block. And you cannot modify that pointer, because you need it to call free(3) once you don't need the block anymore, with exactly the same pointer value that malloc(3) gave to you. If, for some reason you modify the pointer, you need to restore it to the original value to be capable of calling free(3). Lack to do so, you'll probably crash your program at the next call to free(3).
I just see a memory leak. Malloc 2 times into different vars 8 bytes of space and see if the difference is more than 8 bytes or 2 int.
So I have allocated 256 blocks in heap:
char* ptr1 = malloc(128);
char* ptr2 = malloc(128);
Now after I free ptr2 which I assume currently lies on top of the heap, the program break(the current location of the heap) does not decrease. However if I do another malloc the address returned by malloc is the same as the one that is freed.
So I have the following questions:
When I free a block why does not the program break decrease?
When I call free what exactly happens?How does it keep track of the freed memory so that next time I declare malloc the address is the same?
It's unspecified behavior. You can not rely on any single answer, unless you only care about one particular platform/os/compiler/libc combination. You did not specify an OS, and the C standard does not describe, or require any particular implementation. From C99 (I don't have the final published version of C11 yet):
7.20.3
The order and contiguity of storage allocated by successive calls to
the calloc, malloc, and realloc functions is unspecified. The pointer
returned if the allocation succeeds is suitably aligned so that it may
be assigned to a pointer to any type of object and then used to access
such an object or an array of such objects in the space allocated
(until the space is explicitly deallocated). The lifetime of an
allocated object extends from the allocation until the deallocation.
Each such allocation shall yield a pointer to an object disjoint from
any other object. The pointer returned points to the start (lowest
byte address) of the allocated space. If the space cannot be
allocated, a null pointer is returned. If the size of the space
requested is zero, the behavior is implementation- defined: either a
null pointer is returned, or the behavior is as if the size were some
nonzero value, except that the returned pointer shall not be used to
access an object.
This manual of GNU libc , might be of help.
Here's the gist
Occasionally, free can actually return memory to the operating system
and make the process smaller. Usually, all it can do is allow a later
call to malloc to reuse the space. In the meantime, the space remains
in your program as part of a free-list used internally by malloc.
When I free a block why does not the program break decrease?
I believe it doesn't decrease because that memory has already been given to the program.
When I call free() what exactly happens?
That section of memory is marked as allocatable, and its previous contents can be overwritten.
Consider this example...
[allocatedStatus][sideOfAllocation][allocatedMemory]
^-- Returned pointer
Considering this, the free() can then mark the [allocatedStatus] to false, so future allocations on the heap can use that memory.
How does it keep track of the free()d memory so that next time I declare
malloc() the address is the same?
I don't think it does. It just scanned for some free memory and found that previous block that had been marked as free.
Here is a rough idea how memory allocators work:
You have an allocator that has a bunch of "bins" ("free lists") which are just linked lists of free memory blocks. Each bin has a different block size associated with it (I.e.: you can have a list for 8 byte blocks, 16 byte blocks, 32 byte blocks, etc... Even arbitrary sizes like 7 or 10 byte blocks). When your program requests memory (usually through malloc()) the allocator goes to the smallest bin that would fit your data and checks to see if there are any free memory blocks in it. If not then it will request some memory from the OS (usually called a page) and cuts the block it gets back into a bunch of smaller blocks to fill the bin with. Then it returns one of these free blocks to your program.
When you call free, the allocator takes that memory address and puts it back into the bin (aka free list) it came from and everybody is happy. :)
The memory is still there to use so you don't have to keep paging memory, but with respect to your program it is free.
I believe it's entirely up to the operating system once you call free(), it may choose to immediately reclaim that memory or not care and just mark that memory segment as a possible acquisition for a later time (likely the same thing). To my knowledge that memory (if significant) shows up as available in the task manager right after free() on windows.
Keep in mind that the memory we are talking about here is virtual. So that means the operating system can tell you anything it wants and is likely not an accurate representation of the physical state of the machine.
Think about how you would manage memory allocation if you were writing an OS, you likely wouldn't want to do anything hasty that may waste resources. We are talking about 128 bytes here, would you want to waste valuable processing time handling it alone? It may be the reason for that behavior or not, at least plausible.
Do it in a loop and then free() in another loop or just allocate big chunks of memory, see what happens, experiment.
I was experimenting with malloc in C and I have observed that malloc is wasting some space after some memory has been allocated. Below is the piece of code I used to test malloc
#include <stdlib.h>
#include <string.h>
int main(){
char* a;
char* b;
a=malloc(2*sizeof(char));
b=malloc(2*sizeof(char));
memset(a,9,2);
memset(b,9,2);
return 0;
}
In the right-middle of the following picture(open the image in a new tab for clarity) you can see the memory contents;0x804b008 is the address pointed by variable 'a' and 0x804b018 is the memory pointed by variable 'b'. what is happening to memory between from 0x804b00a 0x804b017? The thing is even if I try to allocate 3*sizeof(char) instead of 2*sizeof(char) bytes of memory the memory layout is the same! So, is there something I am missing?
malloc() is allowed to waste as much space as it wants to - the standard doesn't specify anything about the implementation. The only guarantee you have is about alignment (ยง7.20.3 Memory management functions):
The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object and then used to access such an object or an array of such objects in the space allocated (until the space is explicitly deallocated).
Your implementation appears to return you minimum-8-byte-aligned pointers.
Memory Alignment! It's good for perfomance in x86 and mandatory in some architectures like ARM.
Most CPUs require that objects and variables reside at particular offsets in the system's memory. For example, 32-bit processors require a 4-byte integer to reside at a memory address that is evenly divisible by 4. This requirement is called "memory alignment". Thus, a 4-byte int can be located at memory address 0x2000 or 0x2004, but not at 0x2001. On most Unix systems, an attempt to use misaligned data results in a bus error, which terminates the program altogether. On Intel processors, the use of misaligned data is supported but at a substantial performance penalty. Therefore, most compilers automatically align data variables according to their type and the particular processor being used. This is why the size that structs and classes occupy is often larger than the sum of their members'
http://www.devx.com/tips/Tip/13265
The heap is handled by the implementation, not necessarily as you expect. The Standard explicitly doesn't guarantee anything about order or contiguity. There are two main things that cause more heap space to be used than you asked for.
First, allocated memory has to be aligned so that it's suitable for use by any sort of object. Typically, computers expect primitive data objects of N bytes to be allocated at a multiple of N, so the odds are you can't get malloc() to return a value that isn't a multiple of 8.
Second, the heap needs to be managed, so that free() allows reuse of the memory. This means that the heap manager needs to keep track of allocated and unallocated blocks, and their sizes. One practice is to stick some information in memory just before each block, so the manager can know what size block to free and where blocks are that might be reused. If that's what your system does, there will be more memory used between allocated blocks, and given alignment restrictions at 8 bytes it's likely you can't get allocations of less than 16 bytes.
Most modern malloc() implementations allocate in powers of two and have a minimum allocation size, to reduce fragmentation since oddball sizes could generally only be reused when enough contiguous allocations are free()d to make larger blocks. (It also speeds up coalescing contiguous allocations in general, IIRC.) Also keep in mind the block overhead; to get the block size you need to add some amount (8 in GNU malloc(), IIRC) for internal management uses.
malloc is only guaranteed to return you a block of memory that's at least as big as the size you give it. However, processors are generally more efficient when they're operating on blocks of memory that start at multiples of, say, 8 bytes in memory. Look up word size for more information on this.
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
c difference between malloc and calloc
Is there any situation where you would prefer malloc over calloc. i know both malloc and calloc allocate memory dynamically and that calloc also initializes all bits in alloted memory to zero.
From this i would guess its always better to use calloc over malloc. Or is there some situations where malloc is better? Performance may be?
If you need the dynamically allocated memory to be zero-initialized then use calloc.
If you don't need the dynamically allocated memory to be zero-initialized, then use malloc.
You don't always need zero-initialized memory; if you don't need the memory zero-initialized, don't pay the cost of initializing it. For example, if you allocate memory and then immediately copy data to fill the allocated memory, there's no reason whatsoever to perform zero-initialization.
calloc and malloc are functions that do different things: use whichever one is most appropriate for the task you need to accomplish.
Relying on calloc's zero-initialisation can be dangerous if you're not careful. Zeroing memory gives 0 for integral types and \0 for char types as expected. But it doesn't necessarily correspond to float/double 0 or NULL pointers.
You're normally allocating memory with the specific intent of storing something there. That means (at least most of) the space that's zero-initialized by calloc will soon be overwritten with other values. As such, most code uses malloc for a bit of extra speed with no real loss.
Nearly the only use I've seen for calloc was code that was (supposedly) benchmarking the speed of Java relative to C++. In the C++ version, it allocated some memory with calloc, then used memset to initialize the memory again in (what seemed to me) a fairly transparent attempt at producing results that favored Java.