I was curious whether there exists a dynamic memory allocation system that allows the programmer to free part of an allocated block.
For example:
char* a = malloc (40);
//b points to the split second half of the block, or to NULL if it's beyond the end
//a points to a area of 10 bytes
b = partial_free (a+10, /*size*/ 10)
Thoughts on why this is wise/unwise/difficult? Ways to do this?
Seems to me like it could be useful.
Thanks!
=====edit=====
after some research, it seems that the bootmem allocator for the linux kernel allows something similar to this operation with the bootmem_free call. So, I'm curious -- why is it that the bootmem allocator allows this, but ANSI C does not?
No there is no such function which allows parital freeing of memory.
You could however use realloc() to resize memory.
From the c standard:
7.22.3.5 The realloc function
#include <stdlib.h>
void *realloc(void *ptr, size_t size);
The realloc function deallocates the old object pointed to by ptr and returns a pointer to a new object that has the size specified by size. The contents of the new object shall be the same as that of the old object prior to deallocation, up to the lesser of the new and old sizes. Any bytes in the new object beyond the size of the old object have indeterminate values.
There is no ready-made function for this, but doing this isn't impossible. Firstly, there is realloc() . realloc takes a pointer to a block of memory and resizes the allocation to the size specified.
Now, if you have allocated some memory:
char * tmp = malloc(2048);
and you intend to deallocate the first, 1 K of memory, you may do:
tmp = realloc(foo, 2048-1024);
However, the problem in this case is that you cannot be certain that tmp will remain unchanged. Since, the function might just deallocate the entire 2K memory and move it elsewhere.
Now I'm not sure about the exact implementation of realloc, but from what I understand, the code:
myptr = malloc( x - y );
actually mallocs a new memory buffer of size x-y, then it copies the bytes that fit using memcpy and finally frees the original allocated memory.
This may create some potential problems. For example, the new reallocated memory may be located at a different address, so any past pointers you may have may become invalidated. Resulting in undefined runtime errors, segmentation faults and general debugging hell. So I would try to avoid resorting to this.
Firstly, I cannot think of any situation where you would be likely to need such a thing (when there exists realloc to increase/decrease the memory as mentioned in the answers).
I would like to add another thing. In whatever implementations I have seen of the malloc subsystem (which I admit is not a lot), malloc and free are implemented to be dependent on something called as the prefix byte(s). So whatever address is returned to you by malloc, internally the malloc subsystem will allocate some additional byte(s) of memory prior to the address returned to you, to store sanity check information which includes number of allocated bytes and possible what allocation policy you use (if your OS supports multiple mem allocation policies) etc. When you say something like free (x bytes), the malloc subsystem goes back to peek back into the prefix byte to sanity check and only if it finds the prefix in place does the free successfully happen. Therefore, it will not allow you to free some number of blocks starting in between.
Related
I'm starting to learn C programming and stumbled upon a situation that shows my lack of understanding of how memory allocation with malloc() works.
In a loop like the following:
// will only know `xlen` at runtime so I guess I have to use malloc here
char *x = malloc((xlen + 1) * sizeof(*x));
assert(x != NULL); // memory allocation failed
while (fgets(line, sizeof(line), fp) != NULL) {
strncpy(x, line, xlen); // copy first xlen characters of line
x[xlen] = '\0'; // ensure null terminating string
// do something with x
// x can now be "reused" for another line/loop iteration
// free(x) - errors out with "pointer being freed was not allocated"
}
fclose(fp)
free(x);
If the statement free(x) is called inside the loop then when I run this program I get an error message something like a.out(37575,0x7fff964ce3c0) malloc: *** error for object 0x7fab47c02630: pointer being freed was not allocated.
Why am I seeing this error message and what is causing it?
Will the memory address block of x be "re-used" at each iteration of the loop? (I'd say yes, and that would actually be what I wanted in this case) In this case, is it safe to only free the allocated memory of x outside the scope of the loop?
free(x); frees the memory allocated by char *x = malloc(.... It frees all of the memory, and you don't have to worry about how much memory that was, as it keeps track of that. You just have to call free(x); once as you correctly do. This is why you get the error if you free it inside the loop.
Does it mean that the memory address block of x will be "re-used" at
each iteration? (I'd say yes, and that would actually be what I wanted
in this case)
Yes, you use the same memory. It overwrites the memory every time.
In this case, is it safe to only free the allocated memory of x
outside the loop scope?
Yes, you have to free it outside of the loop. Because if you free it, then all of it is freed. If you did that inside the loop, then it would be undefined behavior to keep accessing that memory in following loop iterations.
The malloc() function provides a pointer to an area of memory that can then be used just like any pointer to a memory location.
The way to think of malloc() and free() is to consider it in terms of who owns a memory area. The malloc() function owns the memory that it is managing. When you call the malloc() function, or calloc() function, the function takes some of the memory it is managing and transfers ownership to you. When the function is called and the memory allocated there is a transfer of ownership from the memory management system of malloc() to you and at that point you own that memory and it is up to you to manage it. When the free() function is called to release the memory, you are transferring ownership of the memory area from you back to the memory management system of malloc().
Whenever a transfer of ownership is done, the entity that is giving up ownership is not supposed to do anything further with the memory area since it no longer owns it.
So when you do the malloc() you now own the memory and can use and reuse the memory area as much or as little as you want until such time that you call the free() function to give it back to malloc().
A pointer to a memory region given by malloc() is like any other pointer so far as using it is concerned.
Once you have transferred ownership of a memory area back to malloc() by using thee free() function, you can't call free() again with the same pointer without introducing an error. Once a memory area has been returned using free() you no longer have ownership of the memory so you shouldn't try to do anything else with it and that includes calling free() with the pointer again.
How malloc() manages memory is an implementation detail that will vary depending on the compiler and the C Runtime used. In some cases if you malloc() memory then free() the memory and then malloc() the same amount of memory again you may get lucky and get the same memory area again however it is not something you can count on or should expect.
malloc() provides you a pointer to a memory area that has a couple of guarantees. First of all the memory area provided will be at least as large as you requested. It may be larger but it won't be smaller than what you requested. Secondly the address provided will be on a memory boundary suitable for the machine architecture so that you can use the address provided for any variable type whether a built in such as int or double or a struct or an array of some type.
As the owner of the memory area provided by malloc() you are responsible for giving the memory area back to the malloc() functionality once you are done with it.
Your other primary responsibility is to respect the size of the memory area and to not write a larger data block into the area than the size requested when the malloc() was done. You are guaranteed only the amount of memory requested and writing a larger block of data into the memory will probably overwrite memory that you do not own but is owned by something else.
Unfortunately because the malloc() functionality, at least the optimized or non-debug version of the library, is designed with little as possible overhead, the functionality has few consistency and sanity checks in place. The malloc() functionality trusts that you will obey the rules and guidelines and only use what you own. And often when you break the rules and guidelines, you won't see the effect at the point where you have made a mistake but rather at some other point when some mysterious memory corruption is discovered and your program crashes.
You can use and reuse dynamically allocated memory as long as you have not called free() on it. If you free it and attempt to use it again, then you will have undefined behaviour.
You can continue to reusing memory (eg buf created by char *buf = calloc(sizeOfBuf); ) allocated to you on a single call to calloc(), until free(buf) is called, just as if buf was created statically, eg char buf[sizeOfBuf]; = 0. However, if the size of the buffer initially created by using calloc() needs to change, realloc() is available to do this, and is the preferable method. There are some caveats about using realloc however. Here is an example of using realloc, this one packaged into a function that resizes an existing dynamically allocated buffer, and takes care of some of the caveats:
realloc usage example:
// initial call to calloc
char *buf = calloc(sizeOfBuf);
if(buf)
{
while(someConditionIsTrue)
{
// read new content
//...
//new content needs to be added to buf
char *tmp = realloc(buf, newSizeOfBuffer);
if(!tmp)//if failed...
{
free(buf);//... free original buf to prevent memory loss
return NULL;// return null, caller must test for this
}
buf = tmp;//...else copy new mem location back to original pointer
//continue doing stuff with buf
}
//when done with buf, free it
free(buf);
}
One other suggestion, consider calloc() returns uninitialized memory, i.e. you own a space that contains whatever was occupying it when it was given to you. It is a good idea to follow that command with a method to clean up the buffer:
memset(buf, 0, sizeOfBuf);
Or use calloc().
If do the next:
int* array = malloc(10 * sizeof(int));
and them I use realloc:
array = realloc(array, 5 * sizeof(int));
On the second line (and only it), can it return NULL?
Yes, it can. There are no implementation guarantees on realloc(), and it can return a different pointer even when shrinking.
For example, if a particular implementation uses different pools for different object sizes, realloc() may actually allocate a new block in the pool for smaller objects and free the block in the pool for larger objects. Thus, if the pool for smaller objects is full, it will fail and return NULL.
Or it may simply decide it's better to move the block
I just used the following program to get size of actually allocated memory with glibc:
#include <stdlib.h>
#include <stdio.h>
int main()
{
int n;
for (n = 0; n <= 10; ++n)
{
void* array = malloc(n * sizeof(int));
size_t* a2 = (size_t*) array;
printf("%d -> %zu\n", n, a2[-1]);
}
}
and for n <= 6, it allocates 32 bytes, and for 7-10 it is 48.
So, if it shrank int[10] to int[5], the allocated size would shrink from 48 to 32, effectively giving 16 free bytes. Since (as it just has been noted) it won't allocate anything less than 32 bytes, those 16 bytes are lost.
If it moved the block elsewhere, the whole 48 bytes will be freed, and something could actually be put in there. Of course, that's just a science-fiction story and not a real implementation ;).
The most relevant quote from the C99 standard (7.20.3.4 The realloc function):
Returns
4 The realloc function returns a pointer to the new object (which may have the same value as a pointer to the old object), or a null pointer if the new object could not be allocated.
'May' is the key-word here. It doesn't mention any specific circumstances when that can happen, so you can't rely on any of them, even if they sound obvious at a first glance.
By the way, I think you could consider realloc() somewhat deprecated. If you'd take a look at C++, the newer memory allocation interfaces (new / delete and allocators) don't even support such a thing. They always expect you to allocate a new block. But that's just a loose comment.
The other answers have already nailed the question, but assuming you know the realloc call is a "trimming", you can wrap it with:
void *safe_trim(void *p, size_t n) {
void *p2 = realloc(p, n);
return p2 ? p2 : p;
}
and the return value will always point to an object of size n.
In any case, since the implementation of realloc knows the size of the object and can therefore determine that it's "trimming", it would be pathologically bad from a quality-of-implementation standpoint not to perform the above logic internally. But since realloc is not required to do this, you should do it yourself, either with the above wrapper or with analogous inline logic when you call realloc.
The language (and library) specification makes no such guarantee, just like it does not guarantee that a "trimming" realloc will preserve the pointer value.
An implementation might decide to implement realloc in the most "primitive" way: by doing an unconditional malloc for a new memory block, copying the data and free-ing the old block. Obviously, such implementation can fail in low-memory situations.
Don't count on it. The standard makes no such provision; it merely states "or a null pointer if the new object could not be allocated".
You'd be hard-pressed to find such an implementation, but according to the standard it would still be compliant.
I suspect there may be a theoretical possibility for failure in the scenario you describe.
Depending on the heap implementation, there may be no such a thing as trimming an existing allocation block. Instead a smaller block is allocated first, then the data is copied from the old one, and then it's freed.
For instance this may be the case with bucket-heap strategy (used by some popular heaps, such as tcmalloc).
A bit late, but there is at least one popular implementation which realloc() with a smaler size can fail: TCMalloc. (At least as far as i understand the code)
If you read the file tcmalloc.cc, in the function do_realloc_with_callback(), you will see that if you shrink enough (50% of alloced memory, otherwise it will be ignored), TCMalloc will alloc the new memory first (and possible fail) and then copy it and remove the old memory.
I do not copy the source code, because i am not sure if the copyrights (of TCMalloc and Stackoverflow) will allow that, but here is a link to the source (revision as at May 17, 2019).
realloc will not fails in shrinking the existing memory, so it will not return NULL. It can return NULL only if fails during expansion.
But shrinking can fail in some architecture, where realloc can be implemented in a different manner like allocating a smaller size memory separately and freeing the old memory to avoid fragmentation. In that case shrinking memory can return NULL. But its very rare implementation.
But its better to be in a safer side, to keep NULL checks after shrinking the memory also.
If do the next:
int* array = malloc(10 * sizeof(int));
and them I use realloc:
array = realloc(array, 5 * sizeof(int));
On the second line (and only it), can it return NULL?
Yes, it can. There are no implementation guarantees on realloc(), and it can return a different pointer even when shrinking.
For example, if a particular implementation uses different pools for different object sizes, realloc() may actually allocate a new block in the pool for smaller objects and free the block in the pool for larger objects. Thus, if the pool for smaller objects is full, it will fail and return NULL.
Or it may simply decide it's better to move the block
I just used the following program to get size of actually allocated memory with glibc:
#include <stdlib.h>
#include <stdio.h>
int main()
{
int n;
for (n = 0; n <= 10; ++n)
{
void* array = malloc(n * sizeof(int));
size_t* a2 = (size_t*) array;
printf("%d -> %zu\n", n, a2[-1]);
}
}
and for n <= 6, it allocates 32 bytes, and for 7-10 it is 48.
So, if it shrank int[10] to int[5], the allocated size would shrink from 48 to 32, effectively giving 16 free bytes. Since (as it just has been noted) it won't allocate anything less than 32 bytes, those 16 bytes are lost.
If it moved the block elsewhere, the whole 48 bytes will be freed, and something could actually be put in there. Of course, that's just a science-fiction story and not a real implementation ;).
The most relevant quote from the C99 standard (7.20.3.4 The realloc function):
Returns
4 The realloc function returns a pointer to the new object (which may have the same value as a pointer to the old object), or a null pointer if the new object could not be allocated.
'May' is the key-word here. It doesn't mention any specific circumstances when that can happen, so you can't rely on any of them, even if they sound obvious at a first glance.
By the way, I think you could consider realloc() somewhat deprecated. If you'd take a look at C++, the newer memory allocation interfaces (new / delete and allocators) don't even support such a thing. They always expect you to allocate a new block. But that's just a loose comment.
The other answers have already nailed the question, but assuming you know the realloc call is a "trimming", you can wrap it with:
void *safe_trim(void *p, size_t n) {
void *p2 = realloc(p, n);
return p2 ? p2 : p;
}
and the return value will always point to an object of size n.
In any case, since the implementation of realloc knows the size of the object and can therefore determine that it's "trimming", it would be pathologically bad from a quality-of-implementation standpoint not to perform the above logic internally. But since realloc is not required to do this, you should do it yourself, either with the above wrapper or with analogous inline logic when you call realloc.
The language (and library) specification makes no such guarantee, just like it does not guarantee that a "trimming" realloc will preserve the pointer value.
An implementation might decide to implement realloc in the most "primitive" way: by doing an unconditional malloc for a new memory block, copying the data and free-ing the old block. Obviously, such implementation can fail in low-memory situations.
Don't count on it. The standard makes no such provision; it merely states "or a null pointer if the new object could not be allocated".
You'd be hard-pressed to find such an implementation, but according to the standard it would still be compliant.
I suspect there may be a theoretical possibility for failure in the scenario you describe.
Depending on the heap implementation, there may be no such a thing as trimming an existing allocation block. Instead a smaller block is allocated first, then the data is copied from the old one, and then it's freed.
For instance this may be the case with bucket-heap strategy (used by some popular heaps, such as tcmalloc).
A bit late, but there is at least one popular implementation which realloc() with a smaler size can fail: TCMalloc. (At least as far as i understand the code)
If you read the file tcmalloc.cc, in the function do_realloc_with_callback(), you will see that if you shrink enough (50% of alloced memory, otherwise it will be ignored), TCMalloc will alloc the new memory first (and possible fail) and then copy it and remove the old memory.
I do not copy the source code, because i am not sure if the copyrights (of TCMalloc and Stackoverflow) will allow that, but here is a link to the source (revision as at May 17, 2019).
realloc will not fails in shrinking the existing memory, so it will not return NULL. It can return NULL only if fails during expansion.
But shrinking can fail in some architecture, where realloc can be implemented in a different manner like allocating a smaller size memory separately and freeing the old memory to avoid fragmentation. In that case shrinking memory can return NULL. But its very rare implementation.
But its better to be in a safer side, to keep NULL checks after shrinking the memory also.
I am trying to free dynamically allocated memory using free(), but I found that what it does is to have the argument pointer point to some new location, and leaving the previously-pointed-at location as it was, the memory is not cleared. And if I use malloc again, the pointer may point to this messy block, and it's already filled with garbage, which is really annoying..
I'm kinda new to C and I think delete[] in c++ doesn't have this problem. Any advise?
Thanks
By free the memory is just released from use. It is released from being allocated to you. it is not explicitly cleared. Some old contents might be present at those memory locations.
To avoid this, there are two solutions.
Solution 1:
You will need to do a memset after allocating memory using malloc.
Code Example:
unsigned int len = 20; // len is the length of boo
char* bar = 0;
bar= (char *)malloc(len);
memset(bar, 0, len);
Solution 2:
Or use, calloc() which initiliazes memory to 0 by default.
Code Example:
int *pData = 0;
int i = 10;
pData = (int*) calloc (i,sizeof(int));
I think delete[] in c++ doesn't have this problem.
No
It behaves exactly this same way. Unless you explicitly set the pointer to 0 the delete'd pointer will not be pointing to 0. So do always set the pointer to 0 after you delete it.
When should you use malloc over calloc or vice versa?
Since calloc sets the allocated memory to 0 this may take a little time, so you may probably want to use malloc() if that performance is an issue.(Ofcourse One most profile their usage to see if this really is a problem)
If initializing the memory is more important, use calloc() as it does that explicitly for you.
Also, some OS like Linux have an Lazy Allocation memory model wherein the returned memory address is a virtual address and the actual allocation only happens at run-time. The OS assumes that it will be able to provide this allocation at Run-Time.
The memory allocated by malloc is not backed by real memory until the program actually touches it.
While, since calloc initializes the memory to 0 you can be assured that the OS has already backed the allocation with actual RAM (or swap).
How about realloc?
Yes, similar behavior to malloc.
Excerpt From the documentation:
void * realloc ( void * ptr, size_t size );
Reallocate memory block
The size of the memory block pointed to by the ptr parameter is changed to the size bytes, expanding or reducing the amount of memory available in the block.
The function may move the memory block to a new location, in which case the new location is returned. The content of the memory block is preserved up to the lesser of the new and old sizes, even if the block is moved.If the new size is larger, the value of the newly allocated portion is indeterminate.
In case that ptr is NULL, the function behaves exactly as malloc, assigning a new block of size bytes and returning a pointer to the beginning of it.
In case that the size is 0, the memory previously allocated in ptr is deallocated as if a call to free was made, and a NULL pointer is returned.
You can use calloc( ) instead of malloc( ) to clear the allocated memory to zero.
Why is having newly allocated memory filled with garbage "really annoying"? If you allocate memory, presumably it's because you're going to use it for something -- which means you have to store some meaningful value into it before attempting to read it. In most cases, in well-written code, there's no reason to care what's in newly allocated memory.
If you happen to have a requirement for a newly allocated block of memory you can call memset after calling malloc, or you can use calloc instead of malloc. But consider carefully whether there's any real advantage in doing so. If you're actually going to use those all-bits-zero values (i.e., if all-bits-zero happens to be the "meaningful value" I mentioned above), go ahead and clear the block. (But keep in mind that the language doesn't guarantee that either a null pointer or a floating-point 0.0 is represented as all-bits-zero, though it is in most implementations they are.)
And free() doesn't "have the argument pointer point to some new location". free(ptr) causes the memory pointed to by ptr to be made available for future allocation. It doesn't change the contents of the pointer object ptr itself (though the address stored in ptr does become invalid).
int main()
{
char *s1, *sTemp;
s1 = (char*)malloc(sizeof(char)*7);
*(s1 + 0) = 'a';
*(s1 + 1) = 'b';
*(s1 + 2) = 'c';
*(s1 + 3) = 'd';
*(s1 + 4) = 'e';
*(s1 + 5) = 'f';
*(s1 + 6) = '\0';
sTemp = (s1 + 3);
free(sTemp); // shud delete d onwards. But it doesn't !!
return 0;
}
Hello,
In the C above code sTemp should point to the 3rd cell beyond s1 ( occupied by 'd')
So on calling free(sTemp) i expect to have something deleted from this location onwards.
( I purposely mention 'something' as the motive of my experiment initially was to find out till which location the free() ing works )
However i recieve a SIGABRT at the free().
How does free() know that this is not the start of the chunk. and correspondingly can we free up memory only from start of chunks? [ are they only the free-able pointers that free() can accept?? ]
Looking forward to replies :)
From the man pages
free() frees the memory space pointed
to by ptr, which must have been
returned by a previous call to
malloc(), calloc() or realloc().
Otherwise, or if free(ptr) has already
been called before, undefined
behaviour occurs.
Source: http://linux.die.net/man/3/free
About the actual question "how does free know...":
free knows that it is not at the start of the chunk because it maintains metadata that you don't know about. If you think about it, that's necessarily so, because otherwise it could never know how much memory to free, given only an address.
It is not specified how exactly the runtime keeps book of allocation sizes, only that you must never pass any pointer to free that did not come from a function of the malloc family.
Usually this works by malloc allocating a little more memory than needed[1] and writing some metadata in memory preceding the address that is returned. free then just looks at e.g. ptr[-8] or whatever (implementation detail!) to read this metadata.
It can then optionally do some consistency checks, and can determine the size of the memory block (one trivial consistency check that is probably always done is checking proper alignment of the pointer).
Having mentioned that, please please please, don't even think about playing with this metadata.
[1] It often does that anyway to satisfy alignment requirements and because of allocator internals (most allocators manage memory in different "buckets" of fixed size), so if you allocate, say, 4 bytes, you nominally get 4 bytes, but the runtime really allocated 8 or 16 bytes most of the time (without you knowing).
You can only free() pointers that were actually malloced (or calloced, realloced). If you try to free only a portion of memory by passing in a different pointer (even one that is "part" of another pointer), the C runtime will not be pleased (as you can see.)
I think you cannot free the memory because free doesn't know how much memory to free.Your program has information about the address malloc() returned and not for every address in this space.So you can free(s1) but not free(s1+3).Also you can handle your pointers as an array in this example:
s1[0]='a';
You can only free() memory that was previously malloced, calloced, or realloced, as dlev and Daniel have said.
The reason for this is implementation-dependent, but involves the method of keeping track of allocated memory.
Efficient memory allocation is a difficult problem because different memory allocation algorithms work well depending upon how memory is allocated: a few small chunks, half of which are freed, then slightly larger blocks being grabbed, etc.
The objective is to keep the size of the memory block used by the program at a minimum (usually this chunk will be a continuous block of virtual memory), while keeping the usage of the space within that block extremely high (few gaps between used segments of memory).
Remember that these blocks can't be moved except when being realloced, so there's always going to be some wasted space.
To minimize the waste, metadata about (at least) the size of the block is stored just before it. The memory allocator can look through the used blocks when determining how to handle a new request. If you pick a random memory location, whether part of a previously-allocated region or no, this metadata will not be present and free will be unable to determine what should be freed.
You cannot free that pointer the way you are doing, check this question and its answer: Memory Clobbering Error