I created some utilities which
help me to handle the management of a DinamicList. In the section that I
use to handle the removing of a element in a list, if there is
a element added that is stored in the stack, when I call free()
an undefined behaviour is reached.
Surfing on the net I found out
that there aren't ways to determine whether a pointer points to stack
memory or heap memory.
So I think that to solve this problem I have to handle
the error generated from free(). Is there a way to handle that exception when I call free()?
No.
You need to not call free() for non heap pointers. Easiest way is let whoever allocated the memory take care of freeing it. I.e. your utilities look after whatever memory they allocate but someone else looks after the memory passed to your utilities.
Although "malloc" and "free" are described in terms of creating allocations and destroying them, their actual role is the reverse of that. The "malloc()" function takes some memory from a pool and indicates that the memory manager is not allowed to use any of the bytes within the allocated range (though bytes that were outside that range, including those immediately preceding and following the allocation, remain available to it). The "free()" function adds memory back to the pool, making it available for future use.
In some allocation systems, the function that releases memory accepts an argument indicating how much memory is being released; others may require that each pool only be used to dispense fixed-size objects. Some of the systems that do such things would allow code to add any chunk of memory that a program won't need for any other purpose to a pool by simply "releasing" it, the memory manager knowing or caring whether the memory came from the pool in the first case. Indeed, in some such systems that may be how the pools get created in the first place: initialize a descriptor for an empty memory pool, and then "release" chunks of storage into it as convenient.
Such an approach to adding storage to a memory pool can't work in C, however, since the only way "free" can know how much memory to add to the pool is to make use of information that was stored somewhere by "malloc". Generally, the information is stored in the space immediately prior to the storage malloc() gives the application. If a pointer wasn't produced by malloc(), the storage immediately preceding the storage described by the pointer won't contain the information malloc() needs, but will more likely contain a pattern of bytes that looks like it was created by an allocation of some meaningless size. This would have the effect of inviting the memory manager to do whatever it likes with a large chunk of storage that will likely extend beyond the boundaries of the object whose address was passed to it. Hilarity is likely to ensue when the memory manager takes the application up on that invitation.
Related
i do not seem to find an answer to this question. Why you cant free up an individual adress is it because the space needs to be continuous? and if this is the answer then why fragmentation occurs on Hard-Disks
Can you free individual memory adresses of an array allocated dynamically?
If the memory is at the end of an array, you can free off the unneeded excess by performing a realloc to a smaller size, with the caveat that you may actually get a new pointer to new memory with the prefix contents copied into it, and the original memory freed in its entirety.
Otherwise, no. The free interface is defined to only accept addresses returned from malloc, calloc or realloc.
Why you cant free up an individual adresss is it because the space needs to be continuous?
Well, the direct answer is that there is no interface defined to do so. There is no way to tell free how much of the pointer you passed in should be freed. If you want to free all memory to the end of the allocated block, realloc does that.
If contiguity is not important to your program, just use separate allocations for each array element, and free them individually.
and if this is the answer then why fragmentation occurs on Hard-Disks
One way to imagine a scenario of fragmentation on a file system is that if three files are created one after another, and then the middle one is deleted, there is now a hole between two files.
|---File 1---|--- Hole ---|---File 3---|
Now suppose a new file is created, so it starts out inside the hole between the two files, but as it grows, it cannot fit in the hole, so now the rest of the file is after File 3. In this case, we would say the new file is fragmented.
|---File 1---|---File 4...|---File 3---|...File 4---|
This happens on "Hard-Drives" because a filesystem is designed that way: allow a large file to span the available holes in the physical medium.
A RAM disk used for a filesystem would also eventually have fragmented files.
A non-contiguous data structure could be considered to be "fragmented", e.g., a linked-list or a tree, but that is by design. An array is considered contiguous by its very definition. However, files on a filesystem are not arrays.
Broadly, the reason you cannot release individual portions of allocated memory is that it is not useful enough to justify writing the software to support it.
The C standard specifies services provided by malloc, free, realloc, and related routines. The only provisions it makes for releasing space are by using free to release an allocation and by using realloc to replace an allocation with a smaller one.
C implementations are free to extend the C standard by providing services to release portions of allocated space. However, I am not aware of any that have done so. If a program were allowed to free arbitrary pieces of memory, the memory management software would have to keep track of all of them. That requires extra data and time. Additionally, it can interfere with some schemes for managing memory. Memory management software might organize memory so that allocations of particular sizes can be satisfied quickly out of specialized pools, and having to take back an arbitrary sized portion that was part of a specialized pool could be a problem.
If there were a demand for such a service, it could be written. But programs and algorithms have evolved over the years to use the existing services, and there does not seem to be much need to release individual portions of allocations. Generally, if a program is going to work with many objects that it might free individually, it allocates them individually. This is common when building data structures of all sorts, using pointers to construct trees, hashed arrays of lists or other structures, and so on. Data structures are often built out of individual nodes that can be allocated or freed individually. So there is little need to carve individual pieces to be released out of larger allocations.
The organization of memory has very little to do with the organization of data on hard disk or other storage media. Data is generally transferred between arbitrary places on disk and arbitrary places in memory as needed. In a variety of circumstances, files are “memory mapped,” meaning that the contents of a file are made visible in memory so that one can read the file contents by reading memory and one can modify the file by modifying memory. However, even in this situation, there is not generally any relationship between where the blocks of the file are on disk and where the blocks of the file are in memory. The blocks of a file are managed by the file system and are not necessarily contiguous, and the blocks in memory are managed by the memory system and may be rearranged arbitrarily with support from virtual memory hardware.
First question NO as you can only free the whole memory allocated by one malloc family function
Fragmentation of hard disks does not have anything common with the memory allocations.
Memory allocation is handled as seemingly continuous blocks of memory (it might not be in physical memory though, but that's not relevant).
There is no simple way to "cut a hole" in a single memory allocation, but you could do something like this:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define ARRAY_LEN 11
int main (void)
{
char *array;
array = (char *) malloc(ARRAY_LEN);
strcpy(array,"0123456789");
printf("%s\n",array);
//Remove the 5th element:
memmove(&array[5], &array[5+1], ARRAY_LEN-5);
array = realloc(array, ARRAY_LEN-1);
printf("%s\n",array);
free(array);
return 0;
}
Some Linux filesystems allows for "punching holes" in files, so with a mmap'ed file, you might be able to use the fallocate systemcall on it while using it as an array in memory.
Can you free individual memory adresses of an array allocated dynamically?
You seem to recognize that the answer is "no", because you follow up with
Why you cant free up an individual adress is it because the space needs to be continuous?
Each individual allocation is continuous, but the union of all dynamically-allocated space is by no means certain to be continuous. More on this later.
At the most practical level, you cannot free chunks of a larger allocation because C provides no mechanism for doing so. In particular, among the specifications for the free() function is:
if the argument does not match a pointer earlier returned by a memory management function, or if the space has been deallocated by a call to free or realloc, the behavior is undefined.
Thus, free() exhibits UB if its argument is a pointer into the interior of an allocated block.
Note also that free() accepts only one parameter. It makes no provision for the caller to specify the amount of memory to free, so the memory-management subsystem has to figure that out from the argument argument presented. That's fine for the operating model that one frees only whole, previously-allocated blocks, but it does not easily support freeing an allocation in multiple pieces.
Furthermore, consider that although you cannot free specific chunks of a larger allocation, you can use realloc() to reduce the the size of an allocation (which may also involve moving it).
Anything beyond that is in the realm of implementation-specific behavior, but do bear in mind that
it is very common for allocation to be performed and accounted for in terms of multi-byte blocks -- for example, multiples of 16 bytes -- regardless of the specific sizes requested. An implementation that works this way cannot under any circumstances free partial blocks, though one could imagine being able to free individual blocks from a larger allocation.
some implementations store memory management metadata adjacent to the dynamically-allocated space presented to the program. In such an implementation, it is not useful to free pieces of a larger allocation because they cannot, in general, be reused until the whole allocation is freed, for there is no available place for the needed metadata.
and if this is the answer then why fragmentation occurs on Hard-Disks
You don't need to free allocations in pieces to get memory fragmentation. It can suffice to perform multiple allocations and afterward free only some of them. This is a real issue that can degrade performance and even cause programs to fail.
With that said, however, file systems typically use different methods and data structures for tracking their metadata than do C memory-management implementations, and the underlying hardware has different characteristics and behavior, so there's really no justification for forming expectations about the behavior and capabilities of one variety of storage based on the behavior and capabilities of the other.
When we use malloc() we provide a size in byte.
When we use free() we provide nothing.
This is because the OS of course knows about it already, it must have stored the information somewhere.
By the way, also our software must remember how many memory blocks it has requested, so that we can (for instance) safely iterates starting from the pointer and going ahead.
So, my question is: isn't this redundant? Can't we simply ask the OS the size of the memory pointed by a given pointer since it knows it? And if not, why not?
When we use malloc() we provide a size in byte. When we use free() we
provide nothing. This is because the OS of course knows about it
already, it must have stored the information somewhere.
Even though it gives you memory and it keeps track of what memory range belongs to your process, the OS doesn't concern itself with the internal details of your memory. malloc stores the size of the allocated chunk in its own place, also reserved inside your process (usually, it's a few bytes before the logical address returned by malloc). free simply reads that reserved information and deallocates automatically.
By the way, also our software must remember how many memory blocks it
has requested, so that we can (for instance) safely iterates starting
from the pointer and going ahead.
So, my question is: isn't this redundant? Can't we simply ask the OS
the size of the memory pointed by a given pointer since it knows it?
And if not, why not?
Given the above, it is redundant to store that information, yes. But you pretty much have to store it, because the way malloc does its book-keeping is an implementation detail.
If you know how your particular implementation works and you want to take that risk for your software, you are free (no pun intended) to do it. If you don't want to base your logic on an implementation detail (and you'd be right not to want to), you'll have to do this redundant book-keeping side-by-side with malloc's own book-keeping.
No, it's not redundant. malloc() manages, in cooperation with free() and a few other functions, a zillion tiny, individually addressed blocks within relatively large blocks which are generally obtained with sbrk(). The OS only knows about the large range(s), and has no clue which tiny block within it are in use or not. To add to the differences, sbrk() only lets you move the end of your data segment, not split it into parts to free independently. Though one could allocated memory using sbrk exclusively, you would be unable to free arbitrary chunks for reuse, or coalesce smaller chunks into larger ones, or split chunks without writing a bunch of bookkeeping code for this purpose - which ends up essentially being the same as writing malloc. Additionally, using malloc/free/... allows you to call sbrk only rarely, which is a performance bonus since sbrk is a system call with special overhead.
When we use free() we provide nothing.
Not quite true; we provide the pointer that was returned by malloc.
Can't we simply ask the OS the size of the memory pointed by a given pointer since it knows it?
Nope. Pointers are simply addresses; apart from their type, they carry no information about the size of the object they point to. How malloc/calloc/realloc and free keep track of object sizes and allocated vs. free blocks is up to the individual implementation; they may reserve some space immediately before the allocated memory to store the size, they may build an internal map of addresses and sizes, or they may do something else completely.
It would be nice if you could query a pointer for the size of the object it points to; unfortunately, that's simply not a feature of the language.
I am trying to undestand the C functions malloc and free. I know this has been discussed a lot on StackOverflow. However, I think I kind of know what these functions do by now. I want to know why to use them. Let's take a look at this piece of code:
int n = 10;
char* array;
array = (char*) malloc(n * sizeof(char));
// Check whether memory could be allocated or not...
// Do whatever with array...
free(array);
array = NULL;
I created a pointer of type char which I called array. Then I used malloc to find a chunk of memory that is currently not used and (10 * sizeof(char)) bytes large. That address I casted to type char pointer before assigning it to my previously created char pointer. Now I can work with my char array. When I am done, I'll use free to free that chunk of memory since it's not being used anymore.
I have one question: Why wouldn't I just do char array[10];? Wikipedia has only one small sentence to give to answer that, and that sentence I unfortunately don't understand:
However, the size of the array is fixed at compile time. If one wishes to allocate a similar array dynamically...
The slide from my university is similarily concise:
It is also possible to allocate memory from the heap.
What is the heap? I know a data structure called heap. :)
However, I've someone could explain to me in which case it makes sense to use malloc and free instead of the regular declaration of a variable, that'd be great. :)
C provides three different possible "storage durations" for objects:
Automatic - local storage that's specific to the invocation of the function it's in. There may be more than one instance of objects created with automatic storage, if a function is called recursively or from multiple threads. Or there may be no instances (if/when the function isn't being called).
Static - storage that exists, in exactly one instance, for the entire duration of the running program.
Allocated (dynamic) - created by malloc, and persists until free is called to free it or the program terminates. Allocated storage is the only type of storage with which you can create arbitrarily large or arbitrarily many objects which you can keep even when functions return. This is what malloc is useful for.
First of all there is no need to cast the malloc
array = malloc(n * sizeof(char));
I have one question: Why wouldn't I just do char array[10];?
What will you do if you don't know how many storage space do you want (Say, if you wanted to have an array of arbitrary size like a stack or linked list for example)?
In this case you have to rely on malloc (in C99 you can use Variable Length Arrays but for small memory size).
The function malloc is used to allocate a certain amount of memory during the execution of a program. The malloc function will request a block of memory from the heap. If the request is granted, the operating system will reserve the requested amount of memory.
When the amount of memory is not needed anymore, you must return it to the operating system by calling the function free.
In simple: you use an array when you know the number of elements the array will need to hold at compile time. you use malloc with pointers when you don't know how many elements the array will need to be at compile time.
For more detail read Heap Management With malloc() and free().
Imagine you want to allocate 1,000 arrays.
If you did not have malloc and free... but needed a declaration in your source for each array, then you'd have to make 1,000 declarations. You'd have to give them all names. (array1, array2, ... array1000).
The idea in general of dynamic memory management is to handle items when the quantity of items is not something you can know in advance at the time you are writing your program.
Regarding your question: Why wouldn't I just do char array[10];?. You can, and most of the time, that will be completely sufficient. However, what if you wanted to do something similar, but much much bigger? Or what if the size of your data needs to change during execution? These are a few of the situations that point to using dynamically allocated memory (calloc() or malloc()).
Understanding a little about how/when the stack and heap are used would be good: When you use malloc() or calloc(), it uses memory from the heap, where automatic/static variables are given memory on the stack, and are freed when you leave the scope of that variable, i.e the function or block it was declared in.
Using malloc and calloc become very useful when the size of the data you need is not known until run-time. When the size is determined, you can easily call one of these to allocate memory onto the heap, then when you are finished, free it with free()
Regarding What is the heap? There is a good discussion on that topic here (slightly different topic, but good discussion)
In response to However, I've someone could explain to me in which case it makes sense to use malloc() and free()...?
In short, If you know what your memory requirements are at build time (before run-time) for a particular variable(s), use static / automatic creation of variables (and corresponding memory usage). If you do not know what size is necessary until run-time, use malloc() or calloc() with a corresponding call to free() (for each use) to create memory. This is of course a rule-of-thumb, and a gross generalization. As you gain experience using memory, you will find scenarios where even when size information is known before run-time, you will choose to dynamically allocate due to some other criteria. (size comes to mind)
If you know in advance that you only require an array of 10 chars, you should just say char array[10]. malloc is useful if you don't know in advance how much storage you need. It is also useful if you need storage that is valid after the current function returns. If you declare array as char array[10], it will be allocated on the stack. This data will not be valid after your function returns. Storage that you obtain from malloc is valid until you call free on it.
Also, there is no need to cast the return value of malloc.
Why to use free after malloc can be understood in the way that it is a good style to free memory as soon as you don't need it. However if you dont free the memory then it would not harm much but only your run time cost will increase.
You may also choose to leave memory unfreed when you exit the program. malloc() uses the heap and the complete heap of a process is freed when the process exits. The only reason why people insist on freeing the memory is to avoid memory leaks.
From here:
Allocation Myth 4: Non-garbage-collected programs should always
deallocate all memory they allocate.
The Truth: Omitted deallocations in frequently executed code cause
growing leaks. They are rarely acceptable. but Programs that retain
most allocated memory until program exit often perform better without
any intervening deallocation. Malloc is much easier to implement if
there is no free.
In most cases, deallocating memory just before program exit is
pointless. The OS will reclaim it anyway. Free will touch and page in
the dead objects; the OS won't.
Consequence: Be careful with "leak detectors" that count allocations.
Some "leaks" are good!
Also the wiki has a good point in Heap base memory allocation:-
The heap method suffers from a few inherent flaws, stemming entirely
from fragmentation. Like any method of memory allocation, the heap
will become fragmented; that is, there will be sections of used and
unused memory in the allocated space on the heap. A good allocator
will attempt to find an unused area of already allocated memory to use
before resorting to expanding the heap. The major problem with this
method is that the heap has only two significant attributes: base, or
the beginning of the heap in virtual memory space; and length, or its
size. The heap requires enough system memory to fill its entire
length, and its base can never change. Thus, any large areas of unused
memory are wasted. The heap can get "stuck" in this position if a
small used segment exists at the end of the heap, which could waste
any magnitude of address space, from a few megabytes to a few hundred.
struct element {
unsigned long int ip;
int type;
int rtt;
struct element * next;
struct element * edge;
};
I have a linked list. I create new nodes using malloc.
I tried to free up memory using free (ptr to node)
but when I run the traverse function again, I can traverse the linked list and the rtt value is correct as well as the next and edge pointers as I can follow the linked list. ONly the ip value is corrupted. why is this?
The behaviour of malloc() and free() depends heavily on the operating system and C library that you are using. In most implementations there are actually two memory allocators in play:
The OS memory allocator, which uses the virtual memory facilities of the processor to provide a process with its own address space and maps physical memory pages into that address space for use.
The C library memory allocator, which is in fact part of the application code and uses the pages provided by the OS to provide fine-grained memory management facilities, as provided by malloc() and free().
In general, calling free() does one or more of the following:
It marks the pointed-to memory area as free in the C memory allocator. This allows that memory to be reused. free() does not zero-out freed memory.
It may return memory to the OS, depending on the settings of the C memory allocator and whether it is actually possible to free that part of the heap. If memory is not returned to the OS, it can be reused by future malloc() calls by the same application.
If you try to access memory that has been freed, usually one of three things will happen:
The memory has been returned to the OS, and your program, typically, crashes. If you ask me, that is probably the best scenario - you have a problem, sure, but you know it.
The memory has not been reused, therefore your old data is still there. Your program goes on as if nothing was wrong. This is in my opinion the worst case scenario. Your code appears to work correctly and, if Murphy has a say in this, it will continue to do so until it reaches your end users - then it will fail spectacularly.
The memory has been reused by your program, and your code will start messing around with its own data. If you are careful (and lucky?), you will probably notice that the results are off. If not, well...
If you are on Linux/Unix Valgrind is a good tool to catch memory management problems like this. There are also replacement libraries for the C memory allocator, such as DUMA that will also allow you to detect such issues.
Memory is not wiped when you free it - that would be a waste of processor time. It is just allocated to the "free list". That is why your data is still there
Whenever you free a block, you should set the corresponding pointer to NULL, so you don't accidentally reference it - it could be reused at any time.
Actually free doesn't delete anything, it just tells the OS it can use that memory again, for example next time you call malloc() it could overwrite some of your nodes.
Freeing the memory releases it for reuse. It doesn't necessarily destroy the data that was at that location. It is up to you not to access a memory region that has been released because the behavior is undefined (i.e. usually very bad).
If you want the data destroyed for some strange reason, then overwrite the memory area prior to freeing it (e.g. memset(buf, 0, len)).
Given a pointer to some variable.. is there a way to check whether it was statically or dynamically allocated??
Quoting from your comment:
im making a method that will basically get rid of a struct. it has a data member which is a pointer to something that may or may not be malloced.. depending on which one, i would like to free it
The correct way is to add another member to the struct: a pointer to a deallocation function.
It is not just static versus dynamic allocation. There are several possible allocators, of which malloc() is just one.
On Unix-like systems, it could be:
A static variable
On the stack
On the stack but dynamically allocated (i.e. alloca())
On the heap, allocated with malloc()
On the heap, allocated with new
On the heap, in the middle of an array allocated with new[]
On the heap, within a struct allocated with malloc()
On the heap, within a base class of an object allocated with new
Allocated with mmap
Allocated with a custom allocator
Many more options, including several combinations and variations of the above
On Windows, you also have several runtimes, LocalAlloc, GlobalAlloc, HeapAlloc (with several heaps which you can create easily), and so on.
You must always release memory with the correct release function for the allocator you used. So, either the part of the program responsible for allocating the memory should also free the memory, or you must pass the correct release function (or a wrapper around it) to the code which will free the memory.
You can also avoid the whole issue by either requiring the pointer to always be allocated with a specific allocator or by providing the allocator yourself (in the form of a function to allocate the memory and possibly a function to release it). If you provide the allocator yourself, you can even use tricks (like tagged pointers) to allow one to also use static allocation (but I will not go into the details of this approach here).
Raymond Chen has a blog post about it (Windows-centric, but the concepts are the same everywhere): Allocating and freeing memory across module boundaries
The ACE library does this all over the place. You may be able to check how they do it. In general you probably shouldn't need to do this in the first place though...
Since the heap, the stack, and the static data area generally occupy different ranges of memory, it is possible with intimate knowledge of the process memory map, to look at the address and determine which allocation area it is in. This technique is both architecture and compiler specific, so it makes porting your code more difficult.
Most libc malloc implementations work by storing a header before each returned memory block which has fields (to be used by the free() call) which has information about the size of the block, as well as a 'magic' value. This magic value is to protect against the user accidently deleting a pointer which wasn't alloc'd (or freeing a block which was overwritten by the user). It's very system specific so you'd have to look at the implementation of your libc library to see exactly what magic value was there.
Once you know that, you move the given pointer back to point at header and then check it for the magic value.
Can you hook into malloc() itself, like the malloc debuggers do, using LD_PRELOAD or something? If so, you could keep a table of all the allocated pointers and use that. Otherwise, I'm not sure. Is there a way to get at malloc's bookkeeping information?
Not as a standard feature.
A debug version of your malloc library might have some function to do this.
You can compare its address to something you know to be static, and say it's malloced only if it's far away, if you know the scope it should be coming from, but if its scope is unknown, you can't really trust that.
1.) Obtain a map file for the code u have.
2.) The underlying process/hardware target platform should have a memory map file which typically indicates - starting address of memory(stack, heap, global0, size of that block, read-write attributes of that memory block.
3.) After getting the address of the object(pointer variable) from the mao file in 1.) try to see which block that address falls into. u might get some idea.
=AD