I have an array named 'ArrayA' and it is full of ints but I want to add another 5 cell to the end of the array every time a condition is met. How would I do this? ( The internet is not being very helpful )
If this is a static array, you will have to create a new one with more space and copy the data yourself. If it was allocated with malloc(), as the title to your question suggests, then you can use realloc() to do this more-or-less automatically. Note that the address of your array will, in general, have changed.
It is precisely because of the need for "dynamic" arrays that grow (and shrink) as needed, that languages like C++ introduced vectors. They do the management under the covers.
You need the realloc function.
Also note that adding 5 cells is not the best performance solution.
It is best to double the size of your arrays every time an array increase is needed.
Use two variables, one for the size (the number of integers used) and one for capacity (the actual memory size of arrays)
In a modern OS it is generally safe to assume that if you allocate a lot of memory that you don't use then it will not actually consume physical RAM, but only exist as virtual mappings. The OS will provide physical RAM as soon as a page (today generally in chunks of 4Kb) is used for the first time.
You can specifically enforce this behavior by using mmap to create a large anonymous mapping (MAP_PRIVATE | MAP_ANONYMOUS) e.g. as much as you intend to hold at maximum. On modern x64 systems virtual mappings can be up to 64Tb large. It is logically memory available to your program, but in practice pages will be added to it as you start using them.
realloc as described by the other posters is the naiive way to resize a malloc mapping, but make sure that realloc was successful. It can fail!
Problems with memory arise when you use memory once, don't deallocate it and stop using it. In contrast allocated, but untouched memory generally does not actually use resources other then VM table entries.
Related
i do not seem to find an answer to this question. Why you cant free up an individual adress is it because the space needs to be continuous? and if this is the answer then why fragmentation occurs on Hard-Disks
Can you free individual memory adresses of an array allocated dynamically?
If the memory is at the end of an array, you can free off the unneeded excess by performing a realloc to a smaller size, with the caveat that you may actually get a new pointer to new memory with the prefix contents copied into it, and the original memory freed in its entirety.
Otherwise, no. The free interface is defined to only accept addresses returned from malloc, calloc or realloc.
Why you cant free up an individual adresss is it because the space needs to be continuous?
Well, the direct answer is that there is no interface defined to do so. There is no way to tell free how much of the pointer you passed in should be freed. If you want to free all memory to the end of the allocated block, realloc does that.
If contiguity is not important to your program, just use separate allocations for each array element, and free them individually.
and if this is the answer then why fragmentation occurs on Hard-Disks
One way to imagine a scenario of fragmentation on a file system is that if three files are created one after another, and then the middle one is deleted, there is now a hole between two files.
|---File 1---|--- Hole ---|---File 3---|
Now suppose a new file is created, so it starts out inside the hole between the two files, but as it grows, it cannot fit in the hole, so now the rest of the file is after File 3. In this case, we would say the new file is fragmented.
|---File 1---|---File 4...|---File 3---|...File 4---|
This happens on "Hard-Drives" because a filesystem is designed that way: allow a large file to span the available holes in the physical medium.
A RAM disk used for a filesystem would also eventually have fragmented files.
A non-contiguous data structure could be considered to be "fragmented", e.g., a linked-list or a tree, but that is by design. An array is considered contiguous by its very definition. However, files on a filesystem are not arrays.
Broadly, the reason you cannot release individual portions of allocated memory is that it is not useful enough to justify writing the software to support it.
The C standard specifies services provided by malloc, free, realloc, and related routines. The only provisions it makes for releasing space are by using free to release an allocation and by using realloc to replace an allocation with a smaller one.
C implementations are free to extend the C standard by providing services to release portions of allocated space. However, I am not aware of any that have done so. If a program were allowed to free arbitrary pieces of memory, the memory management software would have to keep track of all of them. That requires extra data and time. Additionally, it can interfere with some schemes for managing memory. Memory management software might organize memory so that allocations of particular sizes can be satisfied quickly out of specialized pools, and having to take back an arbitrary sized portion that was part of a specialized pool could be a problem.
If there were a demand for such a service, it could be written. But programs and algorithms have evolved over the years to use the existing services, and there does not seem to be much need to release individual portions of allocations. Generally, if a program is going to work with many objects that it might free individually, it allocates them individually. This is common when building data structures of all sorts, using pointers to construct trees, hashed arrays of lists or other structures, and so on. Data structures are often built out of individual nodes that can be allocated or freed individually. So there is little need to carve individual pieces to be released out of larger allocations.
The organization of memory has very little to do with the organization of data on hard disk or other storage media. Data is generally transferred between arbitrary places on disk and arbitrary places in memory as needed. In a variety of circumstances, files are “memory mapped,” meaning that the contents of a file are made visible in memory so that one can read the file contents by reading memory and one can modify the file by modifying memory. However, even in this situation, there is not generally any relationship between where the blocks of the file are on disk and where the blocks of the file are in memory. The blocks of a file are managed by the file system and are not necessarily contiguous, and the blocks in memory are managed by the memory system and may be rearranged arbitrarily with support from virtual memory hardware.
First question NO as you can only free the whole memory allocated by one malloc family function
Fragmentation of hard disks does not have anything common with the memory allocations.
Memory allocation is handled as seemingly continuous blocks of memory (it might not be in physical memory though, but that's not relevant).
There is no simple way to "cut a hole" in a single memory allocation, but you could do something like this:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define ARRAY_LEN 11
int main (void)
{
char *array;
array = (char *) malloc(ARRAY_LEN);
strcpy(array,"0123456789");
printf("%s\n",array);
//Remove the 5th element:
memmove(&array[5], &array[5+1], ARRAY_LEN-5);
array = realloc(array, ARRAY_LEN-1);
printf("%s\n",array);
free(array);
return 0;
}
Some Linux filesystems allows for "punching holes" in files, so with a mmap'ed file, you might be able to use the fallocate systemcall on it while using it as an array in memory.
Can you free individual memory adresses of an array allocated dynamically?
You seem to recognize that the answer is "no", because you follow up with
Why you cant free up an individual adress is it because the space needs to be continuous?
Each individual allocation is continuous, but the union of all dynamically-allocated space is by no means certain to be continuous. More on this later.
At the most practical level, you cannot free chunks of a larger allocation because C provides no mechanism for doing so. In particular, among the specifications for the free() function is:
if the argument does not match a pointer earlier returned by a memory management function, or if the space has been deallocated by a call to free or realloc, the behavior is undefined.
Thus, free() exhibits UB if its argument is a pointer into the interior of an allocated block.
Note also that free() accepts only one parameter. It makes no provision for the caller to specify the amount of memory to free, so the memory-management subsystem has to figure that out from the argument argument presented. That's fine for the operating model that one frees only whole, previously-allocated blocks, but it does not easily support freeing an allocation in multiple pieces.
Furthermore, consider that although you cannot free specific chunks of a larger allocation, you can use realloc() to reduce the the size of an allocation (which may also involve moving it).
Anything beyond that is in the realm of implementation-specific behavior, but do bear in mind that
it is very common for allocation to be performed and accounted for in terms of multi-byte blocks -- for example, multiples of 16 bytes -- regardless of the specific sizes requested. An implementation that works this way cannot under any circumstances free partial blocks, though one could imagine being able to free individual blocks from a larger allocation.
some implementations store memory management metadata adjacent to the dynamically-allocated space presented to the program. In such an implementation, it is not useful to free pieces of a larger allocation because they cannot, in general, be reused until the whole allocation is freed, for there is no available place for the needed metadata.
and if this is the answer then why fragmentation occurs on Hard-Disks
You don't need to free allocations in pieces to get memory fragmentation. It can suffice to perform multiple allocations and afterward free only some of them. This is a real issue that can degrade performance and even cause programs to fail.
With that said, however, file systems typically use different methods and data structures for tracking their metadata than do C memory-management implementations, and the underlying hardware has different characteristics and behavior, so there's really no justification for forming expectations about the behavior and capabilities of one variety of storage based on the behavior and capabilities of the other.
I'm looking for a way to allocate additional memory (in C) at runtime, for an existing structure (that already had its memory assigned initially). I have a feeling I might be able to use memmove or something similar but that's still just a copy operation, and doesn't increase the amount of memory available to a structure at runtime. Also I don't want to have to copy the entire structure every time I need to do this, which will be many hundreds of times during the program (the structure is already huge). Can anyone help?
UPDATE: Thanks everyone for the replies. To give more detail, what I am trying to do is run an MPI-parallelised code that creates many instances of the structure (call it 'S') initially. Each instance of the structure contains an array 'T' which records the time of a particular event happening as the code is run. These events occur at runtime, and the number of events differs for each instance of S. For example, S[0] might see 100 events (and therefore need an array of 100 elements in length) but S[1] might see only 1 event (and S[2] 30 events, etc.) Therefore it would be very wasteful to allocate huge amounts of memory at the start for every instance of S (for which there are millions) since some might fill the array but others would not even come close. Indeed I have tried this and it is too much for the machine I am running it on.
I will try some of the ideas here and post my progress. Many thanks!
You could probably use realloc().
There is no way to do what you describe, because there is no way to guarantee that there will be available memory next to the one that your structure is currently occupying.
The standard thing to do is to allocate more memory and copy your data.
Of course if you can know (an estimate of) the size of the memory allocation that you need you can preallocate it and avoid copying.
Note, however, that the structures in C have a fixed size once they are declared, so it seems you don't really need to allocate more memory for an existing structure...
realloc is the only way to expand the existing dynamic memory. realloc will tries to expand the existing buffer, if it fails in expansion it will allocate new buffer for the total size required and it will copy the data from old buffer. If you dont want to do realloc every time(which internally will memmove most of the time) then you can try to reallocate more memory than actually you required.
realloc(buf_ptr, (actual_size + additional_size) * 2);
This way will reduce the frequency of calling realloc (and memmove).
Note : Implementation of realloc is different in some architecture, it will never tries to expand the memory it always tries to allocate buffer for total size. So in those platforms memmove will be called for every call to realloc.
It sounds like you are looking for the C feature called flexible array member (example). It is only well-defined for C standard C99 or later.
The last member of the struct will have to be declared as a flexible array member, which you initially malloc, and later realloc (and of course memcpy to do the actual copying).
First, here is where I got the idea from:
There was once an app I wrote that used lots of little blobs of
memory, each allocated with malloc(). It worked correctly but was
slow. I replaced the many calls to malloc with just one, and then
sliced up that large block within my app. It was much much faster.
I was profiling my application, and I got a unexpectedly nice performance boost when I reduced the number of malloc calls. I am still allocating the same amount of memory, though.
So, I would like to do what this guy did, but I am unsure what's the best way to do it.
My Idea:
// static global variables
static void * memoryForStruct1 = malloc(sizeof(Struct1) * 10000);
int struct1Index = 0;
...
// somewhere, I need memory, fast:
Struct1* data = memoryForStruct1[struct1Index++];
...
// done with data:
--struct1Index;
Gotchas:
I have to make sure I don't exceed 10000
I have to release the memory in the same order I occupied. (Not a major issue in my case, since I am using recursion, but I would like to avoid it if possible).
Inspired from Mihai Maruseac:
First, I create a linked list of int that basically tells me which memory indexes are free. I then added a property to my struct called int memoryIndex which helps me return the memory occupied in any order. And Luckily, I am sure my memory needs would never exceed 5 MB at any given time, so I can safely allocate that much memory. Solved.
The system call which gives you memory is brk. The usual malloc and calloc, realloc functions simply use the space given by brk. When that space is not enough, another brk is made to create new space. Usually, the space is increased in sizes of a virtual memory page.
Thus, if you really want to have a premade pool of objects, then make sure to allocate memory in multiples of pagesize. For example, you can create one pool of 4KB. 8KB, ... space.
Next idea, look at your objects. Some of them have one size, some have other size. It will be a big pain to handle allocations for all of them from the same pool. Create pools for objects of various sizes (powers of 2 is best) and allocate from them. For example, if you'll have an object of size 34B you'd allocate space for it from the 64B pool.
Lastly, the remaining space can be either left unused or it can be moved down to the other pools. In the above example, you have 30B left. You'd split it in 16B, 8B, 4B and 2B chunks and add each chunk to their respective pool.
Thus, you'd use linked lists to manage the preallocated space. Which means that your application will use more memory than it actually needs but if this really helps you, why not?
Basically, what I've described is a mix between buddy allocator and slab allocator from the Linux kernel.
Edit: After reading your comments, it will be pretty easy to allocate a big area with malloc(BIG_SPACE) and use this as a pool for your memory.
If you can, look at using glib which has memory slicing API that supports this. It's very easy to use, and saves you from having to re-implement it.
I have to use c/asm to create a memory management system since malloc/free don't yet exist. I need to have malloc/free!
I was thinking of using the memory stack as the space for the memory, but this would fail because when the stack pointer shrinks, ugly things happen with the allocated space.
1) Where would memory be allocated? If I place it randomly in the middle of the Heap/Stack and the Heap/Stack expands, there will be conflicts with allocated space!
12 What Is the simplest/cleanest solution for memory management? These are the only options I've researched:
A memory stack where malloc grows the stack and free(p) shrinks the stack by shifting [p..stack_pointer] (this would invalidate the shifted memory addresses though...).
A linked list (Memory Pool) with a variable-size chunk of memory. However I don't know where to place this in memory... should the linked list be a "global" variable, or "static"?
Thanks!
This article provides a good review of memory management techniques. The resources section at the bottom has links to several open source malloc implementations.
For embedded systems the memory is partitioned at link time into several sections or pools, i.e.:
ro (code + constants)
rw (heap)
zi (zero initialised memory for static variables)
You could add a 4th section in the linker configuration files that would effectively allocate a space in the memory map for dynamic allocations.
However once you have created the raw storage for dynamic memory then you need to understand how many, how large and how frequent the dynamic allocations will occur. From this you can build a picture of how the memory will fragment over time.
Typically an application that is running OS free will not use dynamic memory as you don't want to have to deal with the consequences of malloc failing. If at all possible the better solution is design to avoid it. If this is not at all possible try and simplify the dynamic behaviour using a few large structures that have the data pre-allocated before anything needs to use it.
For example say that you have an application that processes 10bytes of data whilst receiving the next 10 bytes of data to process, you could implement a simple buffering solution. The driver will always be requesting buffers of the same size and there would be a need for 3 buffers. Adding a little meta data to a structure:
{
int inUse;
char data[10];
}
You could take an array of three of theses structures (remembering to initialise inUse to 0 and flick between [0] and [1], with [2] reserved for the situations when a few too many interrupts occur and the next buffer is required buffer one is freed (the need for the 3rd buffer). The alloc algorithm would on need to check for the first buffer !inUse and return a pointer to data. The free would merely need to change inUse back to 0.
Depending on the amount of available RAM and machine (physical / virtual addressing) that you're using there are lots of possible algorithms, but the more complex the algorithm the longer the allocations could take.
Declare a huge static char buffer and use this memory to write your own malloc & free functions.
Algorithms for writing malloc and free could be as complex (and optimized) or as simple as you want.
One simple way could be following...
based on the type of memory allocation needs in your application try to find the most common buffer sizes
declare structures for each size with a char buffer of that length
and a boolean to represent whether buffer is occupied or not.
Then declare static arrays of above structures( decide array sizes
based on the total memory available in the system)
now malloc would simply go the most suitable array based on the
required size and search for a free buffer (use some search algo here
or simply use linear search) and return. Also mark the boolean in the
associated structure to TRUE.
free would simply search for buffer and mark the boolean to FALSE.
hope this helps.
Use the GNU C library. You can use just malloc() and free(), or any other subset of the library. Borrowing the design and/or implementation and not reinventing the wheel is a good way to be productive.
Unless, of course, this is homework where the point of the exercise is to implement malloc and free....
Let's say I have a program(C++, for example) that allocates multiple objects, never bigger than a given size(let's call it MAX_OBJECT_SIZE).
I also have a region(I'll call it a "page") on the heap(allocated with, say, malloc(REGION_SIZE), where REGION_SIZE >= MAX_OBJECT_SIZE).
I keep reserving space in that page until the filled space equals PAGE_SIZE(or at least gets > PAGE_SIZE - MAX_OBJECT_SIZE).
Now, I want to allocate more memory. Obviously my previous "page" won't be enough. So I have at least two options:
Use realloc(page, NEW_SIZE), where NEW_SIZE > PAGE_SIZE;
Allocate a new "page"(page2) and put the new object there.
If I wanted to have a custom allocate function, then:
Using the first method, I'd see how much I had filled, and then put my new object there(and add the size of the object to my filled memory variable).
Using the second method, I'd have a list(vector? array?) of pages, then look for the current page, and then use a method similar to 1 on the selected page.
Eventually, I'd need a method to free memory too, but I can figure out that part.
So my question is: What is the most efficient way to solve a problem like this? Is it option 1, option 2 or some other option I haven't considered here? Is a small benchmark needed/enough to draw conclusions for real-world situations?
I understand that different operations may perform differently, but I'm looking for an overall metric.
In my experience option 2 is much easier to work with has minimal overhead. Realloc does not guarantee it will increase the size of existing memory. And in practice it almost never does. If you use it you will need to go back and remap all of the old objects. That would require that you remember where every object allocated was... That can be a ton over overhead.
But it's hard to qualify "most efficient" without knowing exactly what metrics you use.
This is the memory manager I always use. It works for the entire application not just one object.
allocs:
for every allocation determine the size of the object allocated.
1 look at a link list of frees for objects of that size to see if anything has been freed if so take the first free
2 look for in a look up table and if not found
2.1 allocate an array of N objects of the size being allocated.
3 return the next free object of the desired size.
3.1 if the array is full add a new page.
N objects can be programmer tunned. If you know you have a million 16 byte objects you might want that N to be slightly higher.
for objects over some size X, do not keep an array simply allocate a new object.
frees:
determine the size of the object, add it to the link list of frees.
if the size of the object allocated is less than the size of a pointer the link list does not need to incur any memory overhead. simply use the already allocated memory to store the nodes.
The problem with this method is memory is never returned to the operating system until the application has exited or the programmer decides to defragment the memory. defragmenting is another post. it can be done.
It is not clear from your question why you need to allocate a big block of memory in advance rather than allocating memory for each object as needed. I'm assuming you are using it as a contiguous array. Otherwise, it would make more sense to malloc the memory of each object as it is needed.
If it is indeed acting as an array,malloc-ing another block gives you another chunk of memory that you have to access via another pointer (in your case page2). Thus it is no longer on contiguous block and you cannot use the two blocks as part of one array.
realloc, on the other hand, allocates one contiguous block of memory. You can use it as a single array and do all sorts of pointer arithmetic not possible if there are separate blocks. realloc is also useful when you actually want to shrink the block you are working with, but that is probably not what you are seeking to do here.
So, if you are using this as an array, realloc is basically the better option. Otherwise, there is nothing wrong with malloc. Actually, you might want to use malloc for each object you create rather than having to keep track of and micro-manage blocks of memory.
You have not given any details on what platform you are experimenting. There are some performance differences for realloc between Linux and Windows, for example.
Depending on the situation, realloc might have to allocate a new memory block if it can't grow the current one and copy the old memory to the new one, which is expensive.
If you don't really need a contiguous block of memory you should avoid using realloc.
My sugestion would be to use the second approach, or use a custom allocator (you could implement a simple buddy allocator [2]).
You could also use more advanced memory allocators, like
APR memory pools
Google's TCMalloc
In the worst case, option 1 could cause a "move" of the original memory, that is an extrawork to be done. If the memory is not moved, anyway the "extra" size is initialized, which is other work too. So realloc would be "defeated" by the malloc method, but to say how much, you should do tests (and I think there's a bias on how the system is when the memory requests are done).
Depending on how many times you expect the realloc/malloc have to be performed, it could be an useful idea or an unuseful one. I would use malloc anyway.
The free strategy depends on the implementation. To free all the pages as whole, it is enough to "traverse" them; instead of an array, I would use linked "pages": add sizeof(void *) to the "page" size, and you can use the extra bytes to store the pointer to the next page.
If you have to free a single object, located anywhere in one of the pages, it becomes a little bit more complex. My idea is to keep a list of non-sequential free "block"/"slot" (suitable to hold any object). When a new "block" is requested, first you pop a value from this list; if it is empty, then you get the next "slot" in the last in use page, and eventually a new page is triggered. Freeing an object, means just to put the empty slot address in a stack/list (whatever you prefer to use).
In linux (and probably other POSIX systems) there is a third possibility, that is to use a memory mapped region with shm_open. Such a region is initialized by zeroes once you access it, but AFAIK pages that you never access come with no cost, if it isn't just the address-range in virtual memory that you reserve. So you could just reserve a large chunk of memory at the beginning (more than you ever would need) of your execution and then fill it incrementally from the start.
What is the most efficient way to solve a problem like this? Is it option 1, option 2 or some other option I haven't considered here? Is a small benchmark needed/enough to draw conclusions for real-world situations?
Option 1. For it to be efficient, NEW_SIZE has to depend on old size non-linearly. Otherwise you risk running into O(n^2) performance of realloc() due to the redundant copying. I generally do new_size = old_size + old_size/4 (increase by 25% percent) as theoretically best new_size = old_size*2 might in worst case reserve too much unused memory.
Option 2. It should be more optimal as most modern OSs (thanks to C++'s STL) are already well optimized for flood of small memory allocations. And small allocations have lesser chance to cause memory fragmentation.
In the end it all depends how often you allocate the new objects and how do you handle freeing. If you allocate a lot with #1 you would have some redundant copying when expanding but freeing is dead simple since all objects are in the same page. If you would need to free/reuse the objects, with #2 you would be spending some time walking through the list of pages.
From my experience #2 is better, as moving around large memory blocks might increase rate of heap fragmentation. The #2 is also allows to use pointers as objects do not change their location in memory (though for some applications I prefer to use pool_id/index pairs instead of raw pointers). If walking through pages becomes a problem later, it can be too optimized.
In the end you should also consider option #3: libc. I think that libc's malloc() is efficient enough for many many tasks. Please test it before investing more of your time. Unless you are stuck on some backward *NIX, there should be no problem using malloc() for every smallish object. I used custom memory management only when I needed to put objects in exotic places (e.g. shm or mmap). Keep in mind the multi-threading too: malloc()/realloc()/free() generally are already optimized and MT-ready; you would have to reimplement the optimizations anew to avoid threads being constantly colliding on memory management. And if you want to have memory pools or zones, there are already bunch of libraries for that too.