Here is what I am working with:
char* qdat[][NUMTBLCOLS];
char** tdat[];
char* ptr_web_data;
// Loop thru each table row of the query result set
for(row_index = 0; row_index < number_rows; row_index++)
{
// Loop thru each column of the query result set and extract the data
for(col_index = 0; col_index < number_cols; col_index++)
{
ptr_web_data = (char*) malloc((strlen(Data) + 1) * sizeof(char));
memcpy (ptr_web_data, column_text, strlen(column_text) + 1);
qdat[row_index][web_data_index] = ptr_web_data;
}
}
tdat[row_index] = qdat[col_index];
After the data is used, the memory allocated is released one at a time using free().
for(row_index = 0; row_index < number_rows; row_index++)
{
// Loop thru all columns used
for(col_index = 0; col_index < SARWEBTBLCOLS; col_index++)
{
// Free memory block pointed to by results set array
free(tdat[row_index][col_index]);
}
}
Is there a way to release all the allocated memory at once, for this array?
Thank You.
Not with the standard malloc() allocator - you need to investigate the use of memory pools. These work by allocating a big block of memory up-front, allocating from it and freeing back to it as you request via their own allocation functions, and then freeing the whole lot with a special "deallocate all" function.
I must say I've always found these things a bit ugly - it really isn't that hard to write code that doesn't leak. The only reason I can see for using them is to mitigate heap fragmentation, if that is a real problem for you.
No there is not. Memory which is separately allocated must be separately freed.
The only way you could free it as once is if you allocated it at once is a giant block. You would then have to do a bit of pointer math to assign every row the correct index into the array but it's not terribly difficult. This approach does have a few downsides though
Extra pointer math
Requires one giant contiguous block of memory vs. N smaller blocks of memory. Can be an issue in low memory or high fragmentation environments.
Extra work for no real stated gain.
If you want to release it all at once, you have to allocate it all at once.
A simple manual solution, if you know the total size you'll need in advance, is to allocate it all in once chunk and index into it as appropriate. If you don't know the size in advance you can use realloc to grow the memory, so long as you only access it indexed from the initial pointer, and don't store additional pointers anywhere.
That being said, direct allocation and deallocation is a simple solution, and harder to get wrong than the alternatives. Unless the loop to deallocate is causing you real difficulties, I would stick with what you have.
Related
I'm just starting to learn coding in c, and I have a few questions regarding 2d matrices in combination with the free() command.
I know that you first need to create an array with pointer, pointing to the different columns of the matrix:
double **array = (double **)malloc(5*sizeof(double *));
for(int n = 0; n<5; n++){
array[n] = (double *) malloc(6*sizeof(double));
I know that the correct way to then deallocate this matrix is to first deallocate the individual rows and then the array pointer itself. Something along the lines of:
for (i = 0; i < nX; i++){
free(array[i]); }
free(array);
My question is: Why is this necessary? I know that this incorrect, but why can't you just use: free(array)? This would deallocate the pointer array, to my understanding. Won't the memory that is used by the columns just be overwritten when something else needs acces to it? Would free(array) lead to corrupted memory in any way?
Any help is much appreciated!
Your code, not only allocate memory for array of pointers (the blue array), but in the for loop, you also allocate memory for the red arrays as well. So, free(array) line, alone, will just free the memory allocated by the blue array, but not the red ones. You need to free the red ones, just before loosing the contact with them; that is, before freeing the blue array.
And btw;
Won't the memory that is used by the columns just be overwritten when something else needs acces to it?
No. The operating system will keep track of the memory allocated by your process (program) and will not allow any other process to access the allocated memory until your process terminates. Under normal circumstances —I mean, remembering the C language not having a garbage collector— the OS never knows that you've lost connection with the allocated memory space and will never attempt like, "well, this memory space is not useful for this process anymore; so, let's de-allocate it and serve it for another process."
It would not lead to corruption, no, but would create a memory leak.
If done once in your program, it probably doesn't matter much (a lot of professional/expensive applications have - small,unintentional - memory leaks), but repeat this in a loop, and you may run out of memory after a while. Same thing if your code is called from an external program (if your code is in a library).
Aside: Not freeing buffers can be a useful way (temporarily) to check if the crashes you're getting in your programs originate from corrupt memory allocation or deallocation (when you cannot use Valgrind). But in the end you want to free everything, once.
If you want to perform only one malloc, you could also allocate one big chunk, then compute the addresses of the rows. In the end, just deallocate the big chunk (example here: How do we allocate a 2-D array using One malloc statement)
This is needed because C does not have a garbage collector.
Once you allocate memory with malloc or similar function, it is marked as "in use" for as long as your program is running.
It does not matter if you no longer hold a pointer to this memory in your program.
There is no mechanism in the C language to check this and automatically free the memory.
Also, when you allocate memory with malloc the function does not know what you are using the memory for. For the allocator it is all just bytes.
So when you free a pointer (or array of pointers), there is no logic to "realize" these are pointers that contain memory addresses.
This is simply how the C language is designed: the dynamic memory management is almost1 completely manual - left to the programmer, so you must call free for every call to malloc.
1 C language does handle some of the more tedious tasks needed to dynamically allocate memory in a program such as finding where to get a free continuous chunk of memory of the size you asked for.
Let's take this simple example:
int **ptr = malloc(2*sizeof *ptr);
int *foo = malloc(sizeof *foo);
int *bar = malloc(sizeof *bar);
ptr[0] = foo;
ptr[1] = bar;
free(ptr);
If your suggestion were implemented, foo and bar would now be dangling pointers. How would you solve the scenario if you just want to free ptr?
Let's suppose I have this code:
for (int i=0;i<n;i++) {
//compute z
float *p = (float *) malloc (sizeof(float)*z);
//do something with p
}
Notice that p isn't used anywhere else and each for cycle is independent from the other.
Let's suppose that z is not so big, so the single p is not that expensive in terms of memmory. However, n is potentially big, so the total memory taken by p can be consistent.
Is it correct to free() it with:
for (int i=0;i<n;i++) {
//compute z
float *p = (float *) malloc (sizeof(float)*z);
//do something with p
free(p);
}
Bonus question: if time performance would be the priority (and not memory consumption), would be better to avoid the free(p), since it's time consuming?
Since you tagged this with C++, you should never use malloc and free. Use smart pointers (or new/delete if you don't have access to a C++11-compliant compiler).
for (int i=0;i<n;i++) {
// compute z
std::unique_ptr<float[]> p{new float[z]};
// do something with p
// p gets automatically freed at the end of the scope
}
To answer your questions:
Is it correct to free() it with...
Yes. If you allocate something with malloc, you always need to free it.
would be better to avoid the free(p), since it's time consuming?
Yes. Consider preallocating the memory location outside of the loop.
// preallocate
std::unique_ptr<float[]> p{new float[z]};
for (int i=0;i<n;i++) {
// clear p
// compute z
// do something with p
}
// p gets automatically freed at the end of the scope
You can pre-allocate necessary memory amount before the cycle and re-use it.
In case if you don't know how big z would be - I would recommend to write size of allocated memory somewhere, and if z is bigger than it - then re-allocate, else - just re-use already allocated memory.
Observe: malloc is used. Act: invoke free. It's as simple as that. Portable, well-defined code requires malloc and free.
The amount of memory allocated plays no role here. If it's too much memory, malloc will throw an error, but that has nothing to do with the fact that you always need a free to follow a malloc.
Yes. You must free it. Otherwise you have a memory leak, which is bad. Especially if you loop a large number of times.
A good rule of thumb is that every malloc must be matched by a free. Always. (this is especially important on larger projects)
Consider using a buffer, that is reused in order to avoid unnecessary allocations. This can be done very easily by using std::vector<float>:
std::vector<float> p;
for (int i=0;i<n;i++) {
//compute z
p.resize( z );
//do something with p
}
In the worst case you get O(log n) memory allocations. With your code you will get n memory allocations. Not calling free() just causes a memory leak. std::vector<float> will clean up the memory automatically eventually.
It seems the size is constant, so why you want to allocate it again and again?
Just allocate once before the loop, inside the loop initialise at the beginning.
You can reuse the memory. And after the loop ends free the memory.
And the program ends after the loop then you don't have to free it, all the heap memory that the program consumed would be returned to the OS, but it is always a good practice to free the memory you allocate.
On the bonus question, free is not that time consuming but allocating memory is, so don't worry on the time consumed by free.
If you reuse that memory again - it is obviously more efficient to free it only when you end your program and only allocated it once (and reallocate using realloc() when you need bigger/smaller array).
Every OS you will probably program for will handle releasing dynamically allocated memory when your program terminates. But, as we mostly try to keep our code as cross-platform as possible, you should always free() dynamically allocated memory.
As for if you have a huge amount of memory and care only for speed - then obviously free()ing would 'slow down' the program, but the amount of time it takes is so ridiculously small, 76*(10**-8) on my humble machine, for a 1024 bytes chunk, so it is fairly insignificant.
I want to test how much the OS does allocate when I request 24M memory.
for (i = 0; i < 1024*1024; i++)
ptr = (char *)malloc(24);
When I write like this I get RES is 32M from the top command.
ptr = (char *)malloc(24*1024*1024);
But when I do a little change the RES is 244. What is the difference between them? Why is the result 244?
The allocator has its own data structures about the bookkeeping that require memory as well. When you allocate in small chunks (the first case), the allocator has to keep a lot of additional data about where each chunk is allocated and how long it is. Moreover, you may get gaps of unused memory in between the chunks because malloc has a requirement to return a sufficiently aligned block, most usually on an 8-byte boundary.
In the second case, the allocator gives you just one contiguous block and does bookkeeping only for that block.
Always be careful with a large number of small allocations, as the bookkeeping memory overhead may even outweigh the amount of the data itself.
The second allocation barely touches the memory. The allocator tells you "okay, you can have it" but if you don't actually touch the memory, the OS never actually gives it to you, hoping you'll never use it. Bit like a Ponzi scheme. On the other hand, the other method writes something (a few bytes at most) to many pages, so the OS is forced to actually give you the memory.
Try this to verify, you should get about 24m usage:
memset(ptr, 1, 1024 * 1024 * 24);
In short, top doesn't tell you how much you allocated, i.e. what you asked from malloc. It tells you what the OS allocated to your process.
In addition to what has been said:
It could be that some compilers notice how you allocate multiple 24 Byte Blocks in a loop, assigning their addresses to the same pointer and keeping only the last block you allocated, effectively rendering every other malloc from before useless. So it may optimize your whole loop into something like this:
ptr = (char *)malloc(24);
i = 1024*1024;
I am writing a program to do some analysis on DNA sequences.
Everything works fine except for this thing.
I want to declare a 2D array of size m*n where m and n are read from an input file.
Now the issue is that if m and n goes too large. As an example if m = 200 and n = 50000
then I get a seg fault at the line where I declare my array.
array[m][n];
Any ideas how to overcome this. I do need such an array as my entire logic depends on how to process this array.
Probably you are running out of stack space.
Can you not allocate the array dynamically on heap using malloc?
You may want to have a look at this answer if you do not know how to do that.
As others have said it is not a good idea to allocate a large VLA (variable length array) on the stack. Allocate it with malloc:
double (*array)[n] = malloc(sizeof(double[m][n]));
and you have an object as before, that is that the compiler perfectly knows how to address individual elements array[i][j] and the allocation still gives you one consecutive blob in memory.
Just don't forget to do
free(array);
at the end of your scope.
Not sure what type you're using but for the following code I've assumed int.
Rather than doing this:
int array[200][50000];
Try doing this:
int** array = (int**)malloc(200);
for (int i = 0; i < 200; i++)
{
array[i] = (int*)malloc(50000);
}
This will allocate "heap" memory rather than "stack" memory. You are asking for over 300mb (if you're using a 32bit type) so you probably don't have that much "stack" memory.
Make sure to cleanup after you're done with the array with:
for (int i = 0; i < 200; i++)
{
free(array[i]);
}
free(array);
Feel free to use m and n instead of the constants I used above!
Edit: I originally wrote this in C++, and converted to C. I am a little more rusty with C memory allocation/deallocation, but I believe I got it right.
You are likely running out of stack space.
Windows for instance gives each thread 1MB stack. Assuming the array contains integers and you are creating it on the stack you are creating a 40MB stack variable.
You should instead dynamically allocate it on the heap.
The array (if local) is allocated in the stack. There is certain limits imposed on the stack size for a process/thread. If the stack is overgrown it will cause issues.
But you can allocate the array in heap using malloc . Typical heap size could be 4GB (this can be more or less depending on OS/Architecture). Check the return value of malloc to make sure that memory for the array is correctly allocated.
I have a function in which I make use of a local array. I would like to return a pointer to this, but of course one cannot do this unless one manually allocates memory. Therefore I would like to know if allocation like so:
for(i = 0; i < 26; i++)
{
llist[i] = malloc(sizeof(SomeStruct));
}
can be later freed like so:
// (where ptr is a pointer to the first allocation in llist)
for(i = 0; i < 26, i++)
{
free(ptr);
ptr += sizeof(SomeStruct);
}
Edit:
Seems I can't do this. Is there any way in which I could free all the memory given only a pointer to the first element? Or should I take a different approach entirely?
No, it cannot. There is no guarantee that these blocks are allocated consecutively. In fact, they are almost certainly not with all the modern implementations I've seen. Most store the size of the block and often other information immediately before each block. Also, if your memory is fragmented, the blocks might wind up anywhere in the free space between other data, where there is not sufficient room to place them consecutively.
No, you can't. In the first empty space malloc finds it might only be place for one struct. If that's the case, the next block will be placed somewhere else.
Depends on what you do inside the loop. Some functions will allocate buffers internally, which means that malloc() could be called behind your back. In your simple loop, the values SHOULD be consecutive. As well, once you throw in multi-threading, all bets are out the window, as something in another thread could malloc() memory as well.
The actual allocation spot also depends on how fragmented your memory is. There may not be enough consecutive blocks to handle your request, so the actual allocation could be from anywhere in the free space.