I'm having memory leaks in a larger program and I believe this is the cause of it.
#include <stdlib.h>
#include <Windows.h>
typedef struct _struct{
char* name;
} str;
int main() {
system("PAUSE");
str* Character = (str*)malloc(sizeof(str) * 20000);
for(int i = 0; i < 20000; i++){
Character[i].name = (char*)malloc(20000); // Assign memory.
}
for(int i = 0; i < 20000; i++){
free(Character[i].name); // Free memory.
}
free(Character);
system("PAUSE");
}
Memory at first pause: ~500K.
Memory at second pause: ~1.7M.
Using VS2012 for testing. Any ideas?
How are you measuring the amount of memory occupied by the program? One thing off the top of my head is that you're looking at the size of the working set the OS is keeping track of. Since you've allocated and freed a lot of memory, the size of that set has increased. Some OSs will adjust the size of the working set after a while, some won't. What OS are we looking at here?
When you call malloc, memory is allocated on the heap. If there is insufficient space left on the heap, the program will ask the OS for more memory and another chunk is acquired. Memory acquired from the OS is usually not returned until the program finishes (although this is up to the OS).
Program size alone can not normally be used to check for memory leaks! Use Valgrind or a similar tool to check for memory that never gets freed.
str* Character = (str*)malloc(sizeof(str) * 20000);
In the above line you are allocating the memory by finding the size of the struct. Here the size of the structure you will get will be the size of the pointer width and not the size of the char.
suppose for example if the pointer width is 32 bit the it will allocate (4 * 20000) = 80000 bytes.
If you want to allocate for 20000 struct's,
str* Character = (str*)malloc(sizeof(char) * 20000);
Related
I have a char array, we know that that a char size is 1 byte. Now I have to collect some char -> getchar() of course and simultaneously increase the array by 1 byte (without malloc, only library: stdio.h)
My suggestion would be, pointing to the array and somehow increase that array by 1 till there are no more chars to get OR you run out of Memory...
Is it possible to increase char array while using it, WITHOUT malloc?
No.
You cannot increase the size of a fixed size array.
For that you need realloc() from <stdlib.h>, which it seems you are not "allowed" to use.
Is it possible to increase char array while using it, WITHOUT malloc?
Quick answer: No it is not possible to increase the size of an array without reallocating it.
Fun answer: Don't use malloc(), use realloc().
Long answer:
If the char array has static or automatic storage class, it is most likely impossible to increase its size at runtime because keeping it at the same address that would require objects that are present at higher addresses to be moved or reallocated elsewhere.
If the array was obtained by malloc, it might be possible to extend its size if no other objects have been allocated after it in memory. Indeed realloc() to a larger size might return the same address. The problem is it is impossible to predict and if realloc returns a different address, the current space has been freed so pointers to it are now invalid.
The efficient way to proceed with this reallocation is to increase the size geometrically, by a factor at a time, 2x, 1.5x, 1.625x ... to minimize the number of reallocations and keep linear time as the size of the array grows linearly. You would a different variable for the allocated size of the array and the number of characters that you have stored into it.
Here is an example:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
char *a = NULL;
size_t size = 0;
size_t count = 0;
int c;
while ((c = getchar()) != EOF && c != '\n') {
if (count >= size) {
/* reallocate the buffer to 1.5x size */
size_t newsize = size + size / 2 + 16;
char *new_a = realloc(a, new_size);
if (new_a == NULL) {
fprintf("out of memory for %zu bytes\n", new_size);
free(a);
return 1;
}
a = new_a;
size = new_size;
}
a[count++] = c;
}
for (i = 0; i < count; i++) {
putchar(a[i]);
}
free(a);
return 0;
}
There are two ways to create space for the string without using dynamic memory allocation(malloc...). You can use a static array or an array with automatic storage duration, you need to specify a maximum amount, you might never reach. But always check against it.
#define BUFFER_SIZE 0x10000
Static
static char buffer[BUFFER_SIZE];
Or automatic (You need to ensure BUFFER_SIZE is smaller than the stack size)
int main() {
char buffer[BUFFER_SIZE];
...
};
There are also optimizations done by the operating system. It might lazily allocate the whole (static/automatic) buffer, so that only the used part is in the physical memory. (This also applies to the dynamic memory allocation functions.) I found out that calloc (for big chunks) just allocates the virtual memory for the program; memory pages are cleared only, when they are accessed (probably through some interrupts raised by the cpu). I compared it to an allocation with malloc and memset. The memset does unnessecary work, if not all bytes/pages of the buffer are accessed by the program.
If you cannot allocate a buffer with malloc..., create a static/automatic array with enough size and let the operating system allocate it for you. It does not occupy the same space in the binary, because it is just stored as a size.
I allocate a big memory , char* test= malloc(10000000); , then I put value on this memory , and do some work for each value.
What I want is , each 1000 index, I want to release all the memory until it.
For ex.
for(long i=0; i<10000000;i++)
DoSomeWork(test[i]);
if(i%1000==0)
releaseMemory(i-1000,i);
How can I do it in c?
I know that free can only free all of my allocate, but I don't want to wait to the end of work the free all the memory.
I want each 1000 works free all the 1000 back
I must to allocate all the memory in the begining of program.
What you want can be achieved by allocating the program in smaller chunks.
You have to adjust your algorithm to handle a bunch of small sub-arrays which you then can release after use.
In this case, it might be useful to allocate the chunks in reversed direction to give the libc the chance to release the freed memory to the underlying OS.
Let me enhance a bit here:
Assume you want an array with 10000000 (10 million) entries. Instead of allocating it as one chunk as depicted in the question, it could be possible to have
#define CHUNKSIZE 10000
#define ENTRYSIZE 8
#define NUM_CHUNKS 1000
void test(void)
{
void** outer_array = malloc(NUM_CHUNKS * sizeof(void*))
for (int i = 0; i < NUM_CHUNKS; i++) {
void * chunk = malloc(CHUNKSIZE * ENTRYSIZE);
outer_array[NUM_CHUNKS - 1 - i] = chunk;
// allocate them in reverse order
}
// now, set item #123456
size_t item_index = 123456;
// TODO check if the index is below the maximum
size_t chunk_index = item_index / CHUNKSIZE;
size_t index_into_chunk = item_index % CHUNKSIZE;
void * item_address = &outer_array[chunk_index][index_into_chunk * ENTRY_SIZE];
// after having processed one chunk, you can free it:
free(outer_array[0]);
outer_array[0] = NULL;
}
There are (roughly) two possibilities how a program can enhance the heap in order to allocate memory:
It can obtain a completely new memory block from the OS, indepedent from the "main address space". Then it can use it for allocation and return it to the OS as soon as it is free()d. This happens in some allocators if the allocation size is above a certain threshold.
It can enhance the program address space. Then, the new memory is added at the end. After free()ing the last memory block, the program address space can be reduced again. This happens in some allocators if the allocation size is below a certain threshold.
This way, your program's memory footprint decreases over time.
I allocate a 2D array of char * and every string length is 12.
50 rows and 2000000 columns.
Lets calculate it:
50*2000000 * (12(length)+8(for pointer)). I use 64 bit.
50*2000000 * 20 =2000000000 bits .. -> 2 GB.
When I check the memory monitor it shows that the process takes 4 GB.
(All that happened after allocation)
This is the code:
int col=2000000,row=50,i=0,j=0;
char *** arr;
arr=(char***)malloc(sizeof(char**)*row);
for(i=0;i<row;i++)
{
arr[i]=(char ** )malloc(sizeof(char*)*col);
for(j=0;j<col;j++)
{
arr[i][j]=(char*)malloc(12);
strcpy(arr[i][j],"12345678901");
arr[i][j][11]='\0';
}
}
May that be from the paging in Linux?
Each call of malloc is taking more memory than you ask. Malloc needs to store somewhere its internal info about allocated place, like size of allocated space, some info about neighbors chunks, etc. Also (very probably) each returned pointer is aligned to 16 bytes. In my estimation each allocation of 12 bytes takes 32 bytes of memory. If you want to save memory allocate all strings in one malloc and split them into sizes per 12 at your own.
Try the following:
int col=2000000,row=50,i=0,j=0;
char *** arr;
arr= malloc(sizeof(*arr)*row);
for(i=0;i<row;i++)
{
arr[i]= malloc(sizeof(*arr[i])*col);
char *colmem = malloc(12 * col);
for(j=0;j<col;j++)
{
arr[i][j] = colmem + j*12;
strcpy(arr[i][j],"12345678901");
}
}
I would re-write the code from scratch. For some reason, around 99% of all C programmers don't know how to correctly allocate true 2D arrays dynamically. I'm not even sure I'm one of the 1% who do, but lets give it a shot:
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
int main()
{
const int COL_N = 2000000;
const int ROW_N = 50;
char (*arr)[ROW_N] = malloc( sizeof(char[COL_N][ROW_N]) );
if(arr == NULL)
{
printf("Out of memory");
return 0;
}
for(int row=0; row<ROW_N; row++)
{
strcpy(arr[row], "12345678901");
puts(arr[row]);
}
free(arr);
return 0;
}
The important parts here are:
You should always allocate multi-dimensional arrays in adjacent memory cells or they are not arrays, but rather pointer-based lookup tables. Thus you only need one single malloc call.
This should save a bit of memory since you only need one pointer and it is allocated on the stack. No pointers are allocated on the heap.
Casting the return value of malloc is pointless (but not dangerous on modern compilers).
Ensure that malloc actually worked, particularly when allocating ridiculous amounts of memory.
strcpy copies the null termination, you don't need to do it manually.
There is no need for nested loops. You want to allocate a 2D array, not a 3D one.
Always clean up your own mess with free(), even though the OS might do it for you.
I'm allocating memory using malloc:
main()
{
int *array;
int i;
for(i = 0; i<40; i++)
{
array = malloc(100 * sizeof(int));
}
free(array);
}
This should allocate 15.625KB but if i run this same in valgrind, peak memory is 15.92KB. How it comes?
How to free all 40 pointers?
malloc always allocates a bit more than you asked for, necessary for internal accounting, caused by fragmentation, etc.
At a minimum - the size of each allocated block need to be stored somewhere, often some pointers (e.g. to the next allocated/free block) are also stored, and in some cases (e.g. debug builds) additional debugging information is also stored. Most implementation store as much information as possible in the unallocated space, so only a few bytes (e.g. size) would be stored in each allocated block.
As to fragmentation, many implementations have a minimal allocated size or round up the requested size to maintain some kind of alignment.
Regarding freeing all 40 pointers, you could for example have an array of pointers holding the pointers returned from malloc and go over it at the end of your function.
Something along the lines of:
main()
{
int *arrays[40];
int i;
for(i = 0; i<40; i++)
{
arrays[i]=malloc(100 * sizeof(int));
}
for(i = 0; i<40; i++)
{
free(arrays[i]);
}
}
free (array) should be inside the loop to free all pointers instead of the last one only.
When you call malloc(),The amount of memory actually used is slightly more than what is requested.This extra includes information that records how big the block is, where is next free block available etc.
This extra information is the reason so that free() function knows how much to free.
I need to identify a global struct (array), consisted of 4 integers.
The problem is, size of that struct array is not known in advance.
I'm trying to make sth. like this:
typedef struct
{
int value;
int MAXleft;
int MAXright;
int MAX;
} strnum;
int main ()
{
int size;
scanf("%d", &size);
strnum numbers[size];
return 0;
}
I heard that, it is possible to do this by pointers but I don't know how to do.
You can allocate the space for several structures dynamically like this:
strnum *numbers = malloc( size * sizeof(strnum) );
Then you can use it like any regular array (mostly).
It might be more convenient to use calloc instead of malloc. It allocates a number of blocks and fills them with zeros. Please note, that malloc doesn't clear allocated memory.
strnum *numbers = calloc( size, sizeof(strnum) );
When you are done with the memory don't forget to call free( numbers ), which will return the allocated memory back to a memory manager.
If you don't free it when it's no longer required and allocate some more and more, a memory footprint of the program will grow for no good reason as the program continues to work. This is called a memory leak and should be avoided. It might eventually result in the lack of memory for a program and unpredictable results.
And don't forget to include a stdlib.h header with prototypes of memory allocation functions.
You can start with malloc() and then do realloc() when the size keeps increasing.
I would suggest you to allocate a pool of 10 structures at once so that the number of calls to realloc() are reduced.
It is called Dynamic Memory Allocation
The thing you are trying to do can be done as follows:
strnum* number;
int size = 0;
scanf("%d",&size);
number = malloc(size * sizeof(strnum));
Also, don't forget to free the memory once you have done using the array.
free(number);