Related
I have a char array, we know that that a char size is 1 byte. Now I have to collect some char -> getchar() of course and simultaneously increase the array by 1 byte (without malloc, only library: stdio.h)
My suggestion would be, pointing to the array and somehow increase that array by 1 till there are no more chars to get OR you run out of Memory...
Is it possible to increase char array while using it, WITHOUT malloc?
No.
You cannot increase the size of a fixed size array.
For that you need realloc() from <stdlib.h>, which it seems you are not "allowed" to use.
Is it possible to increase char array while using it, WITHOUT malloc?
Quick answer: No it is not possible to increase the size of an array without reallocating it.
Fun answer: Don't use malloc(), use realloc().
Long answer:
If the char array has static or automatic storage class, it is most likely impossible to increase its size at runtime because keeping it at the same address that would require objects that are present at higher addresses to be moved or reallocated elsewhere.
If the array was obtained by malloc, it might be possible to extend its size if no other objects have been allocated after it in memory. Indeed realloc() to a larger size might return the same address. The problem is it is impossible to predict and if realloc returns a different address, the current space has been freed so pointers to it are now invalid.
The efficient way to proceed with this reallocation is to increase the size geometrically, by a factor at a time, 2x, 1.5x, 1.625x ... to minimize the number of reallocations and keep linear time as the size of the array grows linearly. You would a different variable for the allocated size of the array and the number of characters that you have stored into it.
Here is an example:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
char *a = NULL;
size_t size = 0;
size_t count = 0;
int c;
while ((c = getchar()) != EOF && c != '\n') {
if (count >= size) {
/* reallocate the buffer to 1.5x size */
size_t newsize = size + size / 2 + 16;
char *new_a = realloc(a, new_size);
if (new_a == NULL) {
fprintf("out of memory for %zu bytes\n", new_size);
free(a);
return 1;
}
a = new_a;
size = new_size;
}
a[count++] = c;
}
for (i = 0; i < count; i++) {
putchar(a[i]);
}
free(a);
return 0;
}
There are two ways to create space for the string without using dynamic memory allocation(malloc...). You can use a static array or an array with automatic storage duration, you need to specify a maximum amount, you might never reach. But always check against it.
#define BUFFER_SIZE 0x10000
Static
static char buffer[BUFFER_SIZE];
Or automatic (You need to ensure BUFFER_SIZE is smaller than the stack size)
int main() {
char buffer[BUFFER_SIZE];
...
};
There are also optimizations done by the operating system. It might lazily allocate the whole (static/automatic) buffer, so that only the used part is in the physical memory. (This also applies to the dynamic memory allocation functions.) I found out that calloc (for big chunks) just allocates the virtual memory for the program; memory pages are cleared only, when they are accessed (probably through some interrupts raised by the cpu). I compared it to an allocation with malloc and memset. The memset does unnessecary work, if not all bytes/pages of the buffer are accessed by the program.
If you cannot allocate a buffer with malloc..., create a static/automatic array with enough size and let the operating system allocate it for you. It does not occupy the same space in the binary, because it is just stored as a size.
How do I free dynamically allocated memory?
Suppose input (assume it is given by user) is 1000 and now if I allocate memory of 1000 and after this(second time) if user gives input as 500 can I reuse already allocated memory ?
If user now inputs value as say 3000 , how do I go with it ? can I reuse already allocated 1000 blocks of memory and then create another 2000 blocks of memory ? or should I create all 3000 blocks of memory ?
which of these is advisable?
#include <stdio.h>
#include <stdlib.h>
typedef struct a
{
int a;
int b;
}aa;
aa* ptr=NULL;
int main() {
//code
int input=2;
ptr=malloc(sizeof(aa)*input);
for(int i=0;i<input;i++)
{
ptr[i].a=10;
ptr[i].b=20;
}
for(int i=0;i<input;i++)
{
printf("%d %d\n",ptr[i].a,ptr[i].b);
}
return 0;
}
I believe, you need to read about the "lifetime" of allocated memory.
For allocator functions, like malloc() and family, (quoting from C11, chapter ยง7.22.3, for "Memory management functions")
[...] The lifetime of an allocated object extends from the allocation
until the deallocation. [....]
So, once allocated, the returned pointer to the memory remains valid until it is deallocated. There are two ways it can be deallocated
Using a call to free() inside the program
Once the program terminates.
So, the allocated memory is available, from the point of allocation, to the termination of the program, or the free() call, whichever is earlier.
As it stands, there can be two aspects, let me clarify.
Scenario 1:
You allocate memory (size M)
You use the memory
You want the allocated memory to be re-sized (expanded/ shrinked)
You use some more
You're done using
is this is the flow you expect, you can use realloc() to resize the allocated memory size. Once you're done, use free().
Scenario 2:
You allocate memory (size M)
You use the memory
You're done using
If this is the case, once you're done, use free().
Note: In both the cases, if the program is run multiple times, there is no connection between or among the allocation happening in each individual invocation. They are independent.
When you use dynamically allocated memory, and adjust its size, it is important to keep track of exactly how many elements you have allocated memory for.
I personally like to keep the number of elements in use in variable named used, and the number of elements I have allocated memory for in size. For example, I might create a structure for describing one-dimensional arrays of doubles:
typedef struct {
size_t size; /* Number of doubles allocated for */
size_t used; /* Number of doubles in use */
double *data; /* Dynamically allocated array */
} double_array;
#define DOUBLE_ARRAY_INIT { 0, 0, NULL }
I like to explicitly initialize my dynamically allocated memory pointers to NULL, and their respective sizes to zero, so that I only need to use realloc(). This works, because realloc(NULL, size) is exactly equivalent to malloc(NULL). I also often utilize the fact that free(NULL) is safe, and does nothing.
I would probably write a couple of helper functions. Perhaps a function that ensures there is room for at_least entries in the array:
void double_array_resize(double_array *ref, size_t at_least)
{
if (ref->size < at_least) {
void *temp;
temp = realloc(ref->data, at_least * sizeof ref->data[0]);
if (!temp) {
fprintf(stderr, "double_array_resize(): Out of memory (%zu doubles).\n", at_least);
exit(EXIT_FAILURE);
}
ref->data = temp;
ref->size = at_least;
}
/* We could also shrink the array if
at_least < ref->size, but usually
this is not needed/useful/desirable. */
}
I would definitely write a helper function that not only frees the memory used, but also updates the fields to reflect that, so that it is completely safe to call double_array_resize() after freeing:
void double_array_free(double_array *ref)
{
if (ref) {
free(ref->data);
ref->size = 0;
ref->used = 0;
ref->data = NULL;
}
}
Here is how a program might use the above.
int main(void)
{
double_array stuff = DOUBLE_ARRAY_INIT;
/* ... Code and variables omitted ... */
if (some_condition) {
double_array_resize(&stuff, 321);
/* stuff.data[0] through stuff.data[320]
are now accessible (dynamically allocated) */
}
/* ... Code and variables omitted ... */
if (weird_condition) {
/* For some reason, we want to discard the
possibly dynamically allocated buffer */
double_array_free(&stuff);
}
/* ... Code and variables omitted ... */
if (other_condition) {
double_array_resize(&stuff, 48361242);
/* stuff.data[0] through stuff.data[48361241]
are now accessible. */
}
double_array_free(&stuff);
return EXIT_SUCCESS;
}
If I wanted to use the double_array as a stack, I might do
void double_array_clear(double_array *ref)
{
if (ref)
ref->used = 0;
}
void double_array_push(double_array *ref, const double val)
{
if (ref->used >= ref->size) {
/* Allocate, say, room for 100 more! */
double_array_resize(ref, ref->used + 100);
}
ref->data[ref->used++] = val;
}
double double_array_pop(double_array *ref, const double errorval)
{
if (ref->used > 0)
return ref->data[--ref->used];
else
return errorval; /* Stack was empty! */
}
The above double_array_push() reallocates for 100 more doubles, whenever the array runs out of room. However, if you pushed millions of doubles, this would mean tens of thousands of realloc() calls, which is usually considered wasteful. Instead, we usually apply a reallocation policy, that grows the size proportionally to the existing size.
My preferred policy is something like (pseudocode)
If (elements in use) < LIMIT_1 Then
Resize to LIMIT_1
Else If (elements in use) < LIMIT_2 Then
Resize to (elements in use) * FACTOR
Else
Resize to (elements in use) + LIMIT_2
End If
The LIMIT_1 is typically a small number, the minimum size ever allocated. LIMIT_2 is typically a large number, something like 220 (two million plus change), so that at most LIMIT_2 unused elements are ever allocated. FACTOR is between 1 and 2; many suggest 2, but I prefer 3/2.
The goal of the policy is to keep the number of realloc() calls at an acceptable (unnoticeable) level, while keeping the amount of allocated but unused memory low.
The final note is that you should only try to keep around a dynamically allocated buffer, if you reuse it for the same (or very similar) purpose. If you need an array of a different type, and don't need the earlier one, just free() the earlier one, and malloc() a new one (or let realloc() in the helpers do it). The C library will try to reuse the same memory anyway.
On current desktop machines, something like a hundred or a thousand malloc() or realloc() calls is probably unnoticeable compared to the start-up time of the program. So, it is not that important to minimize the number of those calls. What you want to do, is keep your code easily maintained and adapted, so logical reuse and variable and type names are important.
The most typical case where I reuse a buffer, is when I read text input line by line. I use the POSIX.1 getline() function to do so:
char *line = NULL;
size_t size = 0;
ssize_t len; /* Not 'used' in this particular case! :) */
while (1) {
len = getline(&line, &size, stdin);
if (len < 1)
break;
/* Have 'len' chars in 'line'; may contain '\0'! */
}
if (ferror(stdin)) {
fprintf(stderr, "Error reading standard input!\n");
exit(EXIT_FAILURE);
}
/* Since the line buffer is no longer needed, free it. */
free(line);
line = NULL;
size = 0;
As far of my concern if realloc fails we loose the information and realloc set the Buffer(pointer) to NULL
Consider de following program:
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
int main(void){
char *ptr = malloc(256);
if (!ptr){
printf("Error, malloc\n");
exit(1);
}
strcpy(ptr, "Michi");
ptr = realloc (ptr, 1024 * 102400000uL); /* I ask for a big chunk here to make realloc to fail */
if (!ptr){
printf("Houston we have a Problem\n");
}
printf("PTR = %s\n", ptr);
if (ptr){
free(ptr);
ptr = NULL;
}
}
And the output of course is:
Houston we have a Problem
PTR = (null)
I just lost the information inside ptr.
Now to fix this we should use a temporary buffer(pointer) before to see if we get that chunk of memory and if we get it we can use it, if not we still have the main buffer(pointer) safe.
Now please consider the following program, where instead of calling realloc I call malloc on a temporary buffer(pointer):
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
int main(void){
char *ptr = malloc(256);
char *tmpPTR = NULL;
if (!ptr){
printf("Error, malloc\n");
exit(1);
}
strcpy(ptr, "Michi");
tmpPTR = malloc (1024 * 102400000uL);
if (tmpPTR){
strcpy(tmpPTR, ptr);
strcat(tmpPTR, " - Aloha");
if (ptr){
free(ptr);
ptr = NULL;
}
}else{
printf("Malloc failed on tmpPTR\n\n");
}
if (ptr){
printf("PTR = %s\n", ptr);
free(ptr);
ptr = NULL;
}else if (tmpPTR){
printf("tmpPTR = %s\n", tmpPTR);
free(tmpPTR);
ptr = NULL;
}
}
And the output is:
Malloc failed on tmpPTR
PTR = Michi
Now why should I ever use realloc?
Is there any benefit of using realloc instead of malloc based on this context?
Your problem is with how you use realloc. You don't have to assign the result of realloc to the same pointer that you re-allocate. And as you point out it even poses a problem if the realloc fails. If you immediately assign the result to ptr then indeed you lose the previous buffer when something goes wrong. However, if you assign the result of realloc to tmpPTR, then ptr remains fine, even if the realloc fails. Use realloc as follows:
char * ptr = malloc(256);
if(!ptr){
return 1;
}
char * tmpPTR = realloc(ptr, 512);
if(!tmpPTR){
printf("Houston, we have a problem");
// ptr is fine
}else{
ptr = tmpPTR;
}
// ptr is realloc()ed
In the above code, tmpPTR is not a new (temporary) buffer, but just a (temporary) pointer. If the realloc is succesful it points to the same buffer (but possibly in a different location), and if it fails it is NULL. realloc doesn't always need to allocate a new buffer, but may be able to change the existing one to fit the new size. But if it fails, the original buffer will not be changed.
If you use malloc with a temporary buffer, then (for this example) you need at least 256 + 512 = 768 bytes and you always need to copy the old data. realloc may be able to re-use the old buffer so copying is not necessary and you don't use more memory than requested.
You can use your malloc approach, but realloc is almost always more efficient.
The realloc scheme is simple. You do not need a separate call to malloc. For example if you initially have 256 bytes allocated for ptr, simply use a counter (or index, i below) to keep track of how much of the memory within the block allocated to ptr has been used, and when the counter reaches the limit (1 less than the max for 0-based indexes, or 2 less than the max if you are using ptr as a string), realloc.
Below shows a scheme where you are simply adding 256 additional bytes to ptr each time the allocation limit is reached:
int i = 0, max = 256;
char *ptr = malloc(max);
/* do whatever until i reaches 255 */
if (i + 1 >= max) {
void *tmp = realloc (ptr, max + 256);
if (!tmp) {
fprintf (stderr, "error: realloc - memory exhausted.\n")
/* handle error */
}
ptr = tmp;
max += 256;
}
note: your handle error can exit whatever loop you are in to preserve the existing data in ptr. You do not need to exit at that point.
The advantage of realloc over malloc is that it may be able to extend the original dynamic memory area so without the need to copy all the previous elements; you can't do that with malloc1. And whether this optimization is available costs no work to you.
Let's assume you have a previously allocated pointer:
char *some_string = malloc(size); // assume non-NULL
Then
if (realloc_needed) {
char *tmp = realloc(some_string, new_size);
if ( tmp == NULL )
// handle error
else
some_string = tmp; // (1)
At (1), you update the old pointer with the new one. Two things can happen: the address has effectively changed (and the elements been automatically copied) or it hasn't - you don't really care. Either way, your data is now at some_string.
Only the actual implementation (OS / libc) knows whether it's possible to enlarge the block: you don't get to see it, it's an implementation detail. You can however check your implementation's code and see how it's implemented.
Now to fix this we should use a temporary buffer(pointer) before to see if we get that chunk of memory and if we get it we can use it, if not we still have the main buffer(pointer) safe.
That not only doesn't help, it makes things worse because now you no longer have the pointer to the block you tried to reallocate. So how can you free it?
So it:
Wastes memory.
Require an extra allocate, copy, and free.
Makes the realloc more likely to fail because of 1.
Leaks memory since the pointer to the block you tried to reallocate is lost.
So no, that's not a good way to handle realloc returning NULL. Save the original pointer when you call realloc so you can handle failure sanely. The point of realloc to save you from having to manage two copies of the data and to avoid even making them when that's possible. So let realloc do this work for you whenever you can.
It is technically malloc(size) that is unneeded, because realloc(NULL, size) performs the exact same task.
I often read inputs of indeterminate length. As in the following function example, I rarely use malloc(), and instead use realloc() extensively:
#include <stdlib.h>
#include <errno.h>
struct record {
/* fields in each record */
};
struct table {
size_t size; /* Number of records allocated */
size_t used; /* Number of records in table */
struct record item[]; /* C99 flexible array member */
};
#define MAX_ITEMS_PER_READ 1
struct table *read_table(FILE *source)
{
struct table *result = NULL, *temp;
size_t size = 0;
size_t used = 0, n;
int err = 0;
/* Read loop */
while (1) {
if (used + MAX_ITEMS_PER_READ > size) {
/* Array size growth policy.
* Some suggest doubling the size,
* or using a constant factor.
* Here, the minimum is
* size = used + MAX_ITEMS_PER_READ;
*/
const size_t newsize = 2*MAX_ITEMS_PER_READ + used + used / 2;
temp = realloc(result, sizeof (struct table) +
newsize * sizeof (result->item[0]));
if (!temp) {
err = ENOMEM;
break;
}
result = temp;
size = newsize;
}
/* Read a record to result->item[used],
* or up to (size-used) records starting at result->item + used.
* If there are no more records, break.
* If an error occurs, set err = errno, and break.
*
* Increment used by the number of records read: */
used++;
}
if (err) {
free(result); /* NOTE: free(NULL) is safe. */
errno = err;
return NULL;
}
if (!used) {
free(result);
errno = ENODATA; /* POSIX.1 error code, not C89/C99/C11 */
return NULL;
}
/* Optional: optimize table size. */
if (used < size) {
/* We don't mind even if realloc were to fail here. */
temp = realloc(result, sizeof (struct table) +
used * sizeof table->item[0]);
if (temp) {
result = temp;
size = used;
}
}
result->size = size;
result->used = used;
errno = 0; /* Not normally zeroed; just my style. */
return result;
}
My own practical reallocation policies tend to be very conservative, limiting the size increase to a megabyte or so. There is a very practical reason for this.
On most 32-bit systems, userspace applications are limited to 2 to 4 gigabyte virtual address space. I wrote and ran simulation systems on a lot of different x86 systems (32-bit), all with 2 to 4 GB of memory. Usually, most of that memory is needed for a single dataset, which is read from disk, and manipulated in place. When the data is not in final form, it cannot be directly memory-mapped from disk, as a translation -- usually from text to binary -- is needed.
When you use realloc() to grow the dynamically allocated array to store such huge (on 32-bit) datasets, you are only limited by the available virtual address space (assuming there is enough memory available). (This especially applies to 32-bit applications on 64-bit systems.)
If, instead, you use malloc() -- i.e., when you notice your dynamically allocated array is not large enough, you malloc() a new one, copy the data over, and discard the old one --, your final data set size is limited to a lesser size, the difference depending on your exact array size growth policy. If you use the typical double when resizing policy, your final dataset is limited to about half (the available virtual address space, or available memory, whichever is smaller).
On 64-bit systems with lots and lots of memory, realloc() still matters, but is much more of a performance issue, rather than on 32-bit, where malloc() is a limiting factor. You see, when you use malloc() to allocate a completely new array, and copy the old data to the new array, the resident set size -- the actual amount of physical RAM needed by your application -- is larger; you use 50% more physical RAM to read the data than you would when using realloc(). You also do a lot of large memory-to-memory copies (when reading a huge dataset), which are limited to physical RAM bandwidth, and indeed slow down your application (although, if you are reading from a spinning disk, that is the actual bottleneck anyway, so it won't matter much).
The nastiest effect, and the most difficult to benchmark, are the indirect effects. Most operating systems use "free" RAM to cache recently accessed files not modified yet, and this really does decrease the wall clock time used by most workloads. (In particular, caching typical libraries and executables may shave off seconds from the startup time of large application suites, if the storage media is slow (ie. a spinning disk, and not a SSD).) Your memory-wasting malloc()-only approach gobbles up much more actual physical RAM than needed, which evicts cached, often useful, files from memory!
You might benchmark your program, and note that there is no real difference in run times between using your malloc()-only approach and realloc() approach I've shown above. But, if it works with large datasets, the users will notice that using the malloc()-only program slows down other programs much more than the realloc()-using program, with the same data!
So, although on 64-bit systems with lots of RAM using malloc() only is basically an inefficient way to approach things, on 32-bit systems it limits the size of dynamically allocated arrays when the final size is unknown beforehand. Only using realloc() can you there achieve the maximum possible dataset size.
Your assumption is wrong. Please do note that a pointer is not a buffer. When the function realloc() succeeds, it deallocates the old pointer(frees the original buffer) and return a new pointer to the new allocation(buffer), but when it fails, it leaves the old buffer intact and returns NULL.
So, you do not need a temporary buffer. You need a temporary pointer. I am going to borrow the example from kninnug, this is what you need to do:
char * ptr = malloc(256);
if (!ptr) {
return 1;
}
char * tmpPTR = realloc(ptr, 512);
if (!tmpPTR) {
printf("Houston, we have a problem");
// ptr is fine
}
else {
ptr = tmpPTR;
}
// ptr is realloc()ed
I want to create a program that will simulate an out-of-memory (OOM) situation on a Unix server. I created this super-simple memory eater:
#include <stdio.h>
#include <stdlib.h>
unsigned long long memory_to_eat = 1024 * 50000;
size_t eaten_memory = 0;
void *memory = NULL;
int eat_kilobyte()
{
memory = realloc(memory, (eaten_memory * 1024) + 1024);
if (memory == NULL)
{
// realloc failed here - we probably can't allocate more memory for whatever reason
return 1;
}
else
{
eaten_memory++;
return 0;
}
}
int main(int argc, char **argv)
{
printf("I will try to eat %i kb of ram\n", memory_to_eat);
int megabyte = 0;
while (memory_to_eat > 0)
{
memory_to_eat--;
if (eat_kilobyte())
{
printf("Failed to allocate more memory! Stucked at %i kb :(\n", eaten_memory);
return 200;
}
if (megabyte++ >= 1024)
{
printf("Eaten 1 MB of ram\n");
megabyte = 0;
}
}
printf("Successfully eaten requested memory!\n");
free(memory);
return 0;
}
It eats as much memory as defined in memory_to_eat which now is exactly 50 GB of RAM. It allocates memory by 1 MB and prints exactly the point where it fails to allocate more, so that I know which maximum value it managed to eat.
The problem is that it works. Even on a system with 1 GB of physical memory.
When I check top I see that the process eats 50 GB of virtual memory and only less than 1 MB of resident memory. Is there a way to create a memory eater that really does consume it?
System specifications: Linux kernel 3.16 (Debian) most likely with overcommit enabled (not sure how to check it out) with no swap and virtualized.
When your malloc() implementation requests memory from the system kernel (via an sbrk() or mmap() system call), the kernel only makes a note that you have requested the memory and where it is to be placed within your address space. It does not actually map those pages yet.
When the process subsequently accesses memory within the new region, the hardware recognizes a segmentation fault and alerts the kernel to the condition. The kernel then looks up the page in its own data structures, and finds that you should have a zero page there, so it maps in a zero page (possibly first evicting a page from page-cache) and returns from the interrupt. Your process does not realize that any of this happened, the kernels operation is perfectly transparent (except for the short delay while the kernel does its work).
This optimization allows the system call to return very quickly, and, most importantly, it avoids any resources to be committed to your process when the mapping is made. This allows processes to reserve rather large buffers that they never need under normal circumstances, without fear of gobbling up too much memory.
So, if you want to program a memory eater, you absolutely have to actually do something with the memory you allocate. For this, you only need to add a single line to your code:
int eat_kilobyte()
{
if (memory == NULL)
memory = malloc(1024);
else
memory = realloc(memory, (eaten_memory * 1024) + 1024);
if (memory == NULL)
{
return 1;
}
else
{
//Force the kernel to map the containing memory page.
((char*)memory)[1024*eaten_memory] = 42;
eaten_memory++;
return 0;
}
}
Note that it is perfectly sufficient to write to a single byte within each page (which contains 4096 bytes on X86). That's because all memory allocation from the kernel to a process is done at memory page granularity, which is, in turn, because of the hardware that does not allow paging at smaller granularities.
All the virtual pages start out copy-on-write mapped to the same zeroed physical page. To use up physical pages, you can dirty them by writing something to each virtual page.
If running as root, you can use mlock(2) or mlockall(2) to have the kernel wire up the pages when they're allocated, without having to dirty them. (normal non-root users have a ulimit -l of only 64kiB.)
As many others suggested, it seems that the Linux kernel doesn't really allocate the memory unless you write to it
An improved version of the code, which does what the OP was wanting:
This also fixes the printf format string mismatches with the types of memory_to_eat and eaten_memory, using %zi to print size_t integers. The memory size to eat, in kiB, can optionally be specified as a command line arg.
The messy design using global variables, and growing by 1k instead of 4k pages, is unchanged.
#include <stdio.h>
#include <stdlib.h>
size_t memory_to_eat = 1024 * 50000;
size_t eaten_memory = 0;
char *memory = NULL;
void write_kilobyte(char *pointer, size_t offset)
{
int size = 0;
while (size < 1024)
{ // writing one byte per page is enough, this is overkill
pointer[offset + (size_t) size++] = 1;
}
}
int eat_kilobyte()
{
if (memory == NULL)
{
memory = malloc(1024);
} else
{
memory = realloc(memory, (eaten_memory * 1024) + 1024);
}
if (memory == NULL)
{
return 1;
}
else
{
write_kilobyte(memory, eaten_memory * 1024);
eaten_memory++;
return 0;
}
}
int main(int argc, char **argv)
{
if (argc >= 2)
memory_to_eat = atoll(argv[1]);
printf("I will try to eat %zi kb of ram\n", memory_to_eat);
int megabyte = 0;
int megabytes = 0;
while (memory_to_eat-- > 0)
{
if (eat_kilobyte())
{
printf("Failed to allocate more memory at %zi kb :(\n", eaten_memory);
return 200;
}
if (megabyte++ >= 1024)
{
megabytes++;
printf("Eaten %i MB of ram\n", megabytes);
megabyte = 0;
}
}
printf("Successfully eaten requested memory!\n");
free(memory);
return 0;
}
A sensible optimisation is being made here. The runtime does not actually acquire the memory until you use it.
A simple memcpy will be sufficient to circumvent this optimisation. (You might find that calloc still optimises out the memory allocation until the point of use.)
Not sure about this one but the only explanation that I can things of is that linux is a copy-on-write operating system. When one calls fork the both processes point to the same physically memory. The memory is only copied once one process actually WRITES to the memory.
I think here, the actual physical memory is only allocated when one tries to write something to it. Calling sbrk or mmap may well only update the kernel's memory book-keep. The actual RAM may only be allocated when we actually try to access the memory.
Basic Answer
As mentioned by others, the allocation of memory, until used, does not always commit the necessary RAM. This happens if you allocate a buffer larger than one page (usually 4Kb on Linux).
One simple answer would be for your "eat memory" function to always allocate 1Kb instead of increasingly larger blocks. This is because each allocated blocks start with a header (a size for allocated blocks). So allocating a buffer of a size equal to or less than one page will always commit all of those pages.
Following Your Idea
To optimize your code as much as possible, you want to allocate blocks of memory aligned to 1 page size.
From what I can see in your code, you use 1024. I would suggest that you use:
int size;
size = getpagesize();
block_size = size - sizeof(void *) * 2;
What voodoo magic is this sizeof(void *) * 2?! When using the default memory allocation library (i.e. not SAN, fence, valgrin, ...), there is a small header just before the pointer returned by malloc() which includes a pointer to the next block and a size.
struct mem_header { void * next_block; intptr_t size; };
Now, using block_size, all your malloc() should be aligned to the page size we found earlier.
If you want to properly align everything, the first allocation needs to use an aligned allocation:
char *p = NULL;
int posix_memalign(&p, size, block_size);
Further allocations (assuming your tool only does that) can use malloc(). They will be aligned.
p = malloc(block_size);
Note: please verify that it is indeed aligned on your system... it works on mine.
As a result you can simplify your loop with:
for(;;)
{
p = malloc(block_size);
*p = 1;
}
Until you create a thread, the malloc() does not use mutexes. But it still has to look for a free memory block. In your case, though, it will be one after the other and there will be no holes in the allocated memory so it will be pretty fast.
Can it be faster?
Further note about how memory is generally allocated in a Unix system:
the malloc() function and related functions will allocate a block in your heap; which at the start is pretty small (maybe 2Mb)
when the existing heap is full it gets grown using the sbrk() function; as far as your process is concerned, the memory address always increases, that's what sbrk() does (contrary to MS-Windows which allocates blocks all over the place)
using sbrk() once and then hitting the memory every "page size" bytes would be faster than using malloc()
char * p = malloc(size); // get current "highest address"
p += size;
p = (char*)((intptr_t)p & -size); // clear bits (alignment)
int total_mem(50 * 1024 * 1024 * 1024); // 50Gb
void * start(sbrk(total_mem));
char * end((char *)start + total_mem);
for(; p < end; p += size)
{
*p = 1;
}
note that the malloc() above may give you the "wrong" start address. But your process really doesn't do much, so I think you'll always be safe. That for() loop, however, is going to be as fast as possible. As mentioned by others, you'll get the total_mem of virtual memory allocated "instantly" and then the RSS memory allocated each time you write to *p.
WARNING: Code not tested, use at your own risk.
I'm having memory leaks in a larger program and I believe this is the cause of it.
#include <stdlib.h>
#include <Windows.h>
typedef struct _struct{
char* name;
} str;
int main() {
system("PAUSE");
str* Character = (str*)malloc(sizeof(str) * 20000);
for(int i = 0; i < 20000; i++){
Character[i].name = (char*)malloc(20000); // Assign memory.
}
for(int i = 0; i < 20000; i++){
free(Character[i].name); // Free memory.
}
free(Character);
system("PAUSE");
}
Memory at first pause: ~500K.
Memory at second pause: ~1.7M.
Using VS2012 for testing. Any ideas?
How are you measuring the amount of memory occupied by the program? One thing off the top of my head is that you're looking at the size of the working set the OS is keeping track of. Since you've allocated and freed a lot of memory, the size of that set has increased. Some OSs will adjust the size of the working set after a while, some won't. What OS are we looking at here?
When you call malloc, memory is allocated on the heap. If there is insufficient space left on the heap, the program will ask the OS for more memory and another chunk is acquired. Memory acquired from the OS is usually not returned until the program finishes (although this is up to the OS).
Program size alone can not normally be used to check for memory leaks! Use Valgrind or a similar tool to check for memory that never gets freed.
str* Character = (str*)malloc(sizeof(str) * 20000);
In the above line you are allocating the memory by finding the size of the struct. Here the size of the structure you will get will be the size of the pointer width and not the size of the char.
suppose for example if the pointer width is 32 bit the it will allocate (4 * 20000) = 80000 bytes.
If you want to allocate for 20000 struct's,
str* Character = (str*)malloc(sizeof(char) * 20000);