My professor gave us an "assignment" to find why realloc() won't work in this specific example.
I tried searching this site and I think that it won't work because there is no real way to determine the size of a memory block allocated with malloc() so realloc() doesn't know the new size of the memory block that it needs to reallocate.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <windows.h>
int main ()
{
MEMORYSTATUS memInfo;
memInfo.dwLength = sizeof(MEMORYSTATUS);
GlobalMemoryStatus(&memInfo);
double slobodno = memInfo.dwAvailVirtual/1024./1024.;
printf("%g MB\n",slobodno);
int br=0,i,j;
char **imena,*ime,*temp,*bbb=NULL;
imena=(char**) malloc(sizeof(char*)*(br+1));
while(1)
{
printf("Unesite ime: ");
ime=(char*) malloc(sizeof(char)*4000000);
gets(ime);
printf("%u\n", strlen(ime));
ime=(char*) realloc(ime,strlen(ime)+1);
GlobalMemoryStatus(&memInfo);
slobodno = memInfo.dwAvailVirtual/1024./1024.;
printf("%g MB\n",slobodno);
if (strcmp(ime,".")==0)
{free(ime);free(imena[br]);break;}
imena[br++]=ime;
imena=(char**) realloc(imena,sizeof(char*)*(br+1));
}
for (i=0;i<br-1;i++)
for (j=i+1;j<br;j++)
if (strcmp(imena[i],imena[j])>0)
{
temp=imena[i];
imena[i]=imena[j];
imena[j]=temp;
}
//ovde ide sortiranje
for (i=0;i<br;i++)
printf("%s\n",imena[i]);
for(i=0;i<br;i++)
free(imena[i]);
free(imena);
return 0;
}
Note: Professor added the lines for printing out the available memory so we can see that realloc() doesn't work. Every new string we enter just takes up sizeof(char)+4000000 bytes and can't be reallocated. I'm trying to find out why. Thanks in advance
I have a feeling that it has something to do with the page sizes on Windows.
For example, if you change 4000000 to 400000, you can see that the memory can be re-used.
I think that allocating 4000000 forces Windows to use "huge" page sizes (of 4MB) and for some (unknown to me) reason, realloc doesn't work on them in the way that you would expect (i.e. making unused memory available for other allocations).
This seems to be related to Realloc() does not correctly free memory in Windows, which mentions VirutalAlloc, but I'm not sure it clarifies the exact reason that realloc doesn't work.
From MSDN:
The memblock argument points to the beginning of the memory block. If memblock is NULL, realloc behaves the same way as malloc and allocates a new block of size bytes.
So the line ime=(char*) realloc(NULL,sizeof(char)*4000000); just malloc's new memory each time.
realloc doesn't free memory. These functions work with a big block of memory (called a "heap") and carve chunks out when you call realloc/malloc/calloc. If you need more memory than is in the heap at the moment, then the heap is expanded by asking the operating system for more memory.
When you call realloc to make a memory block smaller, all that happens is that the memory you don't need any more is made available for *alloc to hand out again on a different request. Neither realloc nor free ever shrink the heap to return memory back to the operating system. (If you need that to happen, you need to call the operating system's native memory allocation procedures, such as VirtualAlloc on Windows.)
The problem is not that realloc doesn't know the size of the original block. Even though that information is not available for us programmers, it is required to be available to realloc (even if the block was allocated with malloc or calloc).
The line
ime=(char*) realloc(ime,strlen(ime)+1);
looks like it is shrinking the previously allocated block to fit the contents exactly, but there is no requirement that is actually shrinks the block of memory and makes the remainder available again for a new allocation.
Edit
Another thing I just thought of: The shrinking with realloc might work OK, but the memory is not returned by the runtime library to the OS because the library keeps it around for a next allocation.
Only, the next allocation is for such a large block that it does not fit the memory freed up with realloc.
Related
I wrote a code to test to stress test the memory management of Linux and Windows OS. Just for further tests I went ahead and checked what values are present in the memory returned by malloc().
The values that are being return are all 0 (zero). I have read the man page of malloc, checked on both Windows and Linux, but I am not able to find the reason for this behavior. According to the manpage the
The malloc() function allocates size bytes and returns a pointer to the allocated memory. The memory is not initialized.
To clear the memory segment, one has to manually use memset().
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <stdbool.h>
int eat(long total,int chunk){
long i;
for(i=0;i<total;i+=chunk){
short *buffer=malloc(sizeof(char)*chunk);
if(buffer==NULL){
return -1;
}
printf("\nDATA=%d",*buffer);
memset(buffer,0,chunk);
}
return 0;
}
int main(int argc, char *argv[]){
int i,chunk=1024;
long size=10000;
printf("Got %ld bytes in chunks of %d...\n",size,chunk);
if(eat(size,chunk)==0){
printf("Done, press any key to free the memory\n");
getchar();
}else{
printf("ERROR: Could not allocate the memory");
}
}
Maybe I am missing something.
The code is adapted from here
EDIT: The problem has been been answered here for the GCC specific output. I believe Windows operating system would be also following the same procedures.
The memory returned by malloc() is not initialized, which means it may be anything. It may be zero, and it may not be; 'not initialized' means it could be anything (zero included). To get a guaranteed zeroed page use calloc().
The reason you are seeing zeroed pages (on Linux anyway) is that if an application requests new pages, these pages are zeroed by the OS (or more precisely they are copy-on-write images of a fixed page of zeroes known as the 'global zero page'). But if malloc() happens to use memory already allocated to the application which has since been freed (rather than expanding the heap) you may well see non-zero data. Note the zeroing of pages provided by the OS is an OS specific trait (primarily there for security so that one process doesn't end up with pages that happen to have data from another process), and is not mandated by the C standard.
You asked for a source for get_free_page zeroing the page: that says 'get_free_page() takes one parameter, a priority. ... It takes a page off of the free_page_list, updates mem_map, zeroes the page and returns the physical address of the page.' Here's another post that explains it well, and also explains why using calloc() is better than malloc()+memset().
Note that you aren't checking the entire allocated chunk for zero. You want something like this (untested):
int n;
char nonzero=0;
char *buffer=malloc(sizeof(char)*chunk);
if(buffer==NULL){
return -1;
}
for (n = 0; n<chunk; n++)
nonzero = nonzero || buffer[n];
printf("\nDATA=%s\n",nonzero?"nonzero":"zero");
You're absolutely correct; this behaviour is not guaranteed by the C language standard.
What you're observing could just be chance (you're only checking a couple of bytes in each allocation), or it could be an artifact of how your OS and C runtime library are allocating memory.
With this statement:
printf("\nDATA=%d",*buffer);
You only check the first sizeof(short) amount of bytes that have just been malloc()'ed (typically two (2) bytes).
Furthermore, the first time you may get lucky of getting all zeroes but after having had your program execute (and use) the heap memory then the contents-after-malloc() will be undefined.
the memory allocation function: calloc() will return a pointer to the 'new area and set all the bytes to zero.
The memory allocation function: realloc() will return a pointer to a (possibly new) area and have copied the bytes from the old area. The new area will be the 'new' requested length
The memory allocation function malloc will return a pointer to the new area but will not set the bytes to any specific value
The values that are being return are all 0 (zero).
But that's not guaranteed. It's because you're just running your program. If you malloc, random fill, and free a lot, you'll start noticing the previously freed memory is being reused, so you'll start to get non-zero chunks in your mallocs.
Yes you are right malloc() doesn't zero-initialize values. It arbitrarily pulls the amount of memory it's told to allocate from the heap, which essentially means there could be anything stored already within. You should therefore use malloc() only, where you're certain, that you are going to set it to a value. If you're going to do arithmetic with it right out of the box you might get some fishy results (I have already several times personally experienced this; you're going to have functional code with sometimes crazy output).
So set stuff you're not setting to a value to zero with memset(). Or my advise is to use calloc(). Calloc, other than malloc, does zero-initialize values. And is as far as I know faster than the combination of malloc() and memset() on the other hand malloc alone is faster than calloc. So try to find the fastest version possible at point of issue by keeping you're memory in form.
Look also at this post here: MPI matrix-vector-multiplication returns sometimes correct sometimes weird values. The question was a different one, but the cause the same.
I was curious whether there exists a dynamic memory allocation system that allows the programmer to free part of an allocated block.
For example:
char* a = malloc (40);
//b points to the split second half of the block, or to NULL if it's beyond the end
//a points to a area of 10 bytes
b = partial_free (a+10, /*size*/ 10)
Thoughts on why this is wise/unwise/difficult? Ways to do this?
Seems to me like it could be useful.
Thanks!
=====edit=====
after some research, it seems that the bootmem allocator for the linux kernel allows something similar to this operation with the bootmem_free call. So, I'm curious -- why is it that the bootmem allocator allows this, but ANSI C does not?
No there is no such function which allows parital freeing of memory.
You could however use realloc() to resize memory.
From the c standard:
7.22.3.5 The realloc function
#include <stdlib.h>
void *realloc(void *ptr, size_t size);
The realloc function deallocates the old object pointed to by ptr and returns a pointer to a new object that has the size specified by size. The contents of the new object shall be the same as that of the old object prior to deallocation, up to the lesser of the new and old sizes. Any bytes in the new object beyond the size of the old object have indeterminate values.
There is no ready-made function for this, but doing this isn't impossible. Firstly, there is realloc() . realloc takes a pointer to a block of memory and resizes the allocation to the size specified.
Now, if you have allocated some memory:
char * tmp = malloc(2048);
and you intend to deallocate the first, 1 K of memory, you may do:
tmp = realloc(foo, 2048-1024);
However, the problem in this case is that you cannot be certain that tmp will remain unchanged. Since, the function might just deallocate the entire 2K memory and move it elsewhere.
Now I'm not sure about the exact implementation of realloc, but from what I understand, the code:
myptr = malloc( x - y );
actually mallocs a new memory buffer of size x-y, then it copies the bytes that fit using memcpy and finally frees the original allocated memory.
This may create some potential problems. For example, the new reallocated memory may be located at a different address, so any past pointers you may have may become invalidated. Resulting in undefined runtime errors, segmentation faults and general debugging hell. So I would try to avoid resorting to this.
Firstly, I cannot think of any situation where you would be likely to need such a thing (when there exists realloc to increase/decrease the memory as mentioned in the answers).
I would like to add another thing. In whatever implementations I have seen of the malloc subsystem (which I admit is not a lot), malloc and free are implemented to be dependent on something called as the prefix byte(s). So whatever address is returned to you by malloc, internally the malloc subsystem will allocate some additional byte(s) of memory prior to the address returned to you, to store sanity check information which includes number of allocated bytes and possible what allocation policy you use (if your OS supports multiple mem allocation policies) etc. When you say something like free (x bytes), the malloc subsystem goes back to peek back into the prefix byte to sanity check and only if it finds the prefix in place does the free successfully happen. Therefore, it will not allow you to free some number of blocks starting in between.
I basically have this piece of code.
char (* text)[1][80];
text = calloc(2821522,80);
The way I calculated it, that calloc should have allocated 215.265045 megabytes of RAM, however, the program in the end exceeded that number and allocated nearly 700mb of ram.
So it appears I cannot properly know how much memory that function will allocate.
How does one calculate that propery?
calloc (and malloc for that matter) is free to allocate as much space as it needs to satisfy the request.
So, no, you cannot tell in advance how much it will actually give you, you can only assume that it's given you the amount you asked for.
Having said that, 700M seems a little excessive so I'd be investigating whether the calloc was solely responsible for that by, for example, a program that only does the calloc and nothing more.
You might also want to investigate how you're measuring that memory usage.
For example, the following program:
#include <stdio.h>
#include <stdlib.h>
#include <malloc.h>
int main (void) {
char (* text)[1][80];
struct mallinfo mi;
mi = mallinfo(); printf ("%d\n", mi.uordblks);
text = calloc(2821522,80);
mi = mallinfo(); printf ("%d\n", mi.uordblks);
return 0;
}
outputs, on my system:
66144
225903256
meaning that the calloc has allocated 225,837,112 bytes which is only a smidgeon (115,352 bytes or 0.05%) above the requested 225,721,760.
Well it depends on the underlying implementation of malloc/calloc.
It generally works like this - there's this thing called the heap pointer which points to the top of the heap - the area from where dynamic memory gets allocated. When memory is first allocated, malloc internally requests x amount of memory from the kernel - i.e. the heap pointer increments by a certain amount to make that space available. That x may or may not be equal to the size of the memory block you requested (it might be larger to account for future mallocs). If it isn't, then you're given at least the amount of memory you requested(sometimes you're given more memory because of alignment issues). The rest is made part of an internal free list maintained by malloc. To sum it up malloc has some underlying data structures and a lot depends on how they are implemented.
My guess is that the x amount of memory was larger (for whatever reason) than you requested and hence malloc/calloc was holding on to the rest in its free list. Try allocating some more memory and see if the footprint increases.
Simple question about the realloc function in C:
If I use realloc to shrink the memory block that a pointer is pointing to, does the "extra" memory get freed? Or does it need to be freed manually somehow?
For example, if I do
int *myPointer = malloc(100*sizeof(int));
myPointer = realloc(myPointer,50*sizeof(int));
free(myPointer);
Will I have a memory leak?
No, you won't have a memory leak. realloc will simply mark the rest "available" for future malloc operations.
But you still have to free myPointer later on. As an aside, if you use 0 as the size in realloc, it will have the same effect as free on some implementations. As Steve Jessop and R.. said in the comments, you shouldn't rely on it.
There is definitely not a memory leak, but any of at least 3 things could happen when you call realloc to reduce the size:
The implementation splits the allocated memory block at the new requested length and frees the unused portion at the end.
The implementation makes a new allocation with the new size, copies the old contents to the new location, and frees the entire old allocation.
The implementation does nothing at all.
Option 3 would be a rather bad implementation, but perfectly legal; there's still no "memory leak" because the whole thing will still be freed if you later call free on it.
As for options 1 and 2, which is better depends a lot on whether you favor performance or avoiding memory fragmentation. I believe most real-world implementations will lean towards doing option 1.
The new code still leaks the original allocation if the realloc fails. I expect most implementations won't ever fail to shrink a block, but it's allowed. The correct way to call realloc, whether growing or shrinking the block, is void *tmp = realloc(myPointer, 50*sizeof(int)); if (!tmp) { /* handle error somehow. myPointer still points to the old block, which is still allocated */ } myPointer = tmp;. – Steve Jessop 48 mins ago
Hey, I couldn't figure out how to reply to your comment, sorry.
Do I need to cast tmp to the type of myPointer? In this case, do I need to write
myPointer = (int*)tmp
Also, in this case, when I do
free(myPointer)
The memory pointed at by tmp will be freed as well, right? So no need to do
free(myPointer)
free(tmp)
In the way you have given your code, yes, it might have a leak. The idea of realloc is that it can return you a new location of your data. Like you do it in your question you lose that pointer that realloc sends you.
int *myPointer2 = realloc(myPointer,50*sizeof(int));
assert(myPointer2);
myPointer = myPointer2;
I guess the process for realloc() is, it first dealocates the previous memory block then, allocates again (My college teacher told that). If that remains the case, it would have already freed the extra 50 bytes. And if there is data in 100 byte memory, will get assigned to 50 reallocated bytes memory, which may, lead to loss of data. So there shouldn't be a Memory Leak.
I have a doubt regarding heap in program execution layout diagram of a C program.
I know that all the dynamically allocated memory is allotted in heap which grows dynamically. But I would like to know what is the max heap size for a C program ??
I am just attaching a sample C program ... here I am trying to allocate 1GB memory to string and even doing the memset ...
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
char *temp;
mybuffer=malloc(1024*1024*1024*1);
temp = memset(mybuffer,0,(1024*1024*1024*1));
if( (mybuffer == temp) && (mybuffer != NULL))
printf("%x - %x\n", mybuffer, &mybuffer[((1024*1024*1024*1)-1)]]);
else
printf("Wrong\n");
sleep(20);
free(mybuffer);
return 0;
}
If I run above program in 3 instances at once then malloc should fail atleast in one instance [I feel so] ... but still malloc is successfull.
If it is successful can I know how the OS takes care of 3GB of dynamically allocated memory.
Your machine is very probably overcomitting on RAM, and not using the memory until you actually write it. Try writing to each block after allocating it, thus forcing the operating system to ensure there's real RAM mapped to the address malloc() returned.
From the linux malloc page,
BUGS
By default, Linux follows an optimistic memory allocation strategy.
This means that when malloc() returns non-NULL there is no guarantee
that the memory really is available. This is a really bad bug. In
case it turns out that the system is out of memory, one or more pro‐
cesses will be killed by the infamous OOM killer. In case Linux is
employed under circumstances where it would be less desirable to sud‐
denly lose some randomly picked processes, and moreover the kernel ver‐
sion is sufficiently recent, one can switch off this overcommitting
behavior using a command like:
# echo 2 > /proc/sys/vm/overcommit_memory
See also the kernel Documentation directory, files vm/overcommit-
accounting and sysctl/vm.txt.
You're mixing up physical memory and virtual memory.
http://apollo.lsc.vsc.edu/metadmin/references/sag/x1752.html
http://en.wikipedia.org/wiki/Virtual_memory
http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory
Malloc will allocate the memory but it does not write to any of it. So if the virtual memory is available then it will succeed. It is only when you write something to it will the real memory need to be paged to the page file.
Calloc if memory serves be correctly(!) write zeros to each byte of the allocated memory before returning so will need to allocate the pages there and then.