I was trying to execute a malloc sentence like the following one:
TYPODATO *prof=(TYPODATO *)malloc((size_t)H*V*B2*sizeof(TYPODATO));
with being H*V*B2*sizeof(TYPODATO) equal to 13037160840 B = 13.04 GB. If I execute the command free -m in the console to obtain the available memory in MB, I get 13486 MB = 13.486 GB.
However, the previous malloc statement returns a NULL pointer, indicating that such an amount of memory cannot be allocated. The requested amount of memory is not as much as the available in the system, but even if it was I should be able to successfully allocate it, cause the system can use the virtual memory. Why is not the system using the virtual memory in this case?
malloc() does not work this way. malloc() allocates a single contiguous block of memory.
You are trying to allocate more memory than contiguously available, hence the error.
Here's my question: Does calling free or delete ever release memory back to the "system". By system I mean, does it ever reduce the data segment of the process?
Let's consider the memory allocator on Linux, i.e ptmalloc.
From what I know (please correct me if I am wrong), ptmalloc maintains a free list of memory blocks and when a request for memory allocation comes, it tries to allocate a memory block from this free list (I know, the allocator is much more complex than that but I am just putting it in simple words). If, however, it fails, it gets the memory from the system using say sbrk or brk system calls. When a memory is free'd, that block is placed in the free list.
Now consider this scenario, on peak load, a lot of objects have been allocated on heap. Now when the load decreases, the objects are free'd. So my question is: Once the object is free'd will the allocator do some calculations to find whether it should just keep this object in the free list or depending upon the current size of the free list it may decide to give that memory back to the system i.e decrease the data segment of the process using sbrk or brk?
Documentation of glibc tells me that if the allocation request is much larger than page size, it will be allocated using mmap and will be directly released back to the system once free'd. Cool. But let's say I never ask for allocation of size greater than say 50 bytes and I ask a lot of such 50 byte objects on peak load on the system. Then what?
From what I know (correct me please), a memory allocated with malloc will never be released back to the system ever until the process ends i.e. the allocator will simply keep it in the free list if I free it. But the question that is troubling me is then, if I use a tool to see the memory usage of my process (I am using pmap on Linux, what do you guys use?), it should always show the memory used at peak load (as the memory is never given back to the system, except when allocated using mmap)? That is memory used by the process should never ever decrease(except the stack memory)? Is it?
I know I am missing something, so please shed some light on all this.
Experts, please clear my concepts regarding this. I will be grateful. I hope I was able to explain my question.
There isn't much overhead for malloc, so you are unlikely to achieve any run-time savings. There is, however, a good reason to implement an allocator on top of malloc, and that is to be able to trace memory leaks. For example, you can free all memory allocated by the program when it exits, and then check to see if your memory allocator calls balance (i.e. same number of calls to allocate/deallocate).
For your specific implementation, there is no reason to free() since the malloc won't release to system memory and so it will only release memory back to your own allocator.
Another reason for using a custom allocator is that you may be allocating many objects of the same size (i.e you have some data structure that you are allocating a lot). You may want to maintain a separate free list for this type of object, and free/allocate only from this special list. The advantage of this is that it will avoid memory fragmentation.
No.
It's actually a bad strategy for a number of reasons, so it doesn't happen --except-- as you note, there can be an exception for large allocations that can be directly made in pages.
It increases internal fragmentation and therefore can actually waste memory. (You can only return aligned pages to the OS, so pulling aligned pages out of a block will usually create two guaranteed-to-be-small blocks --smaller than a page, anyway-- to either side of the block. If this happens a lot you end up with the same total amount of usefully-allocated memory plus lots of useless small blocks.)
A kernel call is required, and kernel calls are slow, so it would slow down the program. It's much faster to just throw the block back into the heap.
Almost every program will either converge on a steady-state memory footprint or it will have an increasing footprint until exit. (Or, until near-exit.) Therefore, all the extra processing needed by a page-return mechanism would be completely wasted.
It is entirely implementation dependent. On Windows VC++ programs can return memory back to the system if the corresponding memory pages contain only free'd blocks.
I think that you have all the information you need to answer your own question. pmap shows the memory that is currenly being used by the process. So, if you call pmap before the process achieves peak memory, then no it will not show peak memory. if you call pmap just before the process exits, then it will show peak memory for a process that does not use mmap. If the process uses mmap, then if you call pmap at the point where maximum memory is being used, it will show peak memory usage, but this point may not be at the end of the process (it could occur anywhere).
This applies only to your current system (i.e. based on the documentation you have provided for free and mmap and malloc) but as the previous poster has stated, behavior of these is implmentation dependent.
This varies a bit from implementation to implementation.
Think of your memory as a massive long block, when you allocate to it you take a bit out of your memory (labeled '1' below):
111
If I allocate more more memory with malloc it gets some from the system:
1112222
If I now free '1':
___2222
It won't be returned to the system, because two is in front of it (and memory is given as a continous block). However if the end of the memory is freed, then that memory is returned to the system. If I freed '2' instead of '1'. I would get:
111
the bit where '2' was would be returned to the system.
The main benefit of freeing memory is that that bit can then be reallocated, as opposed to getting more memory from the system. e.g:
33_2222
I believe that the memory allocator in glibc can return memory back to the system, but whether it will or not depends on your memory allocation patterns.
Let's say you do something like this:
void *pointers[10000];
for(i = 0; i < 10000; i++)
pointers[i] = malloc(1024);
for(i = 0; i < 9999; i++)
free(pointers[i]);
The only part of the heap that can be safely returned to the system is the "wilderness chunk", which is at the end of the heap. This can be returned to the system using another sbrk system call, and the glibc memory allocator will do that when the size of this last chunk exceeds some threshold.
The above program would make 10000 small allocations, but only free the first 9999 of them. The last one should (assuming nothing else has called malloc, which is unlikely) be sitting right at the end of the heap. This would prevent the allocator from returning any memory to the system at all.
If you were to free the remaining allocation, glibc's malloc implementation should be able to return most of the pages allocated back to the system.
If you're allocating and freeing small chunks of memory, a few of which are long-lived, you could end up in a situation where you have a large chunk of memory allocated from the system, but you're only using a tiny fraction of it.
Here are some "advantages" to never releasing memory back to the system:
Having already used a lot of memory makes it very likely you will do so again, and
when you release memory the OS has to do quite a bit of paperwork
when you need it again, your memory allocator has to re-initialise all its data structures in the region it just received
Freed memory that isn't needed gets paged out to disk where it doesn't actually make that much difference
Often, even if you free 90% of your memory, fragmentation means that very few pages can actually be released, so the effort required to look for empty pages isn't terribly well spent
Many memory managers can perform TRIM operations where they return entirely unused blocks of memory to the OS. However, as several posts here have mentioned, it's entirely implementation dependent.
But lets say I never ask for allocation of size greater than say 50 bytes and I ask a lot of such 50 byte objects on peak load on the system. Then what ?
This depends on your allocation pattern. Do you free ALL of the small allocations? If so and if the memory manager has handling for a small block allocations, then this may be possible. However, if you allocate many small items and then only free all but a few scattered items, you may fragment memory and make it impossible to TRIM blocks since each block will have only a few straggling allocations. In this case, you may want to use a different allocation scheme for the temporary allocations and the persistant ones so you can return the temporary allocations back to the OS.
I run
calloc(1024 * 1024 * 1024, sizeof(int));
I check my program's usage and it's zero but I never made a call to free
Edit:
Running Debian Jessie
Edit 2:
I am using the top as the system monitor
Linux does lazy memory allocation. Only when a page fault occurs on a page marked as allocated does Linux actually consider it as being used. Try writing to a byte inside the allocated data and checking the memory usage again. For more information on memory allocation in Linux, check http://www.tldp.org/LDP/tlk/mm/memory.html.
Additionally, even though calloc zeroes the allocated memory, it can still be done in a lazy way that leads to the behavior you described. See How to lazy allocate zeroed memory?.
Your example only allocates a few K, which may be too small to see. But even if you were to ask for a much larger amount of memory, you won't see usage until you actually use the memory.
Linux by default does not actually allocate memory pages until you touch them somehow. Although calloc is supposed to initialize the memory to zero, this does not count as touching it as internally keeps track of uninitialized pages and returns zeros when read.
How does memory allocation take place in this case?
I observed that this is not the same as using a malloc on 1000000*10000 directly, which should have lead to 4*10GB (4 bytes per int) being allocated. However, this piece of code uses only 200MB on executing.
for(int i=0;i<1000000;i++)
{
int *A = (int*)malloc(sizeof(int)*10000);
}
As mentioned, there are differences in whether the memory is allocated in chunks or as one. The main reason why you're not seeing the memory being allocated is due to the operating systems lying about memory.
If you allocate a block of memory the allocation is virtual. Since all processes have lots of virtual memory available, it will usually succeed (unless you ask for insane amounts of memory, or the OS otherwise determines it's not going to work). The actual reservation of physical memory may occur after you actually use the memory.
So when you look at memory usage, there is not only one number but several. There is shared memory, there is memory that can't be paged out, there's the virtual allocated memory and then there's the actual memory in use.
If you change the code to actually use the memory, for example just write one byte to the allocated section, you will see completely different result. The OS has to handle the memory allocation and get the memory blocks in physical memory.
Also as mentioned you don't check that malloc succeeds. Maybe it succeeds for a few times and then doesn't allocate anything more.
This system also explains why sometimes a process might get killed due to low memory even though all allocations succeeded in all processes. The OS was just being too optimistic and thought it could give out more memory than actually was possible.
The difference is how the memory is allocated.
When you call 10K times malloc to allocate 10k of memory, 10G of virtual memory is allocated to your process. The resulting 10G of memory is not continuous. That is, you get 10k scattered blocks of memory whose size is 10K. Whereas, when you call malloc requesting 10G, malloc will try to allocate a continuos block of memory whose size is 10G.
According to the malloc manual page, malloc fails when it can't allocate the requested memory. You should check if the malloc is successful in your application, in order to understand if the memory has been correctly allocated.
for(int i=0;i<1000000;i++)
{
int *A = (int*)malloc(sizeof(int)*10000);
}
This is a perfect way of memory leak. You are allocating 1000000 times, each time sizeof(int)*10000 bytes. You're guaranteed to leak all of these allocated memory. As you declared A within the loop, so after the loop you do not have handle to that variable any more, and there's no way to free even the last chunk of memory you allocated.
And of course this is different from allocating 1000000*10000*sizeof(int) in one go. The former allocates 1000000 smaller chunks which are mostly to be scattered in many memory location. The latter tries to allocate one gigantic chunk, which likely to fails.
In my program, I create 100 threads, then wait for all of them to join and then repeat this operation again.
In each of the threads, I create some memory and free it. I am fairly sure, all the memory which I am creating in those threads are getting freed.
But, the SIZE output and RSS output of prstat are continously increasing. They are increasing regularly by 4M
What do these values indicate? Does it mean there is some memory leak?
EDIT:
My original intention was to find out what is the meaning of SIZE & RSS columns in prstat output. Also, does increase in the value shown by SIZE confirm the presence of memory leak in the code?
No, prstat SIZE increase doesn't necessarily indicate a memory leak. You can only guess that there might be a memory leak. Moreover, when using the standard Solaris malloc/free libraries, freed memory is not returned to the OS so the SIZE metric is never decreasing. You should really post a sample source code exhibiting the issue to help figuring out if you really experiencing a memory leak.
Yes, continuously increasing memory usage indicates a memory leak.