I allocated some large chunks of memory through malloc and aligned_alloc, and then I setup a fence to a region inside the memory with size of one page size, using mprotect:
void *buf = malloc(128 * PAGE_SIZE);
int ret = mprotect(buf, PAGE_SIZE, PROT_NONE);
Now I'm done with the memory and is calling free(buf); to release it, my questions is do I need to reset mprotect before calling free, like this:
ret = mprotect(buf, PAGE_SIZE, PROT_READ|PROT_WRITE);
free(buf);
Or should I just do free? I read that glibc will sometimes reuse some of the previously allocated memory, so if this region of memory is returned to later malloc, will accessing it cause problems(since it's PROT_NONE)?
On top of #doron 's answer, I digged a little more because I was specifically working on Linux. According posix standard of mprotect:
The behavior of this function is unspecified if the mapping was not established by a call to mmap().
While on Linux, it is a bit different:
On Linux it is always permissible to call mprotect() on any address in a process's address space (except for the kernel vsyscall area). In particular it can be used to change existing code mappings to be writable.
That means it is indeed allowed to call mprotect on malloc'ed memory regions, but free will not reset it as there is no way to know the old protection flags, so I have to reset the flags before calling free.
In fact, this is exactly the problem I met, when the program runs for a while, it randomly crashes in malloc, that is because malloc is writing it's housekeeping data into the previously allocated memory and that memory was set to PROT_NONE by an earlier mprotect, after I set the memory to writable before calling free, the program never crash again.
That's being said, this is only on Linux, to write portable program, one should use mprotect only on mmap'ed memory.
Malloc allocated from the heap does not necessarily request the memory from the system. Likewise free does not necessarily return the memory to the system.
For what you are doing you should use mmap munmap which always goes to the system. If you use mfree there is no need to call mprotect beforehand.
Related
I run
calloc(1024 * 1024 * 1024, sizeof(int));
I check my program's usage and it's zero but I never made a call to free
Edit:
Running Debian Jessie
Edit 2:
I am using the top as the system monitor
Linux does lazy memory allocation. Only when a page fault occurs on a page marked as allocated does Linux actually consider it as being used. Try writing to a byte inside the allocated data and checking the memory usage again. For more information on memory allocation in Linux, check http://www.tldp.org/LDP/tlk/mm/memory.html.
Additionally, even though calloc zeroes the allocated memory, it can still be done in a lazy way that leads to the behavior you described. See How to lazy allocate zeroed memory?.
Your example only allocates a few K, which may be too small to see. But even if you were to ask for a much larger amount of memory, you won't see usage until you actually use the memory.
Linux by default does not actually allocate memory pages until you touch them somehow. Although calloc is supposed to initialize the memory to zero, this does not count as touching it as internally keeps track of uninitialized pages and returns zeros when read.
I used C library malloc to allocate 8MB memory, after using that memory I used free to free the 8MB memory.
But when i used malloc again to allocate 8MB of memory, it is allocating the same location as allocated previously.
How to avoid this problem and why does this occur?
EDIT: I'm implementing a tool to test main memory, if malloc allocates the same block of memory it is not possible to check the whole memory
This is not a problem per se, and is by design. Typical implementations of malloc will recycle blocks of memory for reasons of performance. In any case, since malloc returns addresses from a limited pool of values, there's no way it could guarantee not to recycle blocks.
The only sure fire way to stop malloc returning blocks that is has returned before is to stop freeing them. Of course, that's not really very practical.
I'm implementing a tool to test main memory. If malloc allocates the same block of memory it is not possible to check the whole memory.
Your tool to test main memory cannot be implemented with malloc, or indeed by any user mode program. Modern operating systems don't give you access to physical memory. Rather they present a virtualized view of memory. The addresses in your program are not physical addresses, they are virtual address. Testing physical memory requires you to go in at a much lower level than is possible from a user mode program.
This should help you
How malloc works?
To prevent this, allocate a few bytes using malloc/calloc and then free the bigger chunk of memory.
BTW this is not a wrong behavior to get the same memory address.
You might want to call system() to run a few linux commands (that provide detailed memory mgmt options) from your code.Main memory management cannot be done/tested using malloc/free, they are limited to operating on the memory allocated to your program when it is running.
I have allocated memory using malloc() in embedded Linux (around 10 MB). And checked the free memory it was 67080 kB but even after freeing it using free() it remains the same. It is only after the application is terminated the memory is available again. Does free() not make the freed memory available to the system, if so how to make it available.
free is a libc library call. it marks heap space as available for reuse. It does not guarantee that the associated virtual mapping will be released. Only after a dirty virtual mapping is released by your OS, then that memory will be system wide free again. This can only happen in chunks of pages.
Also if you allocated memory using malloc and family and didn't use it then it didn't actually consume physical memory until then - so freeing it will do nothing.
Does free() not make the freed memory available to the system.
No, usually not.
malloc() normally requests memory from the OS by the low level sbrk() or mmap() call. Once assigned to the application, free() just returns the memory to a memory pool that belongs to the application. That is, it's not returned back to the OS for use in another process. (Though some heuristics are in-place to do so in certain circumstances).
If swap space is in place, this becomes less of a problem, the OS will swap out the unused memory of applications to make room for additional physical memory that's required.
if so how to make it available.
Exit the application.
Or you would need to write your own memory allocator that could do this.(which in the general case is not an easy task especially if you don't want to sacrifice overhead and speed).
For a relatively big single piece of 10MB, you could simply request anonymous memory with mmap() and the memory will be released back to the OS when you munmap() that piece of memory.
Taken from the malloc 3 man page:
Normally, malloc() allocates memory from the heap, and adjusts the
size of the heap as required, using sbrk(2). When allocating blocks
of memory larger than MMAP_THRESHOLD bytes, the glibc malloc()
implementation allocates the memory as a private anonymous mapping
using mmap(2). MMAP_THRESHOLD is 128 kB by default, but is
adjustable using mallopt(3)
You can try to modify the MMAP_THRESHOLD so that by using malloc you are invoking mmap. If you do so, free guarantees that the memory allocated through mmap will return back to the system as soon as you free it.
Your malloc() calls obtain memory from the system, and maintain a heap data structure for keeping track of used and free memory within the process. Your free() calls return memory to the heap, where they are marked free, but they're still part of the process's memory.
If you want memory deallocation to return pages to the system, you'll have to write your own memory manager, but keep in mind that it'll only be able to completely free memory under the right conditions: It depends on the behavior of your application, whether your allocations and deallocations span page boundaries and cleanly de-fragment, etc. You need to understand the memory allocation behavior of your application to know whether this will be any benefit.
In linux, is calloc exactly the same as malloc + memset or does this depend on the exact linux/kernel version?
I am particularly interested in the question of whether you can calloc more RAM than you physically have (as you can certainly malloc more RAM than you physically have, you just can't write to it). In other words, does calloc always actually write to the memory you have been allocated as the specs suggest it should.
Of course, that depends on the implementation, but on a modern day Linux, you probably can. Easiest way is to try it, but I'm saying this based on the following logic.
You can malloc more than the memory you have (physical + virtual) because the kernel delays allocation of your memory until you actually use it. I believe that's to increase the chances of your program not failing due to memory limits, but that's not the question.
calloc is the same as malloc but zero initializes the memory. When you ask Linux for a page of memory, Linux already zero-initializes it. So if calloc can tell that the memory it asked for was just requested from the kernel, it doesn't actually have to zero initialize it! Since it doesn't, there is no access to that memory and therefore it should be able to request more memory than there actually is.
As mentioned in the comments this answer provides a very good explanation.
Whether calloc needs to write to the memory depends on whether it got the allocation from heap pages that are already assigned to the process, or it had to request more memory be assigned to the process by the kernel (using a system call such as sbrk() or mmap()). When the kernel assigns new memory to a process, it always zeroes it first (typically using a VM optimization, so it doesn't actually have to write to the page). But if it's reusing memory that was assigned previously, it has to use memset() to zero it.
It is not mentioned in the cited duplicate or here. Linux uses virtual memory and can allocate more memory that physically available in the system. A naive implementation of calloc() that simply does a malloc() plus memset() in user space will touch every page.
As Linux typically allocates in 4k chunks, all of the calloc() blocks are the same and initially read as zero. That is the same 4k chunk of memory can be mapped read only and the entire calloc() space in only taking up approximately size/4k * pointer_size + 4k. As the program writes to the calloc() space, a page fault happens and Linux will allocate a new page (4k) and resume the program.
This is called copy-on-write or COW for short. malloc() will generally behave the same way. For small sizes, the 'C' library will use binning and share 4k pages with other small sized allocation.
So, there are typically two layers involved.
Linux kernel's process memory management.
glibc heap management.
If the memory size requested is large and requires new memory allocated to the process, then most of the above applies (via Linux's process memory management). However, if the memory requested is small, then it will be like a malloc() plus memset(). In the large allocation size, the memset() is damaging as it touches the memory and the kernel thinks it needs a new page to allocate.
You can't malloc(3) more ram than the kernel gives the process doing the malloc(3)-ing. malloc(3) returns NULL if you can't allocate the amount of memory you want to allocate. In addition, malloc(3) and memset(3) are defined by your c library (libc.so) and not your kernel. The Linux kernel defines mmap(2) and other low-level memory allocation functions, not the *alloc(3) family (excluding kalloc()).
I have two related questions hence I am asking them in this single thread.
Q1) How can I confirm if my OS is clearing un-"free"'ed memory (allocated using malloc) automatically when a program terminates? I am using Ubuntu 11.04, 32-bit with gcc-4.5.2
As per a tutorial page by Steven Summit here, "Freeing unused memory (malloc'ed) is a good idea, but it's not mandatory. When your program exits, any memory which it has allocated but not freed should be automatically released. If your computer were to somehow ``lose'' memory just because your program forgot to free it, that would indicate a problem or deficiency in your operating system."
Q2) Suppose, foo.c mallocs a B-bytes memory. Later on, foo.c frees this B-bytes memory locations and returns it to the OS. Now my question is, can those PARTICULAR B-bytes of memory locations be re-allocated to foo.c (by the OS) in the current instance OR those B-bytes can't be allocated to foo.c untill its current instance terminates ?
EDIT : I would recommend everyone who reads my question to read the answer to a similar question here and here. Both answers explain the interaction and working of malloc() and free() in good detail without using very esoteric terms. To understand the DIFFERENCE between memory-management tools used by kernel (e.g. brk(), mmap()) and those used by the C-compiler (e.g. malloc(), free()), this is a MUST READ.
When a process ends either thru a terminating signal, e.g. SIGSEGV, or thru the _exit(2) system call (which happens to be called also when returning from main), all the process resources are released by the kernel. In particular, the process address space, including heap memory (allocated with mmap(2) (or perhaps sbrk(2)) syscall (used by malloc library function) is released.
Of course, the free library function either (often) makes the freed memory zone reusable by further future calls to malloc or (occasionally, for large memory zones) release some big memory chunk to the kernel using e.g. munmap(2) system call.
To know more about the memory map of process 1234, read sequentially the /proc/1234/maps pseudo-file (or /proc/self/maps from inside the process). The /proc file system is the preferred way to query the kernel about processes. (there is also /proc/self/statm and /proc/self/smaps and many other interesting things).
The detailed behavior of free and malloc is implementation dependent. You should view malloc as a way to get heap memory, and free as a way to say that a previously malloc-ed zone is useless, and the system (i.e. standard C library + kernel) can do whatever it wants with it.
Use valgrind to hunt memory leak bugs. You could also consider using Boehm's conservative garbage collector, i.e. use GC_malloc instead of malloc and don't bother about freeing manually memory.
Most Modern OS will reclaim the allocated memory so you need not worry about that.
The OS doesn't understand if your application/program leaked memory or not it simply reclaims what it allocated to an process once the process completes.
Yes freed memory can be reused(if needed) & the reuse can happen in the same instantiation.
Q1. You just have to assume that the operating system is behaving correctly.
Q2. There is no reason why the bytes can't be reallocated to foo.c it just depends on how the memory allocation routines work.
Q1) I'm not sure about how you can confirm. However, about the second paragraph, it's considered good style to always free whatever memory you allocate. A good explanation of this is found here: What REALLY happens when you don't free after malloc?.
Q2) Definitely; those bytes are usually the first to be reallocated (depending on the malloc implementation). For a great explanation, see: How do malloc() and free() work?.