int brk(void *end_data_segment);
void *sbrk(intptr_t increment);
Calling sbrk() with an increment of
0
can be used to find the current location of the program break.
What is program break? Where does it start from,0x00?
Oversimplifying:
A process has several segments of memory:
Code (text) segment, which contains the code to be executed.
Data segment, which contains data the compiler knows about (globals and statics).
Stack segment, which contains (drumroll...) the stack.
(Of course, nowadays it's much more complex. There is a rodata segment, a uninitialized data segment, mappings allocated via mmap, a vdso, ...)
One traditional way a program can request more memory in a Unix-like OS is to increment the size of the data segment, and use a memory allocator (i.e. malloc() implementation) to manage the resulting space. This is done via the brk() system call, which changes the point where the data segment "breaks"/ends.
A program break is end of the process's data segment. AKA...
the program break is the first
location after the end of the
uninitialized data segment
As to where it starts from, it's system dependent but probably not 0x00.
These days, sbrk(2) (and brk) are nearly obsolete system calls (and you can nearly forget about them and ignore the old notion of break; focus on understanding mmap(2)). Notice that the sbrk(2) man page says in its NOTES :
Avoid using brk() and sbrk(): the malloc(3) memory allocation package
is the portable and comfortable way of allocating memory.
(emphasis mine)
Most implementations of malloc(3) (notably the one in musl-libc) are rather using mmap(2) to require memory - and increase their virtual address space - from the kernel (look at that virtual address space wikipage, it has a nice picture). Some malloc-s use sbrk for small allocations, mmap for large ones.
Use strace(1) to find out the system calls (listed in syscalls(2)) done by some given process or command. BTW you'll then find that bash and ls (and probably many other programs) don't make a single call to sbrk.
Explore the virtual address space of some process by using proc(5). Try cat /proc/$$/maps and cat /proc/self/maps and even cat /proc/$$/smaps and read a bit to understand the output.
Be aware of ASLR & vdso(7).
And sbrk is not very thread friendly.
(my answer focuses on Linux)
You are saying that sbrk() is an obsolute system call and that we should use malloc(), but malloc(), according to her documentation, when allocating less memory than 128 KiB (32 pages) uses it. So we shouldn´t use sbrk() directly, but malloc() use it, if allocation is bigger than 128 KiB then malloc() uses mmap() that allocates private pages to the userspace.
Finally its a good idea to understand sbrk(), at least for understanding the "Program Break" concept.
Based on the following widely used diagram:
program break, which is also known as brk in many articles, points to the address of heap segment's end.
When you call malloc, it changes the address of program break.
Related
I'm new to C and heap memory, still struggling to understand dynamic memory allocation.
I traced Linux system calls and found that if I use malloc to request a small amount of heap memory, then malloc calls brk internally.
But if I use malloc to request a very large amount of heap memory, then malloc calls mmap internally.
So there must be a big difference between brk and mmap, but theoretically we should be able to use brk to allocate heap memory regardless of the requested size. So why does malloc call mmap when allocating a large amount of memory?
so why malloc calls mmap when it comes to allocate a large size of memory?
The short answer is for improved efficiency on newer implementations of Linux, and the updated memory allocation algorithms that come with them. But keep in mind that this is a very implementation dependent topic, and the whys and wherefores would vary greatly for differing vintages and flavors of the specific Linux OS being discussed.
Here is fairly recent write-up regarding the low-level parts mmap() and brk() play in Linux memory allocation. And, a not so recent, but still relevant Linux Journal article that includes some content that is very on-point for the topic here, including this:
For very large requests, malloc() uses the mmap() system call to find
addressable memory space. This process helps reduce the negative
effects of memory fragmentation when large blocks of memory are freed
but locked by smaller, more recently allocated blocks lying between
them and the end of the allocated space. In this case, in fact, had
the block been allocated with brk(), it would have remained unusable
by the system even if the process freed it.
(emphasis mine)
Regarding brk():
incidentally, "...mmap() didn't exist in the early versions of Unix. brk() was the only way to increase the size of the data segment of the process at that time. The first version of Unix with mmap() was SunOS in the mid 80's, the first open-source version was BSD-Reno in 1990.". Since that time, modern implementation of memory allocation algorithms have been refactored with many improvements, greatly reducing the need for them to include using brk().
mmap (when used with MAP_ANONYMOUS) allocates a chunk of RAM that can be placed anywhere within the process's virtual address space, and that can be deallocated later (with munmap) independently of all other allocations.
brk changes the ending address of a single, contiguous "arena" of virtual address space: if this address is increased it allocates more memory to the arena, and if it is decreased, it deallocates the memory at the end of the arena. Therefore, memory allocated with brk can only be released back to the operating system when a continuous range of addresses at the end of the arena is no longer needed by the process.
Using brk for small allocations, and mmap for big allocations, is a heuristic based on the assumption that small allocations are more likely to all have the same lifespan, whereas big allocations are more likely to have a lifespan that isn't correlated with any other allocations' lifespan. So, big allocations use the system primitive that lets them be deallocated independently from anything else, and small allocations use the primitive that doesn't.
This heuristic is not very reliable. The current generation of malloc implementations, if I remember correctly, has given up altogether on brk and uses mmap for everything. The malloc implementation I suspect you are looking at (the one in the GNU C Library, based on your tags) is very old and mainly continues to be used because nobody is brave enough to take the risk of swapping it out for something newer that will probably but not certainly be better.
brk() is a traditional way of allocating memory in UNIX -- it just expands the data area by a given amount. mmap() allows you to allocate independent regions of memory without being restricted to a single contiguous chunk of virtual address space.
malloc() uses the data space for "small" allocations and mmap() for "big" ones, for a number of reasons, including reducing memory fragmentation. It's just an implementation detail you shouldn't have to worry about.
Please check this question also.
Reducing fragmentation is commonly given as the reason why mmap is used for large allocations; see ryyker’s answer for details. But I think that’s not the real benefit nowadays; in practice there’s still fragmentation even with mmap, just in a larger pool (the virtual address space, rather than the heap).
The big advantage of mmap is discardability.
When allocating memory with sbrk, if the memory is actually used (so that the kernel maps physical memory at some point), and then freed, the kernel itself can’t know about that, unless the allocator also reduces the program break (which it can’t if the freed block isn’t the topmost previously-used block under the program break). The result is that the contents of that physical memory become “precious” as far as the kernel is concerned; if it ever needs to re-purpose that physical memory, it then has to ensure that it doesn’t lose its contents. So it might end up swapping pages out (which is expensive) even though the owning process no longer cares about them.
When allocating memory with mmap, freeing the memory doesn’t just return the block to a pool somewhere; the corresponding virtual memory allocation is returned to the kernel, and that tells the kernel that any corresponding physical memory, dirty or otherwise, is no longer needed. The kernel can then re-purpose that physical memory without worrying about its contents.
the key part of the reason I think, which I copied from the chat said by Peter
free() is a user-space function, not a system call. It either hands them back to the OS with munmap or brk, or keeps them dirty in user-space. If it doesn't make a system call, the OS must preserve the contents of those pages as part of the process state.
So when you use brk to increase your memory adress, when return back, you have to use the brk a negtive value, so brk only can return the most recently memory block you allocated, when you call malloc(huge), malloc(small), free(huge). the huge cannot be returned back to system, you can only maintain a list of fragmentation for this process, so the huge is actually hold by this process. this is the drawback of brk.
but the mmap and munmap can avoid this.
I want to emphasize another view point.
malloc is system function that allocate memory.
You do not really need to debug it, because in some implementations, it might give you memory from static "arena" (e.g. static char array).
In some other implementations it may just return null pointer.
If you want to see what mallow really do, I suggest you look at
http://gee.cs.oswego.edu/dl/html/malloc.html
Linux gcc malloc is based on this.
You can take a look at jemalloc too. It basically uses same brk and mmap, but organizes the data differently and usually is "better".
Happy researching.
To check the program total memory allocated at end of a program, Since i used free() function for deallocating an array .
There is no standard way to know that, and the notion of "full size of memory" is not well defined (and its "allocation" could happen outside and independently of malloc, e.g. on Linux by direct calls to mmap(2) etc...)
In practice (assuming your code is running in a process on some common operating system on a desktop or laptop), think instead in terms of virtual address space.
Read Operating Systems: Three Easy Pieces (freely downloadable).
On Linux (but this is Linux specific) you could use /proc/ (see proc(5) for details) to query the kernel about the virtual address space and the status of some process. For a process of pid 1234, see /proc/1234/maps and /proc/1234/status etc.
You could (and probably should) use valgrind to hunt memory leaks.
With GNU glibc, you also have mallinfo(3) & malloc_stats(3) (but they are non-standard) etc...
Be aware that malloc and free uses lower-level system calls such as mmap(2) & munmap (or the older sbrk(2), etc...) to change the virtual address space, but that free usually don't release memory to the kernel with munmap but prefers to keep and mark the freed memory zone for future usage by malloc.
You could use other implementations of malloc if you really wanted to (or even provide your own one). But you generally should not.
I've read the linux manual about sbrk() thoroughly:
sbrk() changes the location of the program break, which defines the end
of the process's data segment (i.e., the program break is the first
location after the end of the uninitialized data segment).
And I do know that user space memory's organization is like the following:
The problem is:
When I call sbrk(1), why does it say I am increasing the size of heap? As the manual says, I am changing the end position of "data segment & bss". So, what increases should be the size of data segment & bss, right?
The data and bss segments are a fixed size. The space allocated to the process after the end of those segments is therefore not a part of those segments; it is merely contiguous with them. And that space is called the heap space and is used for dynamic memory allocation.
If you want to regard it as 'extending the data/bss segment', that's fine too. It won't make any difference to the behaviour of the program, or the space that's allocated, or anything.
The manual page on Mac OS X indicates you really shouldn't be using them very much:
The brk and sbrk functions are historical curiosities left over from earlier days before the advent of virtual memory management. The brk() function sets the break or lowest address of a process's data segment (uninitialized data) to addr (immediately above bss). Data addressing is restricted between addr and the lowest stack pointer to the stack segment. Memory is allocated by brk in page size pieces; if addr is not evenly divisible by the system page size, it is increased to the next page boundary.
The current value of the program break is reliably returned by sbrk(0) (see also end(3)). The getrlimit(2) system call may be used to determine the maximum permissible size of the data segment; it will not be possible to set the break beyond the rlim_max value returned from a call to getrlimit, e.g. etext + rlp->rlim_max (see end(3) for the definition of etext).
It is mildly exasperating that I can't find a manual page for end(3), despite the pointers to look at it. Even this (slightly old) manual page for sbrk() does not have a link for it.
Notice that today sbrk(2) is rarely used. Most malloc implementations are using mmap(2) -at least for large allocations- to acquire a memory segment (and munmap to release it). Quite often, free simply marks a memory zone to be reusable by some future malloc (and does not release any memory to the Linux kernel).
(so practically, the heap of a modern linux process is made of several segments, so is more subtle than your picture; and multi-threaded processes have one stack per thread)
Use proc(5), notably /proc/self/maps and /proc/$pid/maps, to understand the virtual address space of some process. Try first to understand the output of cat /proc/self/maps (showing the address space of that cat command) and of cat /proc/$$/maps (showing the address space of your shell). Try also to look at the maps pseudo-file for your web browser (e.g. cat /proc/$(pidof firefox)/maps or cat /proc/$(pidof iceweasel)/maps etc...); I have more than a thousand lines (so process segments) in it.
Use strace(1) to understand the system calls done by a given command or process.
Take advantage that on Linux most (and probably all) C standard library implementations are free software, so you can study their source code. The source code of musl-libc is quite easy to read.
Read also about ELF, ASLR, dynamic linking & ld-linux(8), and the Advanced Linux Programming book then syscalls(2)
I have two related questions hence I am asking them in this single thread.
Q1) How can I confirm if my OS is clearing un-"free"'ed memory (allocated using malloc) automatically when a program terminates? I am using Ubuntu 11.04, 32-bit with gcc-4.5.2
As per a tutorial page by Steven Summit here, "Freeing unused memory (malloc'ed) is a good idea, but it's not mandatory. When your program exits, any memory which it has allocated but not freed should be automatically released. If your computer were to somehow ``lose'' memory just because your program forgot to free it, that would indicate a problem or deficiency in your operating system."
Q2) Suppose, foo.c mallocs a B-bytes memory. Later on, foo.c frees this B-bytes memory locations and returns it to the OS. Now my question is, can those PARTICULAR B-bytes of memory locations be re-allocated to foo.c (by the OS) in the current instance OR those B-bytes can't be allocated to foo.c untill its current instance terminates ?
EDIT : I would recommend everyone who reads my question to read the answer to a similar question here and here. Both answers explain the interaction and working of malloc() and free() in good detail without using very esoteric terms. To understand the DIFFERENCE between memory-management tools used by kernel (e.g. brk(), mmap()) and those used by the C-compiler (e.g. malloc(), free()), this is a MUST READ.
When a process ends either thru a terminating signal, e.g. SIGSEGV, or thru the _exit(2) system call (which happens to be called also when returning from main), all the process resources are released by the kernel. In particular, the process address space, including heap memory (allocated with mmap(2) (or perhaps sbrk(2)) syscall (used by malloc library function) is released.
Of course, the free library function either (often) makes the freed memory zone reusable by further future calls to malloc or (occasionally, for large memory zones) release some big memory chunk to the kernel using e.g. munmap(2) system call.
To know more about the memory map of process 1234, read sequentially the /proc/1234/maps pseudo-file (or /proc/self/maps from inside the process). The /proc file system is the preferred way to query the kernel about processes. (there is also /proc/self/statm and /proc/self/smaps and many other interesting things).
The detailed behavior of free and malloc is implementation dependent. You should view malloc as a way to get heap memory, and free as a way to say that a previously malloc-ed zone is useless, and the system (i.e. standard C library + kernel) can do whatever it wants with it.
Use valgrind to hunt memory leak bugs. You could also consider using Boehm's conservative garbage collector, i.e. use GC_malloc instead of malloc and don't bother about freeing manually memory.
Most Modern OS will reclaim the allocated memory so you need not worry about that.
The OS doesn't understand if your application/program leaked memory or not it simply reclaims what it allocated to an process once the process completes.
Yes freed memory can be reused(if needed) & the reuse can happen in the same instantiation.
Q1. You just have to assume that the operating system is behaving correctly.
Q2. There is no reason why the bytes can't be reallocated to foo.c it just depends on how the memory allocation routines work.
Q1) I'm not sure about how you can confirm. However, about the second paragraph, it's considered good style to always free whatever memory you allocate. A good explanation of this is found here: What REALLY happens when you don't free after malloc?.
Q2) Definitely; those bytes are usually the first to be reallocated (depending on the malloc implementation). For a great explanation, see: How do malloc() and free() work?.
i need to have reliable measurement of allocated memory in a linux process. I've been looking into mallinfo but i've read that it is deprecated. What is the state of the art alternative for this sort of statistics?
basically i'm interested in at least two numbers:
number (and size) of allocated memory blocks/pages from the kernel by any malloc or whatever implementation uses the C library of choice
(optional but still important) number of allocated memory by userspace code (via malloc, new, etc.) minus the deallocated memory by it (via free, delete, etc.)
one possibility i have is to override malloc calls with LD_PRELOAD, but it might introduce an unwanted overhead at runtime, also it might not interact properly with other libraries i'm using that also rely on LD_PRELOAD aop-ness.
another possibility i've read is with rusage.
To be clear, this is NOT for debugging purposes, the memory usage is intrinsic feature of the application (similar to Mathematica or Matlab that display the amount of memory used, only that more precise at the block-level)
For this purpose - a "memory usage" introspection feature within an application - the most appropriate interface is malloc_hook(3). These are GNU extensions that allow you to hook every malloc(), realloc() and free() call, maintaining your statistics.
To see how much memory is mapped by your application from the kernel's point of view, you can read and collate the information in the /proc/self/smaps pseudofile. This also lets you see how much of each allocation is resident, swapped, shared/private, clean/dirty etc.
/proc/PID/status contains a few useful pieces of information (try running cat /proc/$$/status for example).
VmPeak is the largest your process's virtual memory space ever became during its execution. This includes all pages mapped into your process, including executable pages, mmap'ed files, stack, and heap.
VmSize is the current size of your process's virtual memory space.
VmRSS is the Resident Set Size of your process; i.e., how much of it is taking up physical RAM right now. (A typical process will have lots of stuff mapped that it never uses, like most of the C library. If no processes need a page, eventually it will be evicted and become non-resident. RSS measures the pages that remain resident and are mapped into your process.)
VmHWM is the High Water Mark of VmRSS; i.e. the highest that number has been during the lifetime of the process.
VmData is the size of your process's "data" segment; i.e., roughly its heap usage. Note that small blocks on which you have done malloc and then free will still be in use from the kernel's point of view; large blocks will actually be returned to the kernel when freed. (If memory serves, "large" means greater than 128k for current glibc.) This is probably the closest to what you are looking for.
These measurements are probably better than trying to track malloc and free, since they indicate what is "really going on" from a system-wide point of view. Just because you have called free() on some memory, that does not mean it has been returned to the system for other processes to use.