What is the use of sbrk system call in Unix and how is it to be used in C? Most of them say that malloc uses the sbrk system call to allocate memory. So, how does the sbrk system call allocate memory? What are the arguments and return type of sbrk?
sbrk() takes a positive integer as an argument. It increases the size of the current process's heap by that number of bytes, then returns a void * to the new end of the heap.
However, the information you're reading is outdated. To quote the Mac OS X manual page for this system call:
The brk and sbrk functions are historical curiosities left over from earlier days before the advent of virtual memory management.
Modern systems do not use sbrk() for memory management, as it is incredibly limited. In particular, it cannot manage a heap which is not contiguous in memory! Instead, they use the mmap() system call to map new ranges of memory as needed.
Function definition: void *sbrk(intptr_t increment);
sbrk() increments the program's data space by increment bytes. Calling sbrk() with an increment of 0 can be used to find the current location of the program break.
From http://linux.die.net/man/2/sbrk
Related
As far as I know, C has the following functions, e.g: malloc, calloc, realloc, to allocate memory. And the linux kernel also has the following functions, e.g: malloc, mmap, kmalloc, vmalloc... to allocate memory.
I want to know which is the lowest function.
If you say, "Linux kernel is the lowest function, your C program must allocate memory with Linux kernel", then how does the Linux kernel allocate it's own memory?
Or if you say, "Linux kernel is the lowest function.", then when I write a C program and run in the Linux system, to allocate memory, I should through the system call.
Hope to have an answer.
I want to know which is the lowest function.
The user-level malloc function calls brk or malloc (depending on the library used and depending on the Linux version).
... how does the Linux kernel allocate it's own memory?
On a system without MMU this is easy:
Let's say we have system with 8 MB RAM and we know that the address of the RAM is 0x20000000 to 0x20800000.
The kernel contains an array that contains information about which pages are in use. Let's say the size of a "page" is 0x1000 (this is the page size in x86 systems with MMU).
In old Linux versions the array was named mem_map. Each element in the array corresponds to one memory "page". It is zero if the page is free.
When the system is started, the kernel itself initializes this array (writes the initial values in the array).
If the kernel needs one page of memory, it searches for an element in the array whose value is 0. Let's say mem_map[0x123] is 0. The kernel now sets mem_map[0x123]=1;. mem_map[0x123] corresponds to the address 0x20123000, so the kernel "allocated" some memory at address 0x20123000.
If the kernel wants to "free" some memory at address 0x20234000, it simply sets mem_map[0x234]=0;.
On a system with MMU, it is a bit more complicated but the principle is the same.
On the linux OS, the C functions malloc, calloc, realloc used by user mode programs are implemented in the C library and handle pages of memory mapped in the process address space using the mmap system call. mmap associates pages of virtual memory with addresses in the process address space. When the process then accesses these addresses, actual RAM is mapped by the kernel to this virtual space. Not every call to malloc maps memory pages, only those for which not enough space was already requested from the system.
In the kernel space, a similar process takes place but the caller can require that the RAM be mapped immediately.
I new to this can any one tell the exact difference between brk and sbrk with a brief example ? is there any efficiency factor to choose from any of the two?
malloc and new internally call brk or sbrk.
int brk(void *addr);
brk() sets the end of the data segment to the value specified by
addr, when that value is reasonable, the system has enough memory,
and the process does not exceed its maximum data size.
On success, brk() returns zero. On error, -1 is returned, and errno
is set to ENOMEM.
void *sbrk(intptr_t increment);
sbrk() increments the program's data space by increment bytes.
Calling sbrk() with an increment of 0 can be used to find the current
location of the program break.
On success, sbrk() returns the previous program break. (If the break
was increased, then this value is a pointer to the start of the newly
allocated memory). On error, (void *) -1 is returned, and errno is
set to ENOMEM.
From linux manual page
brk sets the upper limit of the data segment, sbrk increments it. In ancient Unixes malloc/free used sbrk. On modern ones things could be very different, for example, OSX does not use brk/sbrk to manage heap allocations but mmap, brk/sbrk exist but are just emulation in a small segment of memory. This is almost the same on Linux (source code of mention the history of transition from brk/sbrk to mmap).
In linux, is calloc exactly the same as malloc + memset or does this depend on the exact linux/kernel version?
I am particularly interested in the question of whether you can calloc more RAM than you physically have (as you can certainly malloc more RAM than you physically have, you just can't write to it). In other words, does calloc always actually write to the memory you have been allocated as the specs suggest it should.
Of course, that depends on the implementation, but on a modern day Linux, you probably can. Easiest way is to try it, but I'm saying this based on the following logic.
You can malloc more than the memory you have (physical + virtual) because the kernel delays allocation of your memory until you actually use it. I believe that's to increase the chances of your program not failing due to memory limits, but that's not the question.
calloc is the same as malloc but zero initializes the memory. When you ask Linux for a page of memory, Linux already zero-initializes it. So if calloc can tell that the memory it asked for was just requested from the kernel, it doesn't actually have to zero initialize it! Since it doesn't, there is no access to that memory and therefore it should be able to request more memory than there actually is.
As mentioned in the comments this answer provides a very good explanation.
Whether calloc needs to write to the memory depends on whether it got the allocation from heap pages that are already assigned to the process, or it had to request more memory be assigned to the process by the kernel (using a system call such as sbrk() or mmap()). When the kernel assigns new memory to a process, it always zeroes it first (typically using a VM optimization, so it doesn't actually have to write to the page). But if it's reusing memory that was assigned previously, it has to use memset() to zero it.
It is not mentioned in the cited duplicate or here. Linux uses virtual memory and can allocate more memory that physically available in the system. A naive implementation of calloc() that simply does a malloc() plus memset() in user space will touch every page.
As Linux typically allocates in 4k chunks, all of the calloc() blocks are the same and initially read as zero. That is the same 4k chunk of memory can be mapped read only and the entire calloc() space in only taking up approximately size/4k * pointer_size + 4k. As the program writes to the calloc() space, a page fault happens and Linux will allocate a new page (4k) and resume the program.
This is called copy-on-write or COW for short. malloc() will generally behave the same way. For small sizes, the 'C' library will use binning and share 4k pages with other small sized allocation.
So, there are typically two layers involved.
Linux kernel's process memory management.
glibc heap management.
If the memory size requested is large and requires new memory allocated to the process, then most of the above applies (via Linux's process memory management). However, if the memory requested is small, then it will be like a malloc() plus memset(). In the large allocation size, the memset() is damaging as it touches the memory and the kernel thinks it needs a new page to allocate.
You can't malloc(3) more ram than the kernel gives the process doing the malloc(3)-ing. malloc(3) returns NULL if you can't allocate the amount of memory you want to allocate. In addition, malloc(3) and memset(3) are defined by your c library (libc.so) and not your kernel. The Linux kernel defines mmap(2) and other low-level memory allocation functions, not the *alloc(3) family (excluding kalloc()).
I am writing a C program on linux and I wonder:
How to limit the total memory my c program consumes?
If I set a memory limit to my c program, say 32M, what happens if it requires much more memory than 32M?
You should use the setrlimit system call, with the RLIMIT_DATA and RLIMIT_STACK resources to limit the heap and stack sizes respectively. It is tempting to use RLIMIT_AS or RLIMIT_RSS but you will discover that they do not work reliably on many old Linux kernels, and I see no sign on the kernel mailing list that the problems have been resolved in the latest kernels. One problem relates to how mmap'd memory is counted or rather not counted toward the limit totals. Since glibc malloc uses mmap for large allocations, even programs that don't call mmap directly can exceed the limits.
If you exceed the RLIMIT_STACK limit (call stack too deep, or allocate too many variables on the stack), your program will receive a SIGSEGV. If you try to extend the data segment past the RLIMIT_DATA limit (brk, sbrk or some intermediary like malloc), the attempt will fail. brk or sbrk will return < 0 and malloc will return a null pointer.
See ulimit command in your system.
From the man page:
-v The maximum amount of virtual memory available to the process
If your program is well-written it should take of the cases where a dynamic memory allocation fails: check the return value of malloc, calloc and realloc functions.
On Linux, within your C program, use setrlimit() to set your program's execution environment limits. When you run out of memory, for instance, then calls to malloc() will return NULL etc.
#include <sys/resource.h>
{ struct rlimit rl = {32000, 32000}; setrlimit(RLIMIT_AS, &rl); }
int brk(void *end_data_segment);
void *sbrk(intptr_t increment);
Calling sbrk() with an increment of
0
can be used to find the current location of the program break.
What is program break? Where does it start from,0x00?
Oversimplifying:
A process has several segments of memory:
Code (text) segment, which contains the code to be executed.
Data segment, which contains data the compiler knows about (globals and statics).
Stack segment, which contains (drumroll...) the stack.
(Of course, nowadays it's much more complex. There is a rodata segment, a uninitialized data segment, mappings allocated via mmap, a vdso, ...)
One traditional way a program can request more memory in a Unix-like OS is to increment the size of the data segment, and use a memory allocator (i.e. malloc() implementation) to manage the resulting space. This is done via the brk() system call, which changes the point where the data segment "breaks"/ends.
A program break is end of the process's data segment. AKA...
the program break is the first
location after the end of the
uninitialized data segment
As to where it starts from, it's system dependent but probably not 0x00.
These days, sbrk(2) (and brk) are nearly obsolete system calls (and you can nearly forget about them and ignore the old notion of break; focus on understanding mmap(2)). Notice that the sbrk(2) man page says in its NOTES :
Avoid using brk() and sbrk(): the malloc(3) memory allocation package
is the portable and comfortable way of allocating memory.
(emphasis mine)
Most implementations of malloc(3) (notably the one in musl-libc) are rather using mmap(2) to require memory - and increase their virtual address space - from the kernel (look at that virtual address space wikipage, it has a nice picture). Some malloc-s use sbrk for small allocations, mmap for large ones.
Use strace(1) to find out the system calls (listed in syscalls(2)) done by some given process or command. BTW you'll then find that bash and ls (and probably many other programs) don't make a single call to sbrk.
Explore the virtual address space of some process by using proc(5). Try cat /proc/$$/maps and cat /proc/self/maps and even cat /proc/$$/smaps and read a bit to understand the output.
Be aware of ASLR & vdso(7).
And sbrk is not very thread friendly.
(my answer focuses on Linux)
You are saying that sbrk() is an obsolute system call and that we should use malloc(), but malloc(), according to her documentation, when allocating less memory than 128 KiB (32 pages) uses it. So we shouldn´t use sbrk() directly, but malloc() use it, if allocation is bigger than 128 KiB then malloc() uses mmap() that allocates private pages to the userspace.
Finally its a good idea to understand sbrk(), at least for understanding the "Program Break" concept.
Based on the following widely used diagram:
program break, which is also known as brk in many articles, points to the address of heap segment's end.
When you call malloc, it changes the address of program break.