Does malloc() use brk() or mmap()? - c

c code:
// program break mechanism
// TLPI exercise 7-1
#include <stdio.h>
#include <stdlib.h>
void program_break_test() {
printf("%10p\n", sbrk(0));
char *bl = malloc(1024 * 1024);
printf("%x\n", sbrk(0));
free(bl);
printf("%x\n", sbrk(0));
}
int main(int argc, char **argv) {
program_break_test();
return 0;
}
When compiling following code:
printf("%10p\n", sbrk(0));
I get warning tip:
format ‘%p’ expects argument of type ‘void *’, but argument 2 has type ‘int’
Question 1: Why is that?
And after I malloc(1024 * 1024), it seems the program break didn't change.
Here is the output:
9b12000
9b12000
9b12000
Question 2: Does the process allocate memory on heap when start for future use? Or the compiler change the time point to allocate? Otherwise, why?
[update] Summary: brk() or mmap()
After reviewing TLPI and check man page (with help from author of TLPI), now I understand how malloc() decide to use brk() or mmap(), as following:
mallopt() could set parameters to control behavior of malloc(), and there is a parameter named M_MMAP_THRESHOLD, in general:
If requested memory is less than it, brk() will be used;
If requested memory is larger than or equals to it, mmap() will be used;
The default value of the parameter is 128kb (on my system), but in my testing program I used 1Mb, so mmap() was chosen, when I changed requested memory to 32kb, I saw brk() would be used.
The book mentioned that in TLPI page 147 and 1035, but I didn't read carefully of that part.
Detailed info of the parameter could be found in man page for mallopt().

If we change the program to see where the malloc'd memory is:
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
void program_break_test() {
printf("%10p\n", sbrk(0));
char *bl = malloc(1024 * 1024);
printf("%10p\n", sbrk(0));
printf("malloc'd at: %10p\n", bl);
free(bl);
printf("%10p\n", sbrk(0));
}
int main(int argc, char **argv) {
program_break_test();
return 0;
}
It's perhaps a bit clearer that sbrk wouldn't change. The memory given to us by malloc is being mapped into a wildly different location.
You could also use strace on Linux to see what system calls are made, and find out that malloc is using mmap to perform the allocation.

malloc is not limited to using sbrk to allocate memory. It might, for example, use mmap to map a large MAP_ANONYMOUS block of memory; normally mmap will assign a virtual address well away from the data segment.
There are other possibilities, too. In particular, malloc, being a core part of the standard library, is not itself limited to standard library functions; it can make use of operating-system-specific interfaces.

If you use malloc in your code, it will call brk() at the beginning, allocated 0x21000 bytes from the heap, that's the address you printed, so the Question 1: the following mallocs requirements can be meet from the pre-allocated space, so these mallocs actually didn't call brk, it is a optimization in malloc. If next time you want to malloc size beyond that boundary, a new brk will be called (if not large than the mmap threshold).

Related

Memcpy with function pointers leads to a segfault

I know I can just copy the function by reference, but I want to understand what's going on in the following code that produces a segfault.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int return0()
{
return 0;
}
int main()
{
int (*r0c)(void) = malloc(100);
memcpy(r0c, return0, 100);
printf("Address of r0c is: %x\n", r0c);
printf("copied is: %d\n", (*r0c)());
return 0;
}
Here's my mental model of what I thought should work.
The process owns the memory allocated to r0c. We are copying the data from the data segment corresponding to return0, and the copy is successful.
I thought that dereferencing a function pointer is the same as calling the data segment that the function pointer points to. If that's the case, then the instruction pointer should move to the data segment corresponding to r0c, which will contain the instructions for function return0. The binary code corresponding to return0 doesn't contain any jumps or function calls that would depend on the address of return0, so it should just return 0 and restore ip... 100 bytes is certainly enough for the function pointer, and 0xc3 is well within the bounds of r0c (it is at byte 11).
So why the segmentation fault? Is this a misunderstanding of the semantics of C's function pointers or is there some security feature that prevents self-modifying code that I'm unaware of?
The memory pages used by malloc to allocate memory are not marked as executable. You can't copy code to the heap and expect it to run.
If you want to do something like that you have to go deeper into the operating system, and allocate pages yourself. Then you need to mark those as executable. You would most likely need administrator rights to be able to set the executable flag on memory pages.
And it's really dangerous. If you do this in a program you distribute and have some kind of bug that lets an attacker use our program to write to those allocated memory pages, then the attacker can gain administrator rights and take control of the computer.
There's also other problems with your code, like pointers to functions might not translate well into general pointers on all platforms. It's very hard (not to mention non-standard) to predict or otherwise get the size of a function. You also print out pointers wrong in your code example. (use the "%p" format to print a void *, casting the pointer to a void * is needed).
Also when you declare a function like int fun() that's not the same as declaring a function that takes no arguments. If you want to declare a function that takes no arguments you should explicitly use void as in int fun(void).
The standard says:
The memcpy function copies n characters from the object pointed to by s2 into the object pointed to by s1.
[C2011, 7.24.2.1/2; emphasis added]
In the standard's terminology, functions are not "objects". The standard does not define behavior for the case where the source pointer points to a function, therefore such a memcpy() call produces undefined behavior.
Additionally, the pointer returned by malloc() is an object pointer. C does not provide for direct conversion of object pointers to function pointers, and it does not provide for objects to be called as functions. It is possible to convert between object pointer and function pointer by means of an intermediate integer value, but the effect of doing so is at minimum doubly implementation-defined. Under some circumstances it is undefined.
As in other cases, UB can turn out to be precisely the behavior you hoped for, but it is not safe to rely on that. In this particular case, other answers present good reasons to not expect to get the behavior you hoped for.
As was said in some comments, you need to make the data executable. This requires communicating with the OS to change protections on the data. On Linux, this is the system call int mprotect(void* addr, size_t len, int prot) (see http://man7.org/linux/man-pages/man2/mprotect.2.html).
Here is a Windows solution using VirtualProtect.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#ifdef _WIN32
#include <Windows.h>
#endif
int return0()
{
return 0;
}
int main()
{
int (*r0c)(void) = malloc(100);
memcpy((void*) r0c, (void*) return0, 100);
printf("Address of r0c is: %p\n", (void*) r0c);
#ifdef _WIN32
long unsigned int out_protect;
if(!VirtualProtect((void*) r0c, 100, PAGE_EXECUTE_READWRITE, &out_protect)){
puts("Failed to mark r0c as executable");
exit(1);
}
#endif
printf("copied is: %d\n", (*r0c)());
return 0;
}
And it works.
Malloc returns a pointer to an allocated memory (100 bytes in your case). This memory area is uninitialized; assuming that memory could be executed by the CPU, for your code to work, you would have to fill those 100 bytes with the executable instructions that the function implements (if indeed it can be held in 100 bytes). But as has been pointed out, your allocation is on the heap, not in the text (program) segment and I don't think it can be executed as instructions. Perhaps this would achieve what it is you want:
int return0()
{
return 0;
}
typedef int (*r0c)(void);
int main(void)
{
r0c pf = return0;
printf("Address of r0c is: %x\n", pf);
printf("copied is: %d\n", pf());
return 0;
}

malloc() 5GB memory on a 32 bit machine

I was reading in a book:
The virtual address space of a process on a 32 bit machine is 2^32 i.e. 4Gb of space. And every address seen in the program is a virtual address. The 4GB of space is further goes through user/kernel split 3-1GB.
To better understand this, I did malloc() of 5Gb space and tried to print the all addresses. If I print the addresses, How is the application going to print whole 5Gb address when It has only 3GB of virtual address space? Am I missing something here?
malloc() takes size_t as an argument. On 32 bit system it's an alias to some unsigned 32 bit integer type. This means that you just cannot pass any value bigger than 2^32-1 as an argument for malloc() making it impossible request allocation of more than 4GB of memory using this function.
The same is true for all other functions that can be used to allocate memory. Ultimately they all end up as either brk() or mmap syscall. The length argument of mmap() is also of type ssize_t an in case of brk() you have to provide a pointer for the new end of your allocated space. The pointer is again 32 bit.
So there is absolutely no way to tell kernel you would like to get more than 4GB of memory allocated with one call) And it's not an accident - this just wouldn't make any sense anyway.
Now it's true that you could do several calls to malloc or other function that allocates memory, requesting more than 4GB in total. If you try this, the subsequent call (that would cause extending allocated memory to more than 3GB) will fail as there is just no address space available.
So I guess that you either didn't check the malloc return value or you did try to run code like this (or something similar):
int main() {
assert(malloc(5*1<<30));
}
and assumed that you succeeded in allocating 5GB without verifying that your argument overflowed and instead of requesting 5368709120 bytes, you requested 1073741824. One example to verify this on Linux is to use:
$ ltrace ./a.out
__libc_start_main(0x804844c, 1, 0xbfbcea74, 0x80484a0, 0x8048490 <unfinished ...>
malloc(1073741824) = 0x77746008
$
There's already a good answer. Just in case, the size of your virtual address space is easily verifiable like this:
#include <stdlib.h>
#include <stdio.h>
int main()
{
size_t size = (size_t)-1L;
void *foo;
printf("trying to allocate %zu bytes\n", size);
if (!(foo = malloc(size)))
{
perror("malloc()");
}
else
{
free(foo);
}
}
> gcc -m32 -omalloc malloc.c && ./malloc
trying to allocate 4294967295 bytes
malloc(): Cannot allocate memory
This must fail because parts of your address space are already occupied: by the mapped part of the kernel, by mapped shared libraries and by your program, of course.
You cannot do this because there is no function for you to alloc 5GB memory.

Confusion after counting maximum allocation that can be done by malloc()

I have the following piece of code that runs in C on Ubuntu. It counts how many Gigabytes that can be allocated by the OS via malloc().
#include <sys/types.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
int main(){
int count = 0;
char* a;
while (1){
a = (char*)malloc(1024*1024*1024);
if (a==NULL) break;
count++;
}
printf("%d\n", count);
return 0;
}
Surprisingly, when running on my machine, it prints more than 100,000. I feel it is so unreasonable. My RAM is 8GB, my hard disk is about 500 GB, where does this 100,000 come from?
This is memory overcommit:
[...]Under the default memory management strategy, malloc() essentially always succeeds, with the kenrel assuming you're not really going to use all of the memory you just asked for. The malloc()'s will continue to succeed, but not until you actually try to use the memory you allocated will the kernel 'really' allocate it. [...]
If we look at a Linux man page for malloc it says (emphasis mine):
By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available. In case it turns out that the system is out of memory, one or more processes will be killed by the OOM killer.
and:
For more information, see the description of /proc/sys/vm/overcommit_memory and /proc/sys/vm/oom_adj in proc(5), and the Linux kernel source file Documentation/vm/overcommit-accounting.
The operating system allows you to allocate much more memory than it has available. If you actually try to use the memory, it will run out.
If you want to see it run out, try, e.g., doing a memset on the memory after each successful malloc.

How much memory calloc and malloc can allocate?

How much memory calloc and malloc can allocate?
As malloc and calloc can allocate memory dynamically
Example
void *malloc (size_in_bytes);
And calloc can allocate memory depending on the number of blocks
Example
void *calloc (number_of_blocks, size_of_each_block_in_bytes);
You can allocate as much bytes as type size_t has different values. So in 32-bit application it is 4GB in 64-bit 16 I don't even know how to call that size
All in all you can allocate all memory of machine.
Aside from being limited by the amount RAM in the PC, it is system dependent, but on Windows, it's _HEAP_MAXREQ according to the MSDN article on malloc. Note though malloc and calloc are not guaranteed to allocate anything. It all depends on how much memory is available on the executing PC.
malloc sets errno to ENOMEM if a memory allocation fails or if the amount of memory requested exceeds _HEAP_MAXREQ.
_HEAP_MAXREQ is defined as follows in malloc.h (at least in the Visual Studio 2010 includes).
#ifdef _WIN64
#define _HEAP_MAXREQ 0xFFFFFFFFFFFFFFE0
#else
#define _HEAP_MAXREQ 0xFFFFFFE0
#endif
You shouldn't really worry about this though. When using malloc you should decide how much memory you really need, and call it with that as the request. If the system cannot provide it, malloc will return NULL. After you make your call to malloc you should always check to see that it's not NULL. Here is the C example from the MSDN article on proper usage. Also note that once you've finished with the memory, you need to call free.
#include <stdlib.h> // For _MAX_PATH definition
#include <stdio.h>
#include <malloc.h>
int main( void )
{
char *string;
// Allocate space for a path name
string = malloc( _MAX_PATH );
// In a C++ file, explicitly cast malloc's return. For example,
// string = (char *)malloc( _MAX_PATH );
if( string == NULL )
printf( "Insufficient memory available\n" );
else
{
printf( "Memory space allocated for path name\n" );
free( string );
printf( "Memory freed\n" );
}
}
As far as the language definition is concerned, the only upper limit per call is what size_t will support (e.g., if the max size_t value is 232-1, then that's the largest number of bytes malloc can allocate for a single block).
Whether you have the resources available for such a call to succeed depends on the implementation and the underyling system.
On linux, indefinite amounts, i. e. much more than you have either physical or virtual memory.
As long, as you don't actually use it, you're fine, it's just when you actually use more memory than available that the out-of-memory killer starts running amok, and shoots down your process.
This is the reason why many people don't bother checking the result of malloc() anymore, because it's not null even when the returned buffer can never be backed...
1) It depends on the resource limits of the user.
2) It also depends on the availability of address space.

Heap size limitation in C

I have a doubt regarding heap in program execution layout diagram of a C program.
I know that all the dynamically allocated memory is allotted in heap which grows dynamically. But I would like to know what is the max heap size for a C program ??
I am just attaching a sample C program ... here I am trying to allocate 1GB memory to string and even doing the memset ...
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
char *temp;
mybuffer=malloc(1024*1024*1024*1);
temp = memset(mybuffer,0,(1024*1024*1024*1));
if( (mybuffer == temp) && (mybuffer != NULL))
printf("%x - %x\n", mybuffer, &mybuffer[((1024*1024*1024*1)-1)]]);
else
printf("Wrong\n");
sleep(20);
free(mybuffer);
return 0;
}
If I run above program in 3 instances at once then malloc should fail atleast in one instance [I feel so] ... but still malloc is successfull.
If it is successful can I know how the OS takes care of 3GB of dynamically allocated memory.
Your machine is very probably overcomitting on RAM, and not using the memory until you actually write it. Try writing to each block after allocating it, thus forcing the operating system to ensure there's real RAM mapped to the address malloc() returned.
From the linux malloc page,
BUGS
By default, Linux follows an optimistic memory allocation strategy.
This means that when malloc() returns non-NULL there is no guarantee
that the memory really is available. This is a really bad bug. In
case it turns out that the system is out of memory, one or more pro‐
cesses will be killed by the infamous OOM killer. In case Linux is
employed under circumstances where it would be less desirable to sud‐
denly lose some randomly picked processes, and moreover the kernel ver‐
sion is sufficiently recent, one can switch off this overcommitting
behavior using a command like:
# echo 2 > /proc/sys/vm/overcommit_memory
See also the kernel Documentation directory, files vm/overcommit-
accounting and sysctl/vm.txt.
You're mixing up physical memory and virtual memory.
http://apollo.lsc.vsc.edu/metadmin/references/sag/x1752.html
http://en.wikipedia.org/wiki/Virtual_memory
http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory
Malloc will allocate the memory but it does not write to any of it. So if the virtual memory is available then it will succeed. It is only when you write something to it will the real memory need to be paged to the page file.
Calloc if memory serves be correctly(!) write zeros to each byte of the allocated memory before returning so will need to allocate the pages there and then.

Resources