The following program is killed by the kernel when the memory is ran out. I would like to know when the global variable should be assigned to "ENOMEM".
#define MEGABYTE 1024*1024
#define TRUE 1
int main(int argc, char *argv[]){
void *myblock = NULL;
int count = 0;
while(TRUE)
{
myblock = (void *) malloc(MEGABYTE);
if (!myblock) break;
memset(myblock,1, MEGABYTE);
printf("Currently allocating %d MB\n",++count);
}
exit(0);
}
First, fix your kernel not to overcommit:
echo "2" > /proc/sys/vm/overcommit_memory
Now malloc should behave properly.
As "R" hinted, the problem is the default behaviour of Linux memory management, which is "overcommiting". This means that the kernel claims to allocate you memory successfuly, but doesn't actually allocate the memory until later when you try to access it. If the kernel finds out that it's allocated too much memory, it kills a process with "the OOM (Out Of Memory) killer" to free up some memory. The way it picks the process to kill is complicated, but if you have just allocated most of the memory in the system, it's probably going to be your process that gets the bullet.
If you think this sounds crazy, some people would agree with you.
To get it to behave as you expect, as R said:
echo "2" > /proc/sys/vm/overcommit_memory
It happens when you try to allocate too much memory at once.
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
int main(int argc, char *argv[])
{
void *p;
p = malloc(1024L * 1024 * 1024 * 1024);
if(p == NULL)
{
printf("%d\n", errno);
perror("malloc");
}
}
In your case the OOM killer is getting to the process first.
I think errno will be set to ENOMEM:
Macro defined in stdio.h. Here is the documentation.
#define ENOMEM 12 /* Out of Memory */
After you call malloc in this statement:
myblock = (void *) malloc(MEGABYTE);
And the function returns NULL -because system is out of memory -.
I found this SO question very interesting.
Hope it helps!
Related
In my C program, based on the user's input, memory will be allocated for a given simulation. The initial problem I faced is that user can ask for a huge number to allocate but malloc() never fails until it runs out of memory then the program crashes.
I investigated the logic behind this and it now makes sense to me, see [1][2]. A possible workaround given here "SIGKILL while allocating memory in C++" suggests to set overcommit_memory in in /proc/sys/vm/overcommit_memory from 0 to 2.
This solved the problem from one side. But since I am using -fsanitize=address I get error from sanitizer.
Is there any better solution to this?
I guess the clang AddressSanitizer is failing because there is a legit leak. So my answer ignores that:
Alternatives:
Disable the overcommit behaviour, as you have already figured out: that is going to affect other processes and requires root.
run you app in a docker image with the oom killer disabled: that doesn't affect other processes but requires root to install docker (this is my favourite solution though).
write after malloc: may take long to alloc a huge chunk of memory and your process can still get killed because of other running process but doesn't require root.
use ulimit -v to limit the amount of memory depending on the machine: that also doesn't require root but your process might be killed anyway.
Code for the third alternative (for linux):
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <setjmp.h>
jmp_buf resume_malloc;
void handle_malloc_error(int sig)
{
longjmp(resume_malloc, sig);
}
void *memalloc(size_t sz) {
void *p = 0;
int sig = setjmp(resume_malloc);
if ( sig == 0 ) {
p = malloc(sz);
signal(SIGSEGV, &handle_malloc_error);
memset(p, 0, sz);
} else {
p = 0;
}
signal(SIGSEGV, SIG_DFL);
return p;
}
int main(int argc, char *argv[])
{
size_t sz = 160L * 1024 * 1024 * 1024L;
void *p;
for (int i=0; i < 100; i++) {
printf("size: %lu\n", sz);
p = memalloc(sz);
if ( p == 0 ) {
printf("out of memory\n");
break;
}
sz *= 2;
}
}
I have written a program using clone() system call having CLONE_VM and CLONE_FILES set.
I am not able to understand why the output is showing Segmentation Fault. Can somebody please correct my code and tell me the reason for the same.
#include<stdio.h>
#include<unistd.h>
#include<fcntl.h>
#include<sys/types.h>
#include<sys/stat.h>
#include<sched.h>
#include<stdlib.h>
int variable, fd;
int do_something() {
// sleep(100);
variable = 42;
close(fd);
_exit(0);
}
int main(int argc, char *argv[]) {
void **child_stack;
char tempch;
variable = 9;
fd = open("test.file", O_RDONLY);
child_stack = (void **) malloc(16384);
printf("The variable was %d\n", variable);
clone(do_something, child_stack, CLONE_VM|CLONE_FILES, NULL);
// sleep(100);
printf("The variable is now %d\n", variable);
if (read(fd, &tempch, 1) < 1) {
perror("File Read Error");
exit(1);
}
printf("We could read from the file\n");
return 0;
}
You need to know which direction stack grows on your processor, and you need to know which end of the stack you must pass to clone().
From man clone:
Stacks grow downwards on all processors that run Linux (except the
HP PA processors), so child_stack usually points to the topmost
address of the memory space set up for the child stack.
You are not passing the topmost address, you are passing the bottommost address, and you are not (I am guessing) on HP-PA.
Fix:
child_stack = (void **) malloc(16384) + 16384 / sizeof(*child_stack);
P.S. I am astonished by the number of obviously wrong non-answers here.
No, close on invalid file descriptor
does not crash on any UNIX and
Linux system in existence.
No, void* vs. void** has nothing at all to do with the problem.
No, you don't need to take an address of do_something, the compiler will do that automatically for you.
And finally, yes: calling close, _exit, or any other libc routine in the clone()d thread is potentially unsafe, although it does not cause the problem here.
The way to fix is to have the child stack actually on the stack .. i.e.
char child_stack [16384];
I suspect that stack pointer can't point to data segment or sth like that...
And even then.. it works with -g .. but crashes with -O !!!
I can get the address of the end of the heap with sbrk(0), but is there any way to programmatically get the address of the start of the heap, other than by parsing the contents of /proc/self/maps?
I think parsing /proc/self/maps is the only reliable way on the Linux to find the heap segment. And do not forget that some allocators (including one in my SLES) do use for large blocks mmap() thus the memory isn't part of the heap anymore and can be at any random location.
Otherwise, normally ld adds a symbol which marks the end of all segments in elf and the symbol is called _end. E.g.:
extern void *_end;
printf( "%p\n", &_end );
It matches the end of the .bss, traditionally the last segment of elf. After the address, with some alignment, normally follows the heap. Stack(s) and mmap()s (including the shared libraries) are at the higher addresses of the address space.
I'm not sure how portable it is, but apparently it works same way on the Solaris 10. On HP-UX 11 the map looks different and heap appears to be merged with data segment, but allocations do happen after the _end. On AIX, procmap doesn't show heap/data segment at all, but allocations too get the addresses past the _end symbol. So it seems to be at the moment quite portable.
Though, all considered, I'm not sure how useful that is.
P.S. The test program:
#include <stdio.h>
#include <stdlib.h>
char *ppp1 = "hello world";
char ppp0[] = "hello world";
extern void *_end; /* any type would do, only its address is important */
int main()
{
void *p = calloc(10000,1);
printf( "end:%p heap:%p rodata:%p data:%p\n", &_end, p, ppp1, ppp0 );
sleep(10000); /* sleep to give chance to look at the process memory map */
return 0;
}
You may call sbrk(0) to get the start of the heap, but you have to make sure no memory has been allocated yet.
The best way to do this is to assign the return value at the very beginning of main(). Note that many functions do allocate memory under the hood, so a call to sbrk(0) after a printf, a memory utility like mtrace or even a call to putenv will already return an offset value.
Although much of what we can find say that the heap is right next to .bss, I am not sure what is in the difference between end and the first break. Reading there seems to results in a segmentation fault.
The difference between the first break and the first address returned by malloc is, among (probably) other thing:
the head of the memory double-linked-list, including the next free block
a structure prefixed to the malloced block incuding:
the length of this block
the address of the previous free block
the address of the next free block
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
void print_heap_line();
int main(int argc, char const *argv[])
{
char* startbreak = sbrk(0);
printf("pid: %d\n", getpid()); // printf is allocating memory
char* lastbreak = sbrk(0);
printf("heap: [%p - %p]\n", startbreak, lastbreak);
long pagesize = sysconf(_SC_PAGESIZE);
long diff = lastbreak - startbreak;
printf("diff: %ld (%ld pages of %ld bytes)\n", diff, diff/pagesize, pagesize);
print_heap_line();
printf("\n\npress a key to finish...");
getchar(); // gives you a chance to inspect /proc/pid/maps yourself
return 0;
}
void print_heap_line() {
int mapsfd = open("/proc/self/maps", O_RDONLY);
if(mapsfd == -1) {
fprintf(stderr, "open() failed: %s.\n", strerror(errno));
exit(1);
}
char maps[BUFSIZ] = "";
if(read(mapsfd, maps, BUFSIZ) == -1){
fprintf(stderr, "read() failed: %s.\n", strerror(errno));
exit(1);
}
if(close(mapsfd) == -1){
fprintf(stderr, "close() failed: %s.\n", strerror(errno));
exit(1);
}
char* line = strtok(maps, "\n");
while((line = strtok(NULL, "\n")) != NULL) {
if(strstr(line, "heap") != NULL) {
printf("\n\nfrom /proc/self/maps:\n%s\n", line);
return;
}
}
}
pid: 29825
heap: [0x55fe05739000 - 0x55fe0575a000]
diff: 135168 (33 pages of 4096 bytes)
from /proc/self/maps:
55fe05739000-55fe0575a000 rw-p 00000000 00:00 0 [heap]
press a key to finish...
I need ideas on how to write a C program that reserve a specified amount of MB RAM until a key [ex. the any key] is pressed on a Linux 2.6 32 bit system.
*
/.eat_ram.out 200
# If free -m is execute at this time, it should report 200 MB more in the used section, than before running the program.
[Any key is pressed]
# Now all the reserved RAM should be released and the program exits.
*
It is the core functionality of the program [reserving the RAM] i do not know how to do, getting arguments from the commandline, printing [Any key is pressed] and so on is not a problem from me.
Any ideas on how to do this?
You want to use malloc() to do this. Depending on your need, you will also want to:
Write data to the memory so that the kernel actually guarantees it. You can use memset() for this.
Prevent the memory from being paged out (swapped), the mlock() / mlockall() functions can help you with this.
Tell the kernel how you actually intend to use the memory, which is accomplished via posix_madvise() (this is preferable to an explicit mlockall()).
In most realities, malloc() and memset() (or, calloc() which effectively does the same) will suit your needs.
Finally, of course, you want to free() the memory when it is no longer needed.
Can't you just use malloc() to allocate that ram to your process? That will reserve that RAM for you, and then you are free to do whatever you wish with it.
Here's an example for you:
#include <stdlib.h>
int main (int argc, char* argv[]) {
int bytesToAllocate;
char* bytesReserved = NULL;
//assume you have code here that fills bytesToAllocate
bytesReserved = malloc(bytesToAllocate);
if (bytesReserved == NULL) {
//an error occurred while reserving the memory - handle it here
}
//when the program ends:
free(bytesReserved);
return 0;
}
If you want more information, have a look at the man page (man malloc in a linux shell). If you aren't on linux, have a look at the online man page.
calloc() is what you want. It will reserve memory for your process and write zero's to it. This ensures that the memory is actually allocated for your process. If you malloc() a large portion of memory, the OS may be lazy about actually allocating memory for you, only actually allocating it when it is written to (which will never happen in this case).
You will need:
malloc() to allocate however many bytes you need (malloc(200000000) or malloc(20 * (1 << 20))).
getc() to wait for a keypress.
free() to deallocate the memory.
The information on these pages should be helpful.
Did this, should work. Although I was able to reserve more RAM than I have installed, this should work for valid values, tho.
#include <stdio.h>
#include <stdlib.h>
enum
{
MULTIPLICATOR = 1024 * 1024 // 1 MB
};
int
main(int argc, char *argv[])
{
void *reserve;
unsigned int amount;
if (argc < 2)
{
fprintf(stderr, "usage: %s <megabytes>\n", argv[0]);
return EXIT_FAILURE;
}
amount = atoi(argv[1]);
printf("About to reserve %ld MB (%ld Bytes) of RAM...\n", amount, amount * MULTIPLICATOR);
reserve = calloc(amount * MULTIPLICATOR, 1);
if (reserve == NULL)
{
fprintf(stderr, "Couldn't allocate memory\n");
return EXIT_FAILURE;
}
printf("Allocated. Press any key to release the memory.\n");
getchar();
free(reserve);
printf("Deallocated reserved memory\n");
return EXIT_SUCCESS;
}
I would like to write a program to consume all the memory available to understand the outcome. I've heard that linux starts killing the processes once it is unable to allocate the memory.
Can anyone help me with such a program.
I have written the following, but the memory doesn't seem to get exhausted:
#include <stdlib.h>
int main()
{
while(1)
{
malloc(1024*1024);
}
return 0;
}
You should write to the allocated blocks. If you just ask for memory, linux might just hand out a reservation for memory, but nothing will be allocated until the memory is accessed.
int main()
{
while(1)
{
void *m = malloc(1024*1024);
memset(m,0,1024*1024);
}
return 0;
}
You really only need to write 1 byte on every page (4096 bytes on x86 normally) though.
Linux "over commits" memory. This means that physical memory is only given to a process when the process first tries to access it, not when the malloc is first executed. To disable this behavior, do the following (as root):
echo 2 > /proc/sys/vm/overcommit_memory
Then try running your program.
Linux uses, by default, what I like to call "opportunistic allocation". This is based on the observation that a number of real programs allocate more memory than they actually use. Linux uses this to fit a bit more stuff into memory: it only allocates a memory page when it is used, not when it's allocated with malloc (or mmap or sbrk).
You may have more success if you do something like this inside your loop:
memset(malloc(1024*1024L), 'w', 1024*1024L);
In my machine, with an appropriate gb value, the following code used 100% of the memory, and even got memory into the swap.
You can see that you need to write only one byte in each page: memset(m, 0, 1);,
If you change the page size: #define PAGE_SZ (1<<12) to a bigger page size: #define PAGE_SZ (1<<13) then you won't be writing to all the pages you allocated, thus you can see in top that the memory consumption of the program goes down.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define PAGE_SZ (1<<12)
int main() {
int i;
int gb = 2; // memory to consume in GB
for (i = 0; i < ((unsigned long)gb<<30)/PAGE_SZ ; ++i) {
void *m = malloc(PAGE_SZ);
if (!m)
break;
memset(m, 0, 1);
}
printf("allocated %lu MB\n", ((unsigned long)i*PAGE_SZ)>>20);
getchar();
return 0;
}
A little known fact (though it is well documented) - you can (as root) prevent the OOM killer from claiming your process (or any other process) as one of its victims. Here is a snippet from something directly out of my editor, where I am (based on configuration data) locking all allocated memory to avoid being paged out and (optionally) telling the OOM killer not to bother me:
static int set_priority(nex_payload_t *p)
{
struct sched_param sched;
int maxpri, minpri;
FILE *fp;
int no_oom = -17;
if (p->cfg.lock_memory)
mlockall(MCL_CURRENT | MCL_FUTURE);
if (p->cfg.prevent_oom) {
fp = fopen("/proc/self/oom_adj", "w");
if (fp) {
/* Don't OOM me, Bro! */
fprintf(fp, "%d", no_oom);
fclose(fp);
}
}
I'm not showing what I'm doing with scheduler parameters as its not relevant to the question.
This will prevent the OOM killer from getting your process before it has a chance to produce the (in this case) desired effect. You will also, in effect, force most other processes to disk.
So, in short, to see fireworks really quickly...
Tell the OOM killer not to bother you
Lock your memory
Allocate and initialize (zero out) blocks in a never ending loop, or until malloc() fails
Be sure to look at ulimit as well, and run your tests as root.
The code I showed is part of a daemon that simply can not fail, it runs at a very high weight (selectively using the RR or FIFO scheduler) and can not (ever) be paged out.
Have a look at this program.
When there is no longer enough memory malloc starts returning 0
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(1)
{
printf("malloc %d\n", (int)malloc(1024*1024));
}
return 0;
}
On a 32-bit Linux system, the maximum that a single process can allocate in its address space is approximately 3Gb.
This means that it is unlikely that you'll exhaust the memory with a single process.
On the other hand, on 64-bit machine you can allocate as much as you like.
As others have noted, it is also necessary to initialise the memory otherwise it does not actually consume pages.
malloc will start giving an error if EITHER the OS has no virtual memory left OR the process is out of address space (or has insufficient to satisfy the requested allocation).
Linux's VM overcommit also affects exactly when this is and what happens, as others have noted.
I just exexuted #John La Rooy's snippet:
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(1)
{
printf("malloc %d\n", (int)malloc(1024*1024));
}
return 0;
}
but it exhausted my memory very fast and cause the system hanged so that I had to restart it.
So I recommend you change some code.
For example:
On my ubuntu 16.04 LTS the code below takes about 1.5 GB ram, physical memory consumption raised from 1.8 GB to 3.3 GB after executing and go down back to 1.8GiB after finishing execution.Though it looks like I have allocate 300GiB ram in the code.
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(int i<300000)
{
printf("malloc %p\n", malloc(1024*1024));
i += 1;
}
return 0;
}
When index i is less then 100000(ie, allocate less than 100 GB), either physical or virtual memory are just very slightly used(less then 100MB), I don't know why, may be there is something to do with virtual memory.
One thing interesting is that when the physical memory begins to shrink, the addresses malloc() returns definitely changes, see picture link below.
I used malloc() and calloc(), seems that they behave similarily in occupying physical memory.
memory address number changes from 48 bits to 28 bits when physical memory begins shrinking
I was bored once and did this. Got this to eat up all memory and needed to force a reboot to get it working again.
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char** argv)
{
while(1)
{
malloc(1024 * 4);
fork();
}
}
If all you need is to stress the system, then there is stress tool, which does exactly what you want. It's available as a package for most distros.