In my C program, based on the user's input, memory will be allocated for a given simulation. The initial problem I faced is that user can ask for a huge number to allocate but malloc() never fails until it runs out of memory then the program crashes.
I investigated the logic behind this and it now makes sense to me, see [1][2]. A possible workaround given here "SIGKILL while allocating memory in C++" suggests to set overcommit_memory in in /proc/sys/vm/overcommit_memory from 0 to 2.
This solved the problem from one side. But since I am using -fsanitize=address I get error from sanitizer.
Is there any better solution to this?
I guess the clang AddressSanitizer is failing because there is a legit leak. So my answer ignores that:
Alternatives:
Disable the overcommit behaviour, as you have already figured out: that is going to affect other processes and requires root.
run you app in a docker image with the oom killer disabled: that doesn't affect other processes but requires root to install docker (this is my favourite solution though).
write after malloc: may take long to alloc a huge chunk of memory and your process can still get killed because of other running process but doesn't require root.
use ulimit -v to limit the amount of memory depending on the machine: that also doesn't require root but your process might be killed anyway.
Code for the third alternative (for linux):
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <setjmp.h>
jmp_buf resume_malloc;
void handle_malloc_error(int sig)
{
longjmp(resume_malloc, sig);
}
void *memalloc(size_t sz) {
void *p = 0;
int sig = setjmp(resume_malloc);
if ( sig == 0 ) {
p = malloc(sz);
signal(SIGSEGV, &handle_malloc_error);
memset(p, 0, sz);
} else {
p = 0;
}
signal(SIGSEGV, SIG_DFL);
return p;
}
int main(int argc, char *argv[])
{
size_t sz = 160L * 1024 * 1024 * 1024L;
void *p;
for (int i=0; i < 100; i++) {
printf("size: %lu\n", sz);
p = memalloc(sz);
if ( p == 0 ) {
printf("out of memory\n");
break;
}
sz *= 2;
}
}
Related
I am performing a memory intensive task. I would like to add a warning message when the operating system runs out of memory to allocate. When I run the code below, the process gets killed and my if statement for when the pointer is equal to NULL does not work.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define ONEGB (1<<30)
int main() {
int count=0;
while(1) {
int *p = malloc(ONEGB);
if (p == NULL) {
printf("There is not much memory left after %d GB\n", count);
return 0;
}
memset(p,1,ONEGB);
fprintf(stderr, "got %d GB\n", ++count);
}
}
The output was
...
got 25 GB
got 26 GB
got 27 GB
Killed
Is there some other way to tell the user that there is no memory left before the OS kills the process? I am mostly interested in running this on linux servers only.
Suppose I'm running a piece of code on a 32 bit CPU and plenty of memory. And a process uses mmap to map a total of 2.8GB worth of file into it's address space. Then the process tries to allocate 500MB of memory using malloc. The allocation is bounded to fail and returns NULL due to not having enough address space; even though the system may have enough allocate-able memory.
The code looks something like this:
int main()
{
int fd = open("some_2.8GB file",...);
void* file_ptr = mmap(..., fd, ...);
void* ptr = malloc(500*1024*1024);
// malloc will fail because on 32bit Linux, a process can only have 3GB of address space
assert(ptr == NULL);
if(out_of_address_space())
printf("You ran out of address space, but system still have free memory\n");
else
printf("Out of memory\n");
}
How could I detect the failure is caused by out of address space instead of allocate-able memory? Is out_of_address_space possible to implement?
How could I detect the failure is caused by out of address space instead of allocate-able memory?
You could calculate the amount of maximum virtual memory like bash does in ulimit -v - by querying getrlimit().
You can calculate the amount of "allocated" virtual memory by summing the differences between second and first column in /proc/pid/maps file.
Then the difference will give you the amount of "free" virtual space. You can compare that with the size you want to allocate and know if there is enough free virtual space.
Example: Let's compile a small program:
$ gcc -xc - <<EOF
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
int main() {
void *p = malloc(1024 * 1024);
printf("%ld %p\\n", (long)getpid(), p);
sleep(100);
}
EOF
The program will allocate 1MB, print it's pid and address and sleep so we have time to do something. On my system if I limit virtual memory to 2.5M the allocation fails:
$ ( ulimit -v 2500; ./a.out; )
94895 (nil)
If I then sum the maps file:
$ sudo cat /proc/94895/maps | awk -F'[- ]' --non-decimal-data '{a=sprintf("%d", "0x"$1); b=sprintf("%d", "0x"$2); sum += b-a; } END{print sum/1024 " Kb"}'
2320 Kb
Knowing that the limit was set to 2500 Kb and the process is using 2320 Kb, there is only space to allocate 180 Kb, not more.
Possible C implementation for fun:
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <stdbool.h>
size_t address_space_max(void) {
struct rlimit l;
if (getrlimit(RLIMIT_AS, &l) < 0) return -1;
return l.rlim_cur;
}
size_t address_space_used(void) {
const unsigned long long pid = getpid();
const char fmt[] = "/proc/%llu/maps";
const int flen = snprintf(NULL, 0, fmt, pid);
char * const fname = malloc(flen + 1);
if (fname == NULL) return -1;
sprintf(fname, fmt, pid);
FILE *f = fopen(fname, "r");
free(fname);
if (f == NULL) return -1;
long long a, b;
long long sum = 0;
while (fscanf(f, "%llx-%llx%*[^\n]*", &a, &b) == 2) {
sum += b - a;
}
fclose(f);
return sum;
}
size_t address_space_free(void) {
const size_t max = address_space_max();
if (max == (size_t)-1) return -1;
const size_t used = address_space_used();
if (used == (size_t)-1) return -1;
return max - used;
}
/**
* Compares if there is enough address space for size
*/
bool out_of_address_space(size_t size) {
return address_space_free() < size;
}
int main() {
printf("%zu Kb\n", address_space_free()/1024);
// ie. use:
// if (out_of_address_space(500 * 1024 * 1024))
}
And a process uses mmap to map a total of 2.8GB worth of file into it's address space. Then the process tries to allocate 500MB of memory using malloc.
Don't mmap(2) the entire file at once !
Do mmap no more than one gigabyte (and no more than 2.5 gigabytes in total on a 32 bits Linux, including malloc-related mmap or sbrk). Then use mremap(2) and/or munmap(2). See also madvise(2). Be aware of the m modifier to fopen(3) mode string. In some cases, stdio(3) functions (using fseek and fread) might be enough and you could replace your mmap with them. Be aware of memory overcommitment and of the page cache. Both could be tunable thru /sys/ or /proc/ (see sysconf(3), sysfs(5), proc(5)...) and might be monitorable thru inotify(7) or userfaultfd(2) and/or signal(7).
Notice that malloc(3), dlopen(3) and shared libraries are also mmap-ing (thru ld.so(8)...) - and some malloc implementations are sometimes using sbrk(2) to manage small memory chunks. As an optimization, free(3) does not always munmap. Check with strace(1) and pmap(1) (or programmatically thru /proc/self/maps or /proc/self/status or /proc/self/statm, see proc(5)).
Some 32 bits Linux kernels could be specially configured (at their compile time) to accept slightly more than 3GBytes of virtual address space. I forgot the details. Ask on https://kernelnewbies.org/
Study the source code of your C standard library (e.g. GNU glibc). Most of them are open-source so you can improve them, e.g. musl-libc. You could use others (e.g. dietlibc), and you usually can redefine malloc. Budget a few months of efforts.
Read also Advanced Linux Programming (also here), Modern C, syscalls(2), the documentation of your C standard library, and a good operating system textbook.
I am having trouble trying to figure out why my program cannot save more than 2GB of data to a file. I cannot tell if this is a programming or environment (OS) problem. Here is my source code:
#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
#define _FILE_OFFSET_BITS 64
#include <math.h>
#include <time.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/*-------------------------------------*/
//for file mapping in Linux
#include<fcntl.h>
#include<unistd.h>
#include<sys/stat.h>
#include<sys/time.h>
#include<sys/mman.h>
#include<sys/types.h>
/*-------------------------------------*/
#define PERMS 0600
#define NEW(type) (type *) malloc(sizeof(type))
#define FILE_MODE (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH)
void write_result(char *filename, char *data, long long length){
int fd, fq;
fd = open(filename, O_RDWR|O_CREAT|O_LARGEFILE, 0644);
if (fd < 0) {
perror(filename);
return -1;
}
if (ftruncate(fd, length) < 0)
{
printf("[%d]-ftruncate64 error: %s/n", errno, strerror(errno));
close(fd);
return 0;
}
fq = write (fd, data,length);
close(fd);
return;
}
main()
{
long long offset = 3000000000; // 3GB
char * ttt;
ttt = (char *)malloc(sizeof(char) *offset);
printf("length->%lld\n",strlen(ttt)); // length=0
memset (ttt,1,offset);
printf("length->%lld\n",strlen(ttt)); // length=3GB
write_result("test.big",ttt,offset);
return 1;
}
According to my test, the program can generate a file large than 2GB and can allocate such large memory as well.
The weird thing happened when I tried to write data into the file. I checked the file and it is empty, which is supposed to be filled with 1.
Can any one be kind and help me with this?
You need to read a little more about C strings and what malloc and calloc do.
In your original main ttt pointed to whatever garbage was in memory when malloc was called. This means a nul terminator (the end marker of a C String, which is binary 0) could be anywhere in the garbage returned by malloc.
Also, since malloc does not touch every byte of the allocated memory (and you're asking for a lot) you could get sparse memory which means the memory is not actually physically available until it is read or written.
calloc allocates and fills the allocated memory with 0. It is a little more prone to fail because of this (it touches every byte allocated, so if the OS left the allocation sparse it will not be sparse after calloc fills it.)
Here's your code with fixes for the above issues.
You should also always check the return value from write and react accordingly. I'll leave that to you...
main()
{
long long offset = 3000000000; // 3GB
char * ttt;
//ttt = (char *)malloc(sizeof(char) *offset);
ttt = (char *)calloc( sizeof( char ), offset ); // instead of malloc( ... )
if( !ttt )
{
puts( "calloc failed, bye bye now!" );
exit( 87 );
}
printf("length->%lld\n",strlen(ttt)); // length=0 (This now works as expected if calloc does not fail)
memset( ttt, 1, offset );
ttt[offset - 1] = 0; // Now it's nul terminated and the printf below will work
printf("length->%lld\n",strlen(ttt)); // length=3GB
write_result("test.big",ttt,offset);
return 1;
}
Note to Linux gurus... I know sparse may not be the correct term. Please correct me if I'm wrong as it's been a while since I've been buried in Linux minutiae. :)
Looks like you're hitting the internal file system's limitation for the iDevice: ios - Enterprise app with more than resource files of size 2GB
2Gb+ files are simply not possible. If you need to store such amount of data you should consider using some other tools or write the file chunk manager.
I'm going to go out on a limb here and say that your problem may lay in memset().
The best thing to do here is, I think, after memset() ing it,
for (unsigned long i = 0; i < 3000000000; i++) {
if (ttt[i] != 1) { printf("error in data at location %d", i); break; }
}
Once you've validated that the data you're trying to write is correct, then you should look into writing a smaller file such as 1GB and see if you have the same problems. Eliminate each and every possible variable and you will find the answer.
The following program is killed by the kernel when the memory is ran out. I would like to know when the global variable should be assigned to "ENOMEM".
#define MEGABYTE 1024*1024
#define TRUE 1
int main(int argc, char *argv[]){
void *myblock = NULL;
int count = 0;
while(TRUE)
{
myblock = (void *) malloc(MEGABYTE);
if (!myblock) break;
memset(myblock,1, MEGABYTE);
printf("Currently allocating %d MB\n",++count);
}
exit(0);
}
First, fix your kernel not to overcommit:
echo "2" > /proc/sys/vm/overcommit_memory
Now malloc should behave properly.
As "R" hinted, the problem is the default behaviour of Linux memory management, which is "overcommiting". This means that the kernel claims to allocate you memory successfuly, but doesn't actually allocate the memory until later when you try to access it. If the kernel finds out that it's allocated too much memory, it kills a process with "the OOM (Out Of Memory) killer" to free up some memory. The way it picks the process to kill is complicated, but if you have just allocated most of the memory in the system, it's probably going to be your process that gets the bullet.
If you think this sounds crazy, some people would agree with you.
To get it to behave as you expect, as R said:
echo "2" > /proc/sys/vm/overcommit_memory
It happens when you try to allocate too much memory at once.
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
int main(int argc, char *argv[])
{
void *p;
p = malloc(1024L * 1024 * 1024 * 1024);
if(p == NULL)
{
printf("%d\n", errno);
perror("malloc");
}
}
In your case the OOM killer is getting to the process first.
I think errno will be set to ENOMEM:
Macro defined in stdio.h. Here is the documentation.
#define ENOMEM 12 /* Out of Memory */
After you call malloc in this statement:
myblock = (void *) malloc(MEGABYTE);
And the function returns NULL -because system is out of memory -.
I found this SO question very interesting.
Hope it helps!
I would like to write a program to consume all the memory available to understand the outcome. I've heard that linux starts killing the processes once it is unable to allocate the memory.
Can anyone help me with such a program.
I have written the following, but the memory doesn't seem to get exhausted:
#include <stdlib.h>
int main()
{
while(1)
{
malloc(1024*1024);
}
return 0;
}
You should write to the allocated blocks. If you just ask for memory, linux might just hand out a reservation for memory, but nothing will be allocated until the memory is accessed.
int main()
{
while(1)
{
void *m = malloc(1024*1024);
memset(m,0,1024*1024);
}
return 0;
}
You really only need to write 1 byte on every page (4096 bytes on x86 normally) though.
Linux "over commits" memory. This means that physical memory is only given to a process when the process first tries to access it, not when the malloc is first executed. To disable this behavior, do the following (as root):
echo 2 > /proc/sys/vm/overcommit_memory
Then try running your program.
Linux uses, by default, what I like to call "opportunistic allocation". This is based on the observation that a number of real programs allocate more memory than they actually use. Linux uses this to fit a bit more stuff into memory: it only allocates a memory page when it is used, not when it's allocated with malloc (or mmap or sbrk).
You may have more success if you do something like this inside your loop:
memset(malloc(1024*1024L), 'w', 1024*1024L);
In my machine, with an appropriate gb value, the following code used 100% of the memory, and even got memory into the swap.
You can see that you need to write only one byte in each page: memset(m, 0, 1);,
If you change the page size: #define PAGE_SZ (1<<12) to a bigger page size: #define PAGE_SZ (1<<13) then you won't be writing to all the pages you allocated, thus you can see in top that the memory consumption of the program goes down.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define PAGE_SZ (1<<12)
int main() {
int i;
int gb = 2; // memory to consume in GB
for (i = 0; i < ((unsigned long)gb<<30)/PAGE_SZ ; ++i) {
void *m = malloc(PAGE_SZ);
if (!m)
break;
memset(m, 0, 1);
}
printf("allocated %lu MB\n", ((unsigned long)i*PAGE_SZ)>>20);
getchar();
return 0;
}
A little known fact (though it is well documented) - you can (as root) prevent the OOM killer from claiming your process (or any other process) as one of its victims. Here is a snippet from something directly out of my editor, where I am (based on configuration data) locking all allocated memory to avoid being paged out and (optionally) telling the OOM killer not to bother me:
static int set_priority(nex_payload_t *p)
{
struct sched_param sched;
int maxpri, minpri;
FILE *fp;
int no_oom = -17;
if (p->cfg.lock_memory)
mlockall(MCL_CURRENT | MCL_FUTURE);
if (p->cfg.prevent_oom) {
fp = fopen("/proc/self/oom_adj", "w");
if (fp) {
/* Don't OOM me, Bro! */
fprintf(fp, "%d", no_oom);
fclose(fp);
}
}
I'm not showing what I'm doing with scheduler parameters as its not relevant to the question.
This will prevent the OOM killer from getting your process before it has a chance to produce the (in this case) desired effect. You will also, in effect, force most other processes to disk.
So, in short, to see fireworks really quickly...
Tell the OOM killer not to bother you
Lock your memory
Allocate and initialize (zero out) blocks in a never ending loop, or until malloc() fails
Be sure to look at ulimit as well, and run your tests as root.
The code I showed is part of a daemon that simply can not fail, it runs at a very high weight (selectively using the RR or FIFO scheduler) and can not (ever) be paged out.
Have a look at this program.
When there is no longer enough memory malloc starts returning 0
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(1)
{
printf("malloc %d\n", (int)malloc(1024*1024));
}
return 0;
}
On a 32-bit Linux system, the maximum that a single process can allocate in its address space is approximately 3Gb.
This means that it is unlikely that you'll exhaust the memory with a single process.
On the other hand, on 64-bit machine you can allocate as much as you like.
As others have noted, it is also necessary to initialise the memory otherwise it does not actually consume pages.
malloc will start giving an error if EITHER the OS has no virtual memory left OR the process is out of address space (or has insufficient to satisfy the requested allocation).
Linux's VM overcommit also affects exactly when this is and what happens, as others have noted.
I just exexuted #John La Rooy's snippet:
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(1)
{
printf("malloc %d\n", (int)malloc(1024*1024));
}
return 0;
}
but it exhausted my memory very fast and cause the system hanged so that I had to restart it.
So I recommend you change some code.
For example:
On my ubuntu 16.04 LTS the code below takes about 1.5 GB ram, physical memory consumption raised from 1.8 GB to 3.3 GB after executing and go down back to 1.8GiB after finishing execution.Though it looks like I have allocate 300GiB ram in the code.
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(int i<300000)
{
printf("malloc %p\n", malloc(1024*1024));
i += 1;
}
return 0;
}
When index i is less then 100000(ie, allocate less than 100 GB), either physical or virtual memory are just very slightly used(less then 100MB), I don't know why, may be there is something to do with virtual memory.
One thing interesting is that when the physical memory begins to shrink, the addresses malloc() returns definitely changes, see picture link below.
I used malloc() and calloc(), seems that they behave similarily in occupying physical memory.
memory address number changes from 48 bits to 28 bits when physical memory begins shrinking
I was bored once and did this. Got this to eat up all memory and needed to force a reboot to get it working again.
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char** argv)
{
while(1)
{
malloc(1024 * 4);
fork();
}
}
If all you need is to stress the system, then there is stress tool, which does exactly what you want. It's available as a package for most distros.