I have a problem with memory fragmentation which can be summarized in this small example:
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char* argv[])
{
void *p[8000];int i,j;
p[0]=malloc(0x7F000);
if (p[0]==NULL)
printf("Alloc after failed!\n");
else
free(p[0]);
for (i=0;i<8000; i++) {
p[i]=malloc(0x40000);
if (p[i]==NULL){
printf("alloc failed for i=%d\n",i);
break;
}
}
for(j=0;j<i;j++) {
free(p[j]);
}
/*Alloc 1 will fail, Alloc 2 *might* fail, AlloC3 succeeds*/
p[0]=malloc(0x7F000);
if (p[0]==NULL)
printf("Alloc1 after failed!\n");
else {printf("alloc1 success\n");free(p[0]);}
p[0]=malloc(0x40000);
if (p[0]==NULL)
printf("Alloc2 after failed!\n");
else {printf("alloc2 success\n");free(p[0]);}
p[0]=malloc(0x10000);
if (p[0]==NULL)
printf("Alloc3 after failed!\n");
else {printf("alloc3 success\n");free(p[0]);}
printf("end");
}
The program prints (compiled with MSVC (both with debug and releas allocator) and MinGW on Win7):
alloc failed for i=7896
Alloc1 after failed!
alloc2 success
alloc3 success
end
Is there anyway I can avoid this? In my real application I can't avoid the scenario, my program reaches the 2GB memory limit... but I want to be able to continue by free-ing something.
Why is the fragmentation happening here in this small example in the first place? When I start to do the "free-s" why aren't the memory blocks compacted, as they should be adjacent.
Thanks!
Memory fragmentation is the consequence of allocating memory of varying sizes, each with varying lifespans. This is what creates holes in the free memory that in total are sufficient to satisfy a single allocation request, but each hole is too small by itself.
The situation in your program seems to be revealing a bug your heap management code that is not coalescing adjacent freed memory. I do expect there is a single 64 KB hole created by your allocation sequence.
To avoid this particular problem, I would simply hold on to the first allocation when I am done with it, storing it in my own "free list" so to speak. Then next time I need it, I take it from the "free list" rather than calling malloc().
Related
Can anyone explain why I get a memory leak (+1.55 kB) with code below and what am I supposed to do to avoid it ?
void TestGuid() {
UUID id;
ZeroMemory(&id, sizeof(UUID));
UuidCreate(&id);
}
int _tmain(int argc, _TCHAR* argv[]) {
TestGuid(); // Memory Snapshot 1 here
return 0; // Memory Snapshot 2 here
}
The very first call of UuidCreate makes some allocations. The first snapshot shows what was allocated, it seems it's related to initialization of random numbers generator:
But if you call UuidCreate once again, no new allocations are made. The second screenshot shows that no leaks found. So formally there are leaks but you can't fix it, and it's not a big deal - very few memory allocated.
I am running a toy example of C code in macOS Sierra 10.12.3, Xcode 8.2.1 and Instruments to visualise memory leaks and allocations.
It seems that either Instruments does not work properly or the compiler or run time engine is smart enough to solve a memory leakage on its own. All this unless I am doing a mistake here which is a very real 3rd option. Let me explain:
This is my code, 1st version:
#include <stdio.h>
#include <stdlib.h>
#define NUM_ARRAY_ELEMENS 10
#define NUM_POSITIONS_BY_ELEMEN 100
#define TIMES 200
int main(int argc, const char * argv[]) {
// insert code here...
printf("Hello, World!\n");
int *a[NUM_ARRAY_ELEMENS];
int cnt = 0;
while (cnt < TIMES) {
cnt++;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
a[i] = (int*)calloc(NUM_POSITIONS_BY_ELEMEN,sizeof(int));
}
int *p;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
//To free or access memory only on half of the positions
//if (i % 2 == 0) {
//To free memory
free(a[i]);
//Just using the memory allocated
//p = a[i];
//*p = 1;
//}
}
}
int c = getchar();
printf("Bye, World!\n");
return 0;
}
See screenshot of profile generated by Instruments for the version 1 of the code
You can see that Instruments reports 2000 MALLOC operations to allocate 400 Bytes and no memory leaks. All okay
--
For version 2 all is still fine (section of code not shown remains the same as in version 1). This time I just want to free half of the allocations and Instruments reports accordingly (ie. 1000 MALLOCs are transient and 1000 are persistent) and a leakage is reported
int *p;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
//To free or access memory only on half of the positions
if (i % 2 == 0) {
//To free memory
free(a[i]);
//Just using the memory allocated
//p = a[i];
//*p = 1;
}
}
I do not enough reputation to paste more than 2 links so will keep my credit to visualise the actual version with the error
--
It is in version 3 where things break (section of code not shown remains the same as in version 1).
int *p;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
//To free or access memory only on half of the positions
if (i % 2 == 0) {
//To free memory
//free(a[i]);
//Just using the memory allocated
//p = a[i];
//*p = 1;
}
}
This time I would expect leakage coming from all MALLOCs and see 2000 persistent allocations. But no problem is reported. Not even the 2000 allocations are shown (Neither transient nor permanent; those allocations are not reported)
See screenshot of profile generated by Instruments for the version 3 of the code
--
So what is it happening? Is the compiler or run time smart to know that the allocated memory is not used and it decides not to allocate it?
So in version 4 I access the allocated memory to see if this "stops" the "optimisation" and it does. Instruments will report allocations and leakage properly as it did in version 2.
int *p;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
//To free or access memory only on half of the positions
if (i % 2 == 0) {
//To free memory
//free(a[i]);
//Just using the memory allocated
p = a[i];
*p = 1;
}
}
--
I am going back to the initial question: Is there really a memory leakage in version 3 even when Instruments is not reporting it? If there is not a memory leakage then why not?
It seems that in version 3 there is no memory leakage (and therefore Instruments works well).
It seems that the compiler is doing an optimisation when it concludes that the memory I am allocating is never going to be used. In this case it will generate code that does not even call calloc.
The ideal way to proof my statement will be to see the generated code or know enough about compilers to be confident this is the case. I cannot do neither of that. Instead I think what I say is true based on the following.
I have analysed the size of the virtual memory used by my process and it is aligned with what Instruments is reporting and with what I will expect for each version of the code. See* below how I have done that
I have played different version of the code as I explained in my comments from 27/Feb and as soon as I assign, free or write to standard output some of the allocated memory Instruments is reporting the calloc operations. It seems this instructions are telling the compiler that it cannot avoid the call to calloc. But when I do not do these instructions and I do not use the allocated memory or just read the values in memory (ie. read into a variable never used) the compiler is optimising and not calling calloc.
*This is how I read the memory consumed by the process:
I run the process in Instruments. My process never finish as I have a getchar to stop the program. Instruments named my process as per my program name (this is visible in the tool bar. In my case MemoryLeak)
I run ps -e | grep Leak to find the pid of my process (Leak does reference the name of my program)
I run ps -p <pid> -ovsize
*** Edited 6/March/17: Based on the comment by #n.m. posted on 5/March to the original question we can see that the code generated by the (a) compiler does not call calloc in the version 3 of the code.
The Matasano blog calls “Checking the return value of malloc()” a “C Programming Anti-Idiom.” Instead malloc() should automatically call abort() for you if it fails. The argument is that, since you’ll usually want to abort the program if malloc() fails, that should be the default behaviour instead of something you have to laboriously type—or maybe forget to type!—every time.
Without getting into the merits of the idea, what’s the easiest way to set this up? I’m looking for something that would automatically detect memory allocation failures by other library functions such as asprintf() too. A portable solution would be wonderful, but I’d also be happy with something mac-specific.
Summarizing the best answers below:
Mac run-time solution
Set the MallocErrorAbort=1 environment variable before running your program. Automatically works for all memory allocation functions.
Mac/linux run-time solution
Use a dynamic library shim to load a custom malloc() wrapper at runtime with LD_PRELOAD or DYLD_INSERT_LIBRARIES. You will likely want to wrap calloc(), realloc(), &c. as well.
Mac/linux compiled solution
Define your own malloc() and free() functions, and access the system versions using dyld(RTLD_NEXT, "malloc") as shown here. Again, you will likely want to wrap calloc(), realloc(), &c. as well.
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
void *(*system_malloc)(size_t) = NULL;
void* malloc(size_t bytes) {
if (system_malloc == NULL) {
system_malloc = dlsym(RTLD_NEXT, "malloc");
}
void* ret = system_malloc(bytes);
if (ret == NULL) {
perror("malloc failed, aborting");
abort();
}
return ret;
}
int main() {
void* m = malloc(10000000000000000l);
if (m == NULL) {
perror("malloc failed, program still running");
}
return 0;
}
Linux compiled solution
Use __malloc_hook and __realloc_hook as described in the glibc manual.
Mac compiled solution
Use the malloc_default_zone() function to access the heap’s data structure, unprotect the memory page, and install a hook in zone->malloc:
#include <malloc/malloc.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <unistd.h>
static void* (*system_malloc)(struct _malloc_zone_t *zone, size_t size);
static void* my_malloc(struct _malloc_zone_t *zone, size_t size) {
void* ret = system_malloc(zone, size);
if (ret == NULL) {
perror("malloc failed, aborting");
abort();
}
return ret;
}
int main() {
malloc_zone_t *zone = malloc_default_zone();
if (zone->version != 8) {
fprintf(stderr, "Unknown malloc zone version %d\n", zone->version);
abort();
}
system_malloc = zone->malloc;
if (mprotect(zone, getpagesize(), PROT_READ | PROT_WRITE) != 0) {
perror("munprotect failed");
abort();
}
zone->malloc = my_malloc;
if (mprotect(zone, getpagesize(), PROT_READ) != 0) {
perror("mprotect failed");
abort();
}
void* m = malloc(10000000000000000l);
if (m == NULL) {
perror("malloc failed, program still running");
}
return 0;
}
For completeness you will likely want to wrap calloc(), realloc(), and the other functions defined in malloc_zone_t in /usr/include/malloc/malloc.h as well.
Just wrap malloc() in some my_malloc() function that does this instead. In a lot of cases it's actually possible to handle not being able to allocate memory so this type of behaviour would be undesirable. It's easy to add functionality to malloc() but not to remove it, which is probably why it behaves this way.
Another thing to keep in mind is that this is a library you're calling into. Would you like to make a library call and have the library kill your application without you being able to have a say in it?
I guess I missed the part about asprintf but libc exports some hooks you can use (what valgrind does essentially) that let you override the malloc behavior. Here's a reference to the hooks themselves, if you know C well enough it's not hard to do.
http://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/Hooks-for-Malloc.html
man malloc on my Mac gives the following information. It looks like you want MallocErrorAbort.
ENVIRONMENT
The following environment variables change the behavior of the allocation-related functions.
MallocLogFile <f>
Create/append messages to the given file path <f> instead of writing to the standard error.
MallocGuardEdges
If set, add a guard page before and after each large block.
MallocDoNotProtectPrelude
If set, do not add a guard page before large blocks, even if the MallocGuardEdges environment variable is set.
MallocDoNotProtectPostlude
If set, do not add a guard page after large blocks, even if the MallocGuardEdges environment variable is set.
MallocStackLogging
If set, record all stacks, so that tools like leaks can be used.
MallocStackLoggingNoCompact
If set, record all stacks in a manner that is compatible with the malloc_history program.
MallocStackLoggingDirectory
If set, records stack logs to the directory specified instead of saving them to the default location (/tmp).
MallocScribble
If set, fill memory that has been allocated with 0xaa bytes. This increases the likelihood that a program making assumptions about the contents of freshly allocated memory will fail. Also if set, fill memory that has been deallocated with 0x55 bytes. This increases the likelihood that a program will fail due to accessing memory that is no longer allocated.
MallocCheckHeapStart <s>
If set, specifies the number of allocations <s> to wait before beginning periodic heap checks every <n> as specified by MallocCheckHeapEach. If MallocCheckHeapStart is set but MallocCheckHeapEach is not specified, the default check repetition is 1000.
MallocCheckHeapEach <n>
If set, run a consistency check on the heap every <n> operations. MallocCheckHeapEach is only meaningful if MallocCheckHeapStart is also set.
MallocCheckHeapSleep <t>
Sets the number of seconds to sleep (waiting for a debugger to attach) when MallocCheckHeapStart is set and a heap corruption is detected. The default is 100 seconds. Setting this to zero means not to sleep at all. Setting this to a negative number means to sleep (for the positive number of seconds) only the very first time a heap corruption is detected.
MallocCheckHeapAbort <b>
When MallocCheckHeapStart is set and this is set to a non-zero value, causes abort(3) to be called if a heap corruption is detected, instead of any sleeping.
MallocErrorAbort
If set, causes abort(3) to be called if an error was encountered in malloc(3) or free(3), such as a calling free(3) on a pointer previously freed.
MallocCorruptionAbort
Similar to MallocErrorAbort but will not abort in out of memory conditions, making it more useful to catch only those errors which will cause memory corruption. MallocCorruptionAbort is always set on 64-bit processes.
MallocHelp
If set, print a list of environment variables that are paid heed to by the allocation-related functions, along with short descriptions. The list should correspond to this documentation.
Note the comments under MallocCorruptionAbort about the behaviour of MallocErrorAbort.
For most of my own code, I use a series of wrapper functions — emalloc(), erealloc(), ecalloc(), efree(), estrdup(), etc — that check for failed allocations (efree() is a straight pass-through function for consistency) and do not return when an allocation fails. They either exit or abort. This is basically what Jesus Ramos suggests in his answer; I agree with what he suggests.
However, not all programs can afford to have that happen. I'm just in the process of fixing up some code I wrote which does use these functions so that it can be reused in a context where it is not OK to fail to on allocation error. For its original purpose (security checks during the very early stages of process startup), it was fine to exit on error, but now it needs to be usable after the system is running, when a premature exit is not allowed. So, the code has to deal with those paths where the code used to be able to assume 'no return on allocation failure'. That's a tad painful. It can still take a conservative view; an allocation failure means the request is not safe and process it appropriately. But not all code can afford to fail with abort on memory allocation failure.
I need ideas on how to write a C program that reserve a specified amount of MB RAM until a key [ex. the any key] is pressed on a Linux 2.6 32 bit system.
*
/.eat_ram.out 200
# If free -m is execute at this time, it should report 200 MB more in the used section, than before running the program.
[Any key is pressed]
# Now all the reserved RAM should be released and the program exits.
*
It is the core functionality of the program [reserving the RAM] i do not know how to do, getting arguments from the commandline, printing [Any key is pressed] and so on is not a problem from me.
Any ideas on how to do this?
You want to use malloc() to do this. Depending on your need, you will also want to:
Write data to the memory so that the kernel actually guarantees it. You can use memset() for this.
Prevent the memory from being paged out (swapped), the mlock() / mlockall() functions can help you with this.
Tell the kernel how you actually intend to use the memory, which is accomplished via posix_madvise() (this is preferable to an explicit mlockall()).
In most realities, malloc() and memset() (or, calloc() which effectively does the same) will suit your needs.
Finally, of course, you want to free() the memory when it is no longer needed.
Can't you just use malloc() to allocate that ram to your process? That will reserve that RAM for you, and then you are free to do whatever you wish with it.
Here's an example for you:
#include <stdlib.h>
int main (int argc, char* argv[]) {
int bytesToAllocate;
char* bytesReserved = NULL;
//assume you have code here that fills bytesToAllocate
bytesReserved = malloc(bytesToAllocate);
if (bytesReserved == NULL) {
//an error occurred while reserving the memory - handle it here
}
//when the program ends:
free(bytesReserved);
return 0;
}
If you want more information, have a look at the man page (man malloc in a linux shell). If you aren't on linux, have a look at the online man page.
calloc() is what you want. It will reserve memory for your process and write zero's to it. This ensures that the memory is actually allocated for your process. If you malloc() a large portion of memory, the OS may be lazy about actually allocating memory for you, only actually allocating it when it is written to (which will never happen in this case).
You will need:
malloc() to allocate however many bytes you need (malloc(200000000) or malloc(20 * (1 << 20))).
getc() to wait for a keypress.
free() to deallocate the memory.
The information on these pages should be helpful.
Did this, should work. Although I was able to reserve more RAM than I have installed, this should work for valid values, tho.
#include <stdio.h>
#include <stdlib.h>
enum
{
MULTIPLICATOR = 1024 * 1024 // 1 MB
};
int
main(int argc, char *argv[])
{
void *reserve;
unsigned int amount;
if (argc < 2)
{
fprintf(stderr, "usage: %s <megabytes>\n", argv[0]);
return EXIT_FAILURE;
}
amount = atoi(argv[1]);
printf("About to reserve %ld MB (%ld Bytes) of RAM...\n", amount, amount * MULTIPLICATOR);
reserve = calloc(amount * MULTIPLICATOR, 1);
if (reserve == NULL)
{
fprintf(stderr, "Couldn't allocate memory\n");
return EXIT_FAILURE;
}
printf("Allocated. Press any key to release the memory.\n");
getchar();
free(reserve);
printf("Deallocated reserved memory\n");
return EXIT_SUCCESS;
}
I would like to write a program to consume all the memory available to understand the outcome. I've heard that linux starts killing the processes once it is unable to allocate the memory.
Can anyone help me with such a program.
I have written the following, but the memory doesn't seem to get exhausted:
#include <stdlib.h>
int main()
{
while(1)
{
malloc(1024*1024);
}
return 0;
}
You should write to the allocated blocks. If you just ask for memory, linux might just hand out a reservation for memory, but nothing will be allocated until the memory is accessed.
int main()
{
while(1)
{
void *m = malloc(1024*1024);
memset(m,0,1024*1024);
}
return 0;
}
You really only need to write 1 byte on every page (4096 bytes on x86 normally) though.
Linux "over commits" memory. This means that physical memory is only given to a process when the process first tries to access it, not when the malloc is first executed. To disable this behavior, do the following (as root):
echo 2 > /proc/sys/vm/overcommit_memory
Then try running your program.
Linux uses, by default, what I like to call "opportunistic allocation". This is based on the observation that a number of real programs allocate more memory than they actually use. Linux uses this to fit a bit more stuff into memory: it only allocates a memory page when it is used, not when it's allocated with malloc (or mmap or sbrk).
You may have more success if you do something like this inside your loop:
memset(malloc(1024*1024L), 'w', 1024*1024L);
In my machine, with an appropriate gb value, the following code used 100% of the memory, and even got memory into the swap.
You can see that you need to write only one byte in each page: memset(m, 0, 1);,
If you change the page size: #define PAGE_SZ (1<<12) to a bigger page size: #define PAGE_SZ (1<<13) then you won't be writing to all the pages you allocated, thus you can see in top that the memory consumption of the program goes down.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define PAGE_SZ (1<<12)
int main() {
int i;
int gb = 2; // memory to consume in GB
for (i = 0; i < ((unsigned long)gb<<30)/PAGE_SZ ; ++i) {
void *m = malloc(PAGE_SZ);
if (!m)
break;
memset(m, 0, 1);
}
printf("allocated %lu MB\n", ((unsigned long)i*PAGE_SZ)>>20);
getchar();
return 0;
}
A little known fact (though it is well documented) - you can (as root) prevent the OOM killer from claiming your process (or any other process) as one of its victims. Here is a snippet from something directly out of my editor, where I am (based on configuration data) locking all allocated memory to avoid being paged out and (optionally) telling the OOM killer not to bother me:
static int set_priority(nex_payload_t *p)
{
struct sched_param sched;
int maxpri, minpri;
FILE *fp;
int no_oom = -17;
if (p->cfg.lock_memory)
mlockall(MCL_CURRENT | MCL_FUTURE);
if (p->cfg.prevent_oom) {
fp = fopen("/proc/self/oom_adj", "w");
if (fp) {
/* Don't OOM me, Bro! */
fprintf(fp, "%d", no_oom);
fclose(fp);
}
}
I'm not showing what I'm doing with scheduler parameters as its not relevant to the question.
This will prevent the OOM killer from getting your process before it has a chance to produce the (in this case) desired effect. You will also, in effect, force most other processes to disk.
So, in short, to see fireworks really quickly...
Tell the OOM killer not to bother you
Lock your memory
Allocate and initialize (zero out) blocks in a never ending loop, or until malloc() fails
Be sure to look at ulimit as well, and run your tests as root.
The code I showed is part of a daemon that simply can not fail, it runs at a very high weight (selectively using the RR or FIFO scheduler) and can not (ever) be paged out.
Have a look at this program.
When there is no longer enough memory malloc starts returning 0
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(1)
{
printf("malloc %d\n", (int)malloc(1024*1024));
}
return 0;
}
On a 32-bit Linux system, the maximum that a single process can allocate in its address space is approximately 3Gb.
This means that it is unlikely that you'll exhaust the memory with a single process.
On the other hand, on 64-bit machine you can allocate as much as you like.
As others have noted, it is also necessary to initialise the memory otherwise it does not actually consume pages.
malloc will start giving an error if EITHER the OS has no virtual memory left OR the process is out of address space (or has insufficient to satisfy the requested allocation).
Linux's VM overcommit also affects exactly when this is and what happens, as others have noted.
I just exexuted #John La Rooy's snippet:
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(1)
{
printf("malloc %d\n", (int)malloc(1024*1024));
}
return 0;
}
but it exhausted my memory very fast and cause the system hanged so that I had to restart it.
So I recommend you change some code.
For example:
On my ubuntu 16.04 LTS the code below takes about 1.5 GB ram, physical memory consumption raised from 1.8 GB to 3.3 GB after executing and go down back to 1.8GiB after finishing execution.Though it looks like I have allocate 300GiB ram in the code.
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(int i<300000)
{
printf("malloc %p\n", malloc(1024*1024));
i += 1;
}
return 0;
}
When index i is less then 100000(ie, allocate less than 100 GB), either physical or virtual memory are just very slightly used(less then 100MB), I don't know why, may be there is something to do with virtual memory.
One thing interesting is that when the physical memory begins to shrink, the addresses malloc() returns definitely changes, see picture link below.
I used malloc() and calloc(), seems that they behave similarily in occupying physical memory.
memory address number changes from 48 bits to 28 bits when physical memory begins shrinking
I was bored once and did this. Got this to eat up all memory and needed to force a reboot to get it working again.
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char** argv)
{
while(1)
{
malloc(1024 * 4);
fork();
}
}
If all you need is to stress the system, then there is stress tool, which does exactly what you want. It's available as a package for most distros.