I am running a toy example of C code in macOS Sierra 10.12.3, Xcode 8.2.1 and Instruments to visualise memory leaks and allocations.
It seems that either Instruments does not work properly or the compiler or run time engine is smart enough to solve a memory leakage on its own. All this unless I am doing a mistake here which is a very real 3rd option. Let me explain:
This is my code, 1st version:
#include <stdio.h>
#include <stdlib.h>
#define NUM_ARRAY_ELEMENS 10
#define NUM_POSITIONS_BY_ELEMEN 100
#define TIMES 200
int main(int argc, const char * argv[]) {
// insert code here...
printf("Hello, World!\n");
int *a[NUM_ARRAY_ELEMENS];
int cnt = 0;
while (cnt < TIMES) {
cnt++;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
a[i] = (int*)calloc(NUM_POSITIONS_BY_ELEMEN,sizeof(int));
}
int *p;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
//To free or access memory only on half of the positions
//if (i % 2 == 0) {
//To free memory
free(a[i]);
//Just using the memory allocated
//p = a[i];
//*p = 1;
//}
}
}
int c = getchar();
printf("Bye, World!\n");
return 0;
}
See screenshot of profile generated by Instruments for the version 1 of the code
You can see that Instruments reports 2000 MALLOC operations to allocate 400 Bytes and no memory leaks. All okay
--
For version 2 all is still fine (section of code not shown remains the same as in version 1). This time I just want to free half of the allocations and Instruments reports accordingly (ie. 1000 MALLOCs are transient and 1000 are persistent) and a leakage is reported
int *p;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
//To free or access memory only on half of the positions
if (i % 2 == 0) {
//To free memory
free(a[i]);
//Just using the memory allocated
//p = a[i];
//*p = 1;
}
}
I do not enough reputation to paste more than 2 links so will keep my credit to visualise the actual version with the error
--
It is in version 3 where things break (section of code not shown remains the same as in version 1).
int *p;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
//To free or access memory only on half of the positions
if (i % 2 == 0) {
//To free memory
//free(a[i]);
//Just using the memory allocated
//p = a[i];
//*p = 1;
}
}
This time I would expect leakage coming from all MALLOCs and see 2000 persistent allocations. But no problem is reported. Not even the 2000 allocations are shown (Neither transient nor permanent; those allocations are not reported)
See screenshot of profile generated by Instruments for the version 3 of the code
--
So what is it happening? Is the compiler or run time smart to know that the allocated memory is not used and it decides not to allocate it?
So in version 4 I access the allocated memory to see if this "stops" the "optimisation" and it does. Instruments will report allocations and leakage properly as it did in version 2.
int *p;
for (int i=0;i < NUM_ARRAY_ELEMENS ;i++) {
//To free or access memory only on half of the positions
if (i % 2 == 0) {
//To free memory
//free(a[i]);
//Just using the memory allocated
p = a[i];
*p = 1;
}
}
--
I am going back to the initial question: Is there really a memory leakage in version 3 even when Instruments is not reporting it? If there is not a memory leakage then why not?
It seems that in version 3 there is no memory leakage (and therefore Instruments works well).
It seems that the compiler is doing an optimisation when it concludes that the memory I am allocating is never going to be used. In this case it will generate code that does not even call calloc.
The ideal way to proof my statement will be to see the generated code or know enough about compilers to be confident this is the case. I cannot do neither of that. Instead I think what I say is true based on the following.
I have analysed the size of the virtual memory used by my process and it is aligned with what Instruments is reporting and with what I will expect for each version of the code. See* below how I have done that
I have played different version of the code as I explained in my comments from 27/Feb and as soon as I assign, free or write to standard output some of the allocated memory Instruments is reporting the calloc operations. It seems this instructions are telling the compiler that it cannot avoid the call to calloc. But when I do not do these instructions and I do not use the allocated memory or just read the values in memory (ie. read into a variable never used) the compiler is optimising and not calling calloc.
*This is how I read the memory consumed by the process:
I run the process in Instruments. My process never finish as I have a getchar to stop the program. Instruments named my process as per my program name (this is visible in the tool bar. In my case MemoryLeak)
I run ps -e | grep Leak to find the pid of my process (Leak does reference the name of my program)
I run ps -p <pid> -ovsize
*** Edited 6/March/17: Based on the comment by #n.m. posted on 5/March to the original question we can see that the code generated by the (a) compiler does not call calloc in the version 3 of the code.
Related
I'm new to C and I would like to read content from range of memory address
Assume that I have following range of address : 0x00065580 - 0x000655c0
this range get from command :
$ cat /proc/a_process_pid/maps | grep heap
00065580-000655c0 ...........heap
( Please see the hex dump image from above range )
I tried to using loop but have no luck ,( just like the think of a Java dev )
#include <stdio.h>
#define START_ADDR 0x00065580
#define END_ADDR 0x000655c0
int main(){
char *start = START_ADDR ;
char *end = END_ADDR;
for( char *i=start ; i <= end ; i++ ){
printf("%s",i);
}
return 0;
}
It generate error :
root#localhost:~# ./test
Segmentation fault (core dumped)
Please tell me what am I wrong and what I need to learn about ?
Modern operating systems use virtual memory: the memory addresses that a process sees are different from the physical addresses, and programs normally can't touch the memory that is owned by another process. You appear to be looking at a memory dump from one particular process, but address 0x00065580 does in reality correspond to a different physical address, e.g. 0x00123456. And inside your own program, address 0x00065580 will correspond to another physical address, e.g. 0x00589263. The operating system will ensure that none of the physical addresses that are used by the other program will be accessible from yours.
If your process is running as root, you can use system calls to indirectly access other processes' memory.
Here is a C++ program which will shed some light on how to print addresses (or data) for particular memory locations.
#include<iostream>
using namespace std;
int main()
{
int* arr = new int[10];
int *start = NULL;
int *end = NULL;
for(int i=0;i<10;i++)
arr[i] = i+1;
start = arr;
end = &arr[9];
for(int *start=arr;start<end;start++)
cout<<" "<<start;
cout<<endl;
system("PAUSE");
return 0;
}
I do not see any issue in your pasted code. But the problem is the START and END, i.e. the location of addresses you are using in your program. These locations does not contain any valid memory for that process hence you are forcefully asking to read data from some junk memory space resulting in SEGFAULT. Hope this helps.
LPTSTR name = NULL;
DWORD nameLength = 0;
namelength = host->nameLength; // returns 10
name = (LPTSTR) malloc( sizeof(nameLength * sizeof(TCHAR))); //crashes here
I don't understand the reason for its crashing at this point. Could somebody explain why?
Update =*(deleted the next line after the crashing line, had copied it by mistake. was just a commented out line in the code)
UPDATE:
Sorry guys, I had tried all the ways you have described before asking the question. Doesn't work.
I think its some other issue. heres a windows service, calling up the function above (from a dll) when the computer starts, so was doing a remote debugging the dll using windbg ( I break-in using a hard-coded debugbreak, just before the function gets called).
when I am over the malloc step and give a "next step" instruction (F10), it doesn't go to the next step, instead says the client is running, but then suddenly breaks in at nt!DbgLoadImageSymbols with "leave" instruction. Giving a go(F5) after this keeps the machine in a hanged state.
If you crash inside of malloc, then it means that you have previously corrupted the heap (or more accurately, the double-linked lists that organize the heap).
Considering that you have a glaring bug here:
name = (LPTSTR) malloc( sizeof(nameLength * sizeof(TCHAR)));
You should review all of your malloc calls and ensure that you are allocating enough memory. What you have likely done is allocate too small of a buffer, written too much data into the returned pointer corrupting the heap, and then crashed in a later call to malloc.
Since you are on Windows, you can also utilize page-heap verification (via the gflags tool). This will help you catch buffer overwrites when they happen.
Not enough info for an answer, but too much for a comment, sorry. I made a simple main() based as closely on your clues as I can see, with any previously commented errors uncorrected, but extra lines FYI how much memory is allocated. The program compiles and runs without complaint. So your problem has not been properly expressed.
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
int main(){
LPTSTR name = NULL;
DWORD nameLength = 10;
name = (LPTSTR) malloc( sizeof(nameLength * sizeof(TCHAR)));
if (name) {
printf ("Memory for %d bytes allocated\n", sizeof(nameLength * sizeof(TCHAR)));
free (name);
} else {
printf ("No memory for %d bytes\n", sizeof(nameLength * sizeof(TCHAR)));
}
return 0;
}
Program output:
Memory for 4 bytes allocated
What is clear, is that it's unlikely to be enough memory for whatever you have said is 10.
I have a problem with memory fragmentation which can be summarized in this small example:
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char* argv[])
{
void *p[8000];int i,j;
p[0]=malloc(0x7F000);
if (p[0]==NULL)
printf("Alloc after failed!\n");
else
free(p[0]);
for (i=0;i<8000; i++) {
p[i]=malloc(0x40000);
if (p[i]==NULL){
printf("alloc failed for i=%d\n",i);
break;
}
}
for(j=0;j<i;j++) {
free(p[j]);
}
/*Alloc 1 will fail, Alloc 2 *might* fail, AlloC3 succeeds*/
p[0]=malloc(0x7F000);
if (p[0]==NULL)
printf("Alloc1 after failed!\n");
else {printf("alloc1 success\n");free(p[0]);}
p[0]=malloc(0x40000);
if (p[0]==NULL)
printf("Alloc2 after failed!\n");
else {printf("alloc2 success\n");free(p[0]);}
p[0]=malloc(0x10000);
if (p[0]==NULL)
printf("Alloc3 after failed!\n");
else {printf("alloc3 success\n");free(p[0]);}
printf("end");
}
The program prints (compiled with MSVC (both with debug and releas allocator) and MinGW on Win7):
alloc failed for i=7896
Alloc1 after failed!
alloc2 success
alloc3 success
end
Is there anyway I can avoid this? In my real application I can't avoid the scenario, my program reaches the 2GB memory limit... but I want to be able to continue by free-ing something.
Why is the fragmentation happening here in this small example in the first place? When I start to do the "free-s" why aren't the memory blocks compacted, as they should be adjacent.
Thanks!
Memory fragmentation is the consequence of allocating memory of varying sizes, each with varying lifespans. This is what creates holes in the free memory that in total are sufficient to satisfy a single allocation request, but each hole is too small by itself.
The situation in your program seems to be revealing a bug your heap management code that is not coalescing adjacent freed memory. I do expect there is a single 64 KB hole created by your allocation sequence.
To avoid this particular problem, I would simply hold on to the first allocation when I am done with it, storing it in my own "free list" so to speak. Then next time I need it, I take it from the "free list" rather than calling malloc().
Here's a small program to a college's task:
#include <unistd.h>
#ifndef BUFFERSIZE
#define BUFFERSIZE 1
#endif
main()
{
char buffer[BUFFERSIZE];
int i;
int j = BUFFERSIZE;
i = read(0, buffer, BUFFERSIZE);
while (i>0)
{
write(1, buffer, i);
i = read(0, buffer, BUFFERSIZE);
}
return 0;
}
There is a alternative using the stdio.h fread and fwrite functions instead.
Well. I compiled this both versions of program with 25 different values of Buffer Size: 1, 2, 4, ..., 2^i with i=0..30
This is a example of how I compile it:
gcc -DBUFFERSIZE=8388608 prog_sys.c -o bin/psys.8M
The question: In my machine (Ubuntu Precise 64, more details in the end) all versions of the program works fine:
./psys.1M < data
(data is a small file with 3 line ascii text.)
The problem is: When the buffer size is 8MB or greater. Both versions (using system call or clib functions) crashes with these buffer sizes (Segmentation Fault).
I tested many things. The first version of the code was like this:
(...)
main()
{
char buffer[BUFFERSIZE];
int i;
i = read(0, buffer, BUFFERSIZE);
(...)
This crashs when I call the read function. But with these versions:
main()
{
char buffer[BUFFERSIZE]; // SEGMENTATION FAULT HERE
int i;
int j = BUFFERSIZE;
i = read(0, buffer, BUFFERSIZE);
main()
{
int j = BUFFERSIZE; // SEGMENTATION FAULT HERE
char buffer[BUFFERSIZE];
int i;
i = read(0, buffer, BUFFERSIZE);
Both them crashs (SEGFAULT) in the first line of main. However, if I move the buffer out of main to the global scope (thus, allocation in the heap instead the stack), this works fine:
char buffer[BUFFERSIZE]; //NOW GLOBAL AND WORKING FINE
main()
{
int j = BUFFERSIZE;
int i;
i = read(0, buffer, BUFFERSIZE);
I use a Ubuntu Precise 12.04 64bits and Intel i5 M 480 1st generation.
#uname -a
Linux hostname 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
I don't know the OS limitations about the stack. Is there some way to allocate big data in stack, even if this is not a good pratice?
I think it is pretty simple: if you try to make the buffer too big, it overflows the stack.
If you use malloc() to allocate the buffer, I bet you will have no problem.
Note that there is a function called alloca() that explicitly allocates stack storage. Using alloca() is pretty much the same thing as declaring a stack variable, except that it happens while your program is running. Reading the man page for alloca() or discussions of it may help you to understand what is going wrong with your program. Here's a good discussion:
Why is the use of alloca() not considered good practice?
EDIT: In a comment, #jim mcnamara told us about ulimit, a command-line tool that can check on user limits. On this computer (running Linux Mint 14, so it should be the same limits as Ubuntu 12.10) the ulimit -s command shows a stack size limit of: 8192 K-bytes, which tracks nicely with the problem as you describe it.
EDIT: In case it wasn't completely clear, I recommend that you solve the problem by calling malloc(). Another acceptable solution would be to statically allocate the memory, which will work fine as long as your code is single-threaded.
You should only use stack allocation for small buffers, precisely because you don't want to blow up your stack. If your buffers are big, or your code will recursively call a function many times such that the little buffers will be allocated many many times, you will clobber your stack and your code will crash. The worst part is that you won't get any warning: it will either be fine, or your program already crashed.
The only disadvantage of malloc() is that it is relatively slow, so you don't want to call malloc() inside time-critical code. But it's fine for initial setup; once the malloc is done, the address of your allocated buffer is just an address in memory like any other memory address within your program's address space.
I specifically recommend against editing the system defaults to make the stack size bigger, because that makes your program much less portable. If you call standard C library functions like malloc() you could port your code to Windows, Mac, Android, etc. with little trouble; if you start calling system functions to change the default stack size, you would have much more problems porting. (And you may have no plans to port this now, but plans change!)
The stack size is quite often limited in Linux. The command ulimit -s will give the current value, in Kbytes. You can change the default in (usually) the file /etc/security/limits.conf. You can also, depending on privileges, change it on a per-process basis via the code:
#include <sys/resource.h>
// ...
struct rlimit x;
if (getrlimit(RLIMIT_STACK, &x) < 0)
perror("getrlimit");
x.rlim_cur = RLIM_INFINITY;
if (setrlimit(RLIMIT_STACK, &x) < 0)
perror("setrlimit");
I would like to write a program to consume all the memory available to understand the outcome. I've heard that linux starts killing the processes once it is unable to allocate the memory.
Can anyone help me with such a program.
I have written the following, but the memory doesn't seem to get exhausted:
#include <stdlib.h>
int main()
{
while(1)
{
malloc(1024*1024);
}
return 0;
}
You should write to the allocated blocks. If you just ask for memory, linux might just hand out a reservation for memory, but nothing will be allocated until the memory is accessed.
int main()
{
while(1)
{
void *m = malloc(1024*1024);
memset(m,0,1024*1024);
}
return 0;
}
You really only need to write 1 byte on every page (4096 bytes on x86 normally) though.
Linux "over commits" memory. This means that physical memory is only given to a process when the process first tries to access it, not when the malloc is first executed. To disable this behavior, do the following (as root):
echo 2 > /proc/sys/vm/overcommit_memory
Then try running your program.
Linux uses, by default, what I like to call "opportunistic allocation". This is based on the observation that a number of real programs allocate more memory than they actually use. Linux uses this to fit a bit more stuff into memory: it only allocates a memory page when it is used, not when it's allocated with malloc (or mmap or sbrk).
You may have more success if you do something like this inside your loop:
memset(malloc(1024*1024L), 'w', 1024*1024L);
In my machine, with an appropriate gb value, the following code used 100% of the memory, and even got memory into the swap.
You can see that you need to write only one byte in each page: memset(m, 0, 1);,
If you change the page size: #define PAGE_SZ (1<<12) to a bigger page size: #define PAGE_SZ (1<<13) then you won't be writing to all the pages you allocated, thus you can see in top that the memory consumption of the program goes down.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define PAGE_SZ (1<<12)
int main() {
int i;
int gb = 2; // memory to consume in GB
for (i = 0; i < ((unsigned long)gb<<30)/PAGE_SZ ; ++i) {
void *m = malloc(PAGE_SZ);
if (!m)
break;
memset(m, 0, 1);
}
printf("allocated %lu MB\n", ((unsigned long)i*PAGE_SZ)>>20);
getchar();
return 0;
}
A little known fact (though it is well documented) - you can (as root) prevent the OOM killer from claiming your process (or any other process) as one of its victims. Here is a snippet from something directly out of my editor, where I am (based on configuration data) locking all allocated memory to avoid being paged out and (optionally) telling the OOM killer not to bother me:
static int set_priority(nex_payload_t *p)
{
struct sched_param sched;
int maxpri, minpri;
FILE *fp;
int no_oom = -17;
if (p->cfg.lock_memory)
mlockall(MCL_CURRENT | MCL_FUTURE);
if (p->cfg.prevent_oom) {
fp = fopen("/proc/self/oom_adj", "w");
if (fp) {
/* Don't OOM me, Bro! */
fprintf(fp, "%d", no_oom);
fclose(fp);
}
}
I'm not showing what I'm doing with scheduler parameters as its not relevant to the question.
This will prevent the OOM killer from getting your process before it has a chance to produce the (in this case) desired effect. You will also, in effect, force most other processes to disk.
So, in short, to see fireworks really quickly...
Tell the OOM killer not to bother you
Lock your memory
Allocate and initialize (zero out) blocks in a never ending loop, or until malloc() fails
Be sure to look at ulimit as well, and run your tests as root.
The code I showed is part of a daemon that simply can not fail, it runs at a very high weight (selectively using the RR or FIFO scheduler) and can not (ever) be paged out.
Have a look at this program.
When there is no longer enough memory malloc starts returning 0
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(1)
{
printf("malloc %d\n", (int)malloc(1024*1024));
}
return 0;
}
On a 32-bit Linux system, the maximum that a single process can allocate in its address space is approximately 3Gb.
This means that it is unlikely that you'll exhaust the memory with a single process.
On the other hand, on 64-bit machine you can allocate as much as you like.
As others have noted, it is also necessary to initialise the memory otherwise it does not actually consume pages.
malloc will start giving an error if EITHER the OS has no virtual memory left OR the process is out of address space (or has insufficient to satisfy the requested allocation).
Linux's VM overcommit also affects exactly when this is and what happens, as others have noted.
I just exexuted #John La Rooy's snippet:
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(1)
{
printf("malloc %d\n", (int)malloc(1024*1024));
}
return 0;
}
but it exhausted my memory very fast and cause the system hanged so that I had to restart it.
So I recommend you change some code.
For example:
On my ubuntu 16.04 LTS the code below takes about 1.5 GB ram, physical memory consumption raised from 1.8 GB to 3.3 GB after executing and go down back to 1.8GiB after finishing execution.Though it looks like I have allocate 300GiB ram in the code.
#include <stdlib.h>
#include <stdio.h>
int main()
{
while(int i<300000)
{
printf("malloc %p\n", malloc(1024*1024));
i += 1;
}
return 0;
}
When index i is less then 100000(ie, allocate less than 100 GB), either physical or virtual memory are just very slightly used(less then 100MB), I don't know why, may be there is something to do with virtual memory.
One thing interesting is that when the physical memory begins to shrink, the addresses malloc() returns definitely changes, see picture link below.
I used malloc() and calloc(), seems that they behave similarily in occupying physical memory.
memory address number changes from 48 bits to 28 bits when physical memory begins shrinking
I was bored once and did this. Got this to eat up all memory and needed to force a reboot to get it working again.
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char** argv)
{
while(1)
{
malloc(1024 * 4);
fork();
}
}
If all you need is to stress the system, then there is stress tool, which does exactly what you want. It's available as a package for most distros.