I'm running a C program which is occupying large physical memory and I want to debug where those large allocations are happening. This program has multiple forks and one of those forks is shooting the RES memory to 11.0G as you can see below (cropped screenshot of htop).
My question is in general how to debug those large memory allocation without reading through the code for malloc/calloc etc.?
Right now I'm looking at the core dump through gdb but this bt output doesn't tell the picture where those allocations are happening.
I was looking for something similar to dotnet dump analyse or PerfView.
My question is in general how to debug those large memory allocation without reading through the code for malloc/calloc etc.?
Reading through malloc etc. will not help with your original question.
You want to use a heap profiler. TCmalloc has support for heap profiling, which can tell you a lot about your application's heap use. There are other heap profilers as well.
This article may be helpful as well.
P.S. Extracting heap profiling info from an existing core dump is possible, but hard. And you are unlikely to find out the allocation stack trace(s) anyway.
Your best bet is to set up heap profiler and reproduce the problem, rather than trying to recover the info post mortem.
Related
I'm working with embedded linux development, and we're currently having some trouble with some memory page allocation faults, which led me to believe we have a leak somewhere.
Currently, I am trying to cross compile valgrind to use on our system, but I'm losing my faith on this solution because of the sheer amount of memory valgrind will use up (we have serious memory restrictions).
This has made me wonder: is there any way of hunting for a memory leak without valgrind or with a valgrind-like application with minimal memory usage? Creating wrappers for malloc() and free() is out of question.
Also, the test that caused the allocation failures was a simple stress test of copying a file n times and checking its md5sum, in case anyone is curious.
I'm using the Linaro toolchain for cross compiling, glibc 2.15, and the system is set up without a swap partition. The system has around 64MB of RAM, making valgrind, or any other memory intensive application a tad difficult to use.
Regards,
Guilherme
Since you are using glibc, you should have its built-in memory-tracing support available to you. Your program would enable this by calling mtrace(3) at or near startup. mtrace() installs hook functions into the memory allocator to log allocations and deallocations, under runtime control via environment variable MALLOC_TRACE.
You probably also want to be aware of mtrace(1), a Perl script for interpreting log files produced by the mtrace facility.
This facility traces only allocations and deallocations, which is much less than Valgrind does. Nevertheless, those are the main items of interest when you are looking for a memory leak.
I am using the valgrind's tool memcheck together with GDB trying to track down a memory leak in my C code.
Valgrind found where the memory was allocated but the pointer to this memory is tossed around a lot in the code before it is free'd. About 99 % of the time this is free'd correctly but sometimes it isn't. Can Valgrind, in some way, help me to find exactly where I lose control of this pointer or do I need to look into this extremely complex code to find it by myself?
Edit: If it is not possible for Valgrind to give me this information I would be happy if someone could tell me why. I mean Valgrind does have the information about where the memory is allocated and if I stop the program with GDB and ask for a leak check it will know if I have lost this the pointer to this memory. Why can't valgrind "watch" this pointer and tell me exactly when I lose it?
The Valgrind Omega tool does this: it shows where the last pointer to your leaked memory is lost. Unfortunately the tool is disabled since Valgrind 3.4:
exp-Omega, an experimental instantaneous leak-detecting tool, was
disabled in 3.4.0 due to a lack of interest and maintenance, although
the source code was still in the distribution. The source
code has now been removed from the distribution.
So you'd have to download and build Valgrind 3.3, and then you can run valgrind --tool=exp-omega.
Links:
Homepage of Valgrind Omega
Old Valgrind releases
looks like the 3.3.0 release isn't available from that site any more, but Google found this one (md5 sum matches)
I am trying to complete a college assignment, and the marking criteria specifies 5% for memory management - specifically for having no memory leaks.
As I understand it, memory leaks in simple C programs, are only caused by pointers which have become abandoned by the program - ie, malloc/calloc/etc calls which are never have a corresponding free.
My question is in 3 parts:
Whats the simplest way on Solaris
and OSX to 'prove' that you haven't
leaked any memory?
Does XCode have
any tools to help determine memory
leaks?
Does the operating system
release all previously allocated
memory within a c program once the
process ends?
Valgrind is your friend.
For every malloc(), you need to ensure that you have exactly one free().
I haven't worked with XCode, but this forum entry may help.
Yes. It's still poor form to let your running program 'leak,' however.
In general, it's a good idea to learn how to avoid leaks without using tools like a memory debugger (early on) -- especially for your simple programs. It's painful, however: when it comes to building anything non-trivial you'll want to start learning how to use the more advanced debugging tools (like Valgrind, as Alex Reynolds suggested in another answer.)
Answer for Mac OS and an example to be avoided (saved you half an hour).
Mac OS doesn't come with Valgrind or dmalloc. Moreover, Valgrind has some compatibility issues when trying to get it installed in Sierra.
There is utility called "leaks", which I get it running by this:
leaks -atExit --/Contents/Developer/usr/lib/libLeaksAtExit.dylib ./a.out
Unfortunately, this doesn't report obvious memory leaks... Maybe I am using it wrong, but I was just searching for an easy way to check that my C program free'd its memory as it should.
If you have time, then maybe read and use Using OSX Leaks for C Programs on the Command Line?
Resources:
Finding Memory Leaks
Using the "leaks" command on a C/C++ executable
PS: Maybe if used with "iprofiler", then it might be useful, but I didn't had it installed.
there's dmalloc too
So, I've recently noticed that our development server has a steady ~300MB out of 4GB ram left after the finished development of a certain project. Assuming this was due to memory leaks during the development phase, will that memory eventually free itself up or will it require a server restart. Are there any tools that can be used to prevent this in the future (aside from the obvious, 'don't write code that produces memory leaks')? Sometimes they go unseen for a little while and over time I guess they add up as you continue testing your app.
What operating system are you running? Most operating systems these days will clean up leaked memory for a process when the process exits. It is possible that the memory you are seeing in use is actually being used for the filesystem cache. This is nothing to worry about -- the OS will reclaim this memory if necessary.
From: http://learnlinux.tsf.org.za/courses/build/internals/ch05.html
The amount of free memory indicated by
the free command includes the current
size of the buffer cache in its
calculation. This is misleading, as
the amount of free memory indicated
will often be very low, as the buffer
cache soon fills most of user memory.
Don't' panic. Applications are
probably not crowding your RAM; it is
merely the buffer cache that is taking
up all available space. The buffer
cache counts as memory space available
for application use (remembering that
it will be shrunk as required), so
subtract the size of the buffer cache
to see the real amount of free memory
available for application use
It's best to fight them during development, because then it's easier to identify the revision that introduces the leak. As you probably see now, doing it after the fact is very, very hard. Expect a lot of reports when running the tools I recommend below:
http://valgrind.org/
http://www.ibm.com/software/awdtools/purify/
http://directory.fsf.org/project/ElectricFence/
I'd suggest you to run this tools, suppress most warnings about leaks, and then fix them one by one, removing the suppresions.
And then, make sure you regularly run these tools and quickly fix any regressions!
Of course the obvious answer is "Don't write code that produces memory leaks" and it's a valid one, because they can be extremely hard to fix if you have reference counting issues, or complex code in which it's hard to track the lifetime of memory.
To address your current situation you might consider using a tool such as DevPartner for Windows, or Valgrind for Linux/Unix, both of which I've found to be very effective for tracking down memory leaks (as well as other issues such as performance bottlenecks).
Another thing you may wish to consider is to look at your use of pointers and slowly replace them with smart pointers if you can, which should help manage your pointer lifetimes.
And no, I doubt that memory is going to be recovered without restarting the process in which your code is running.
Run the program using the exceptional valgrind on Linux x86 boxes.
A commerical equivilant, Purify, is available on Windows.
These runtime analysis of your program will report memory leaks and other errors such as buffer overflows and unitialised variables.
Static code analysis - Lint and Coverity for example - can also uncover memory leaks and more serious errors.
Lets be specific about what memory leaks cause and how they harm your program:
If you 'leak' memory during operation of your program there is a risk that your application will eventually exhaust RAM and swap, or the address space of available to your program (which can be less than physical RAM) and cause the next allocation to fail. The vast majority of programs will fail to catch this error, as error checking is harder than it seems. The majority of programs will either fail by dereferencing a null pointer or will exit.
If this is on Linux, check the output of 'free' and specifically check the amount of 'cached' ram. If your development work includes a lot of disk I/O, it'll use it for caching files, and you'll see very little 'available' but it's still there if it's needed. For all practical purposes, consider free+cached as available.
The 'free' output is distilled from /proc/meminfo, and you can get more detailed information on the running process in /proc/$pid/{maps,smaps}
In theory when your process exits, any memory it had is released. Is your process exiting?
Don't assume anything, run a memory profiler over it and see what it's doing.
When I was at college we used the Borland C++ Builder 6 IDE
It included CodeGuard, which checks for memory leaks and other memory related issues.
I am not sure if this option is still available on newer versions, but it would be weird for a new version to have less features.
On linux, as mentioned before, valgrind is a good memory leak debugger.
Very general:
Is there an easy way to tell which line of code last freed a block of memory when an access violation occurs?
Less general:
My understanding of profilers is that they override the allocation and deallocation processes. If this is true, might they happen to store the line of code that last freed a section of memory so that when it later crashes because of an access violation, you know what freed it last?
Specifics:
Windows, ANSI C, using Visual Studio
Yes!
Install the Windows Debugging Tools and use Application Verifier.
File -> Add Application, select your .exe
Under Basics, select Memory and Heaps.
Run the debug build of your program under ntsd (ntsd yourprogram.exe).
Reproduce the bug.
Now when you make the crash happen, you will get additional information in the debugger from AppVerifier. Use !avrf (may take a long time to run (minutes)) and it will try to give you as much useful information as possible.
You can all use the dps command on the memory address to get all the stored stack info (allocation, deallocation, etc).
You can also use the !heap command on the memory address:
0:004> !heap -p -a 0x0C46CFE0
Which will dump information as well.
Further Reading:
Advanced Windows Debugging, Hewardt and Pravat
Debugging with PageHeap
Short answer: no.
What you need is a debug malloc. I don't keep up with Windows any longer but there are several about, including this free one.
Update
Looks like Visual Studio C has a built in version. See here
When the application is linked with a
debug version of the C run-time
libraries, malloc resolves to
_malloc_dbg. For more information about how the heap is managed during
the debugging process, see The CRT
Debug Heap.
... and see here for _malloc_dbg.
No, not unless you provide your own allocators (e.g. by overloading new/delete) to store this information.
What profilers do is highly dependent on what they're profiling. I'm not aware of any profiler that tracks what you're looking for.
Perhaps if you provided more details on your situation people could suggest an alternative means of diagnosing the problem you're encountering.