I know this is really general, but I get "this" (see below) when I run my .c file in Visual C++ 2008 Express. It happens when I call malloc (). Take my work on this - I dynamically allocate memory properly.
HEAP[Code.exe]: HEAP: Free Heap block 211a10 modified at 211af8 after it was freed
Windows has triggered a breakpoint in Code.exe.
This may be due to a corruption of the heap, which indicates a bug in Code.exe or any of the DLLs it has loaded.
This may also be due to the user pressing F12 while Code.exe has focus.
The output window may have more diagnostic information.
Why do I get this error? What does this even mean?
The error message tells you exactly why you got it:
Free Heap block 211a10 modified at 211af8 after it was freed
You had a heap allocated block that was freed then something wrote to that area of memory. It's not nice to write to a freed block of memory.
The error isn't actually happening when you call malloc; that's just when it triggers a free heap scan. The actual error happened somewhere before. You malloced some memory at address 211a10 (that's what malloc returned to you). Then you (or some other lib) freed it. Then later, when you call malloc in debug mode, it scans the whole heap -- as a courtesy to you, the poor programmer. It discovers that somebody (your or some lib you call) wrote over part of that array, specifically at address 211af8, or 0xe8 bytes into the array. So you're either still hanging onto a pointer that's been freed (most likely) and using it, or you're just trashing random memory.
In my case, with similar symptoms, the issue was the struct alignment mismatch (/Zp option)
I defined for my code a different struct alignment than external libraries (wxWidgets).
However, wxWidgets was built with the makefile, so it was compiled using the defaut /Zp.
And wxWidget is statically linked.
You can do that, but if you try to delete a wxWidgets-class object from your code the compiler becomes confused about the exact size of struct members.
And when running, you get this message:
HEAP[Code.exe]: HEAP: Free Heap block 211a10 modified at 211af8 after it was freed
Windows has triggered a breakpoint in Code.exe.
Solution:
Be sure to use the same "Struct Member Alignment" in all code and libraries.
Best rule is to define /ZP to use "default" value.
In Visual Studio, under Properties C/C++ Code Generation
MSDN cite: "You should not use this option unless you have specific alignment requirements."
See here
Tip: use #pragma pack if you need to control the alignment in some structs
See there
Example:
#pragma pack(1) // - 1 byte alignment
typedef union
{
u64 i;
struct{ // CUSTOMS.s is used by Folders
u32 uidx; // Id, as registered
byte isoS, isoT; // isoS/isoT combination.
byte udd0, udd1; // custom values (TBD)
}s;
}CUSTOMS;
struct Header // exactly 128 bits
{
u32 version;
u32 stamp; // creation time
CUSTOMS customs; // properties
}
#pragma pack() // this pragma restores the **default** alignment
*
Hope this explanation helps, because this is not actually a bug in code, but a serious configuration mistake: difficult to detect because it is located in subtle compiler options. Thanks for all,
*
I dynamically allocate memory properly.
I think that the problem here is that you unallocate the memory inproperly. What I mean by this is that, you might be trying to use freed memory. Sorry I can't help any further, you could probably add the actual code.
Take my work on this - I dynamically allocate memory properly.
But are you sure your buffers are all of the correct size and you free() them properly? Double frees and buffer overflows can easily lead to heap corruption that can cause malloc() to fail in all kind of ways.
If the management structures used internally by malloc() get damaged, it will usually not lead to an error immediately. But later calls to malloc() or free() that try to use these damaged structures will fail do erratic things.
Are you using malloc() on an array? Because I think the error just might be you forgetting to allocate an extra memory location at the end -- what happens is it tries to write to that location, which isn't allocated to it, and assumes it's trying to write to a place that has already been freed.
Related
I have a large body of legacy code that I inherited. It has worked fine until now. Suddenly at a customer trial that I cannot reproduce inhouse, it crashes in malloc. I think that I need to add instrumentation e.g on top of malloc I have my own malloc that stores some meta information about each malloc e.g. who has made the malloc call. When it crashes, I can then look up the meta information and see what was happening. I had done something similar years ago but cannot recall it now...I am sure people have come up with better ideas. Will be glad to have inputs.
Thanks
Is memory allocation broken?
Try valgrind.
Malloc is still crashing.
Okay, I'm going to have to assume that you mean SIGSEGV (segmentation fault) is firing in malloc. This is usually caused by heap corruption. Heap corruption, that itself does not cause a segmentation fault, is usually the result of an array access outside of the array's bounds. This is usually nowhere near the point where you call malloc.
malloc stores a small header of information "in front of" the memory block that it returns to you. This information usually contains the size of the block and a pointer to the next block. Needless to say, changing either of these will cause problems. Usually, the next-block pointer is changed to an invalid address, and the next time malloc is called, it eventually dereferences the bad pointer and segmentation faults. Or it doesn't and starts interpreting random memory as part of the heap. Eventually its luck runs out.
Note that free can have the same thing happen, if the block being released or the free block list is messed up.
How you catch this kind of error depends entirely on how you access the memory that malloc returns. A malloc of a single struct usually isn't a problem; it's malloc of arrays that usually gets you. Using a negative (-1 or -2) index will usually give you the block header for your current block, and indexing past the array end can give you the header of the next block. Both are valid memory locations, so there will be no segmentation fault.
So the first thing to try is range checking. You mention that this appeared at the customer's site; maybe it's because the data set they are working with is much larger, or that the input data is corrupt (e.g. it says to allocate 100 elements and then initializes 101), or they are performing things in a different order (which hides the bug in your in-house testing), or doing something you haven't tested. It's hard to say without more specifics. You should consider writing something to sanity check your input data.
Try Asan
AddressSanitizer (aka ASan) is a memory error detector for C/C++. It finds:
Use after free (dangling pointer dereference)
Heap buffer overflow
Stack buffer overflow
Global buffer overflow
Use after return
Use after scope
Initialization order bugs
Memory leaks
Please find the links to know more and how to use it
https://github.com/google/sanitizers/wiki/AddressSanitizer and
https://github.com/google/sanitizers/wiki/AddressSanitizerFlags
I know this is old, but issues like this will continue to exist as long as we have pointers. Although valgrind is the best tool for this purpose, it has a steep learning curve and often the results are too intimidating to understand.
Assuming you are working on some *nux, another tool I can suggest is electricfence. Quote:
Electric Fence helps you detect two common programming bugs:
software that overruns the boundaries of a malloc() memory allocation,
software that touches a memory allocation that has been released by free().
Unlike other malloc() debuggers, Electric Fence will detect read accesses
as well as writes, and it will pinpoint the exact instruction that causes
an error.
Usage is amazingly simple. Just link your code with an additional library lefence
When you run the application, a corefile will be generated when memory is corrupted, instead of when corrupted memory is used.
Interviewer - If you have no tools to check how would you detect memory leak problems?
Answer - I will read the code and see if all the memory I have allocated has been freed by me in the code itself.
Interviewer wasn't satisfied. Is there any other way to do so?
For all the implementation defined below, one needs to write wrappers for malloc() & free() functions.
To keep things simple, keep track of count of malloc() & free(). If not equal then you have a memory leak.
A better version would be to keep track of the addresses malloc()'ed & free()'ed this way you can identify which addresses are malloc()'ed but not free()'ed. But this again, won't help much either, since you can't relate the addresses to source code, especially it becomes a challenge when you have a large source code.
So here, you can add one more feature to it. For eg, I wrote a similar tool for FreeBSD Kernel, you can modify the malloc() call to store the module/file information (give each module/file a no, you can #define it in some header), the stack trace of the function calls leading to this malloc() and store it in a data structure, along side the above information whenever a malloc() or free() is called. Use addresses returned by malloc() to match with it free(). So, when their's a memory leak, you have information about what addresses were not free()'ed in which file, what were the exact functions called (through the stack trace) to pin point it.
The way, this tool worked was, on a crash, I used to get a core-dump. I had defined globals (this data structure where I was collecting data) in kernel memory space, which I could access using gdb and retrieve the information.
Edit:
Recently while debugging a memeory leak in linux kernel, I came across this tool called kmemleak which implements a similar algorithm I described in point#3 above. Read under the Basic Algorithm section here: https://www.kernel.org/doc/Documentation/kmemleak.txt
My response when I had to do this for real was to build tools... a debugging heap layer, wrapped around the C heap, and macros to switch code to running against those calls rather than accessing the normal heap library directly. That layer included some fencepost logic to detect array bounds violations, some instrumentation to monitor what the heap was doing, optionally some recordkeeping of exactly who allocated and freed each block...
Another approach, of course, is "divide and conquer". Build unit tests to try to narrow down which operations are causing the leak, then to subdivide that code further.
Depending on what "no tools" means, core dumps are also sometimes useful; seeing the content of the heap may tell you what's being leaked, for example.
And so on....
In C, I encountered an error when I coded the following example:
int *pointer;
int i = 0;
pointer = malloc(10 * sizeof(int));
pointer[i - 1] = 4;
Clearly i is a negative index into pointer.
Even if the incorrect piece memory was altered, why was an error only triggered upon free(pointer) [later on in the code]? The error was: double free or corruption (out).
EDIT: The compiler used was gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Most if not all memory managers (eg: malloc) allocate extra data around the memory you've requested. This extra is to help the memory manager manage the various allocations by adding some of its own data to the allocation. When you did your negative index, you overwrote this extra data. It's not invalid memory per se, so the CPU can't protect against it, but malloc will indeed complain when it sees that its data has been borked. That will occur on free().
Fundamentally, accessing pointer[-1] the way you did yields Undefined Behavior; in theory, anything could happen. But in practice, there’s a good explanation for the particular error message you got.
Recall that malloc has to save the size of the allocated chunk of memory somewhere so that free can work correctly. On some systems — and it seems yours is among that bunch — this information is stored in memory just before the address that malloc returns. (It might be interesting to print out that value, if you want to learn about your system’s implementation.)
When you overwrote that chunk of memory, you left the internals of the malloc-free system in an inconsistent state, leading free to report an error when this was discovered. The most likely time for such a discovery, of course, is when the memory management functions are called about this area of memory. (Some malloc corruptions will be detected the next time you call malloc, but that usually takes more deliberate effort to achieve.) Calling realloc should give you a similar error message.
Earlier I encountered a problem with dynamic memory in C (visual studio) .
I had a more or less working program that threw a run-time error when freeing one of the buffers. It was a clear memory corruption, the program wrote over the end of the buffer.
My problem is, that it was very time consuming to track down. The error was thrown way down after the corruption, and i had to manually debug the entire run to find when is the buffer end overwritten.
Is there any tool\ way to assist in tracking down this issue? if the program would have crashed immediately i would have found the problem a lot faster...
an example of the issue:
int *pNum = malloc(10 * sizeof(int));
// ||
// \/
for(int i = 0; i < 13; i++)
{
pNum[i] = 3;
}
// error....
free(pNum);
I use "data breakpoints" for that. In your case, when the program crashes, it might first complain like this:
Heap block at 00397848 modified at 0039789C past requested size of 4c
Then, start your program again, and set a data breakpoint at address 0039789C. When the code writes to that address, the execution will stop. It often happens that i find the bug immediately at this point.
If your program allocates and deallocates memory repeatedly, and it happens to be at this exact address, just disable deallocations:
_CrtSetDbgFlag(_CrtSetDbgFlag(_CRTDBG_REPORT_FLAG) | _CRTDBG_DELAY_FREE_MEM_DF);
I use pageheap. This is a tool from Microsoft that changes how the allocator works. With pageheap on, when you call malloc, the allocation is rounded up to the nearest page(a block of memory), and an additional page of virtual memory that is set to no-read/no-write is placed after it. The dynamic memory you allocate is aligned so that the end of your buffer is just before the end of the page before the virtual page. This way, if you go over the edge of your buffer, often by a single byte, the debugger can catch it easily.
Is there any tool\ way to assist in tracking down this issue?
Yes, that's precisely the type of error which static code analysers try to locate. e.g. splint/PC-Lint
Here's a list of such tools:
http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis
Edit: In trying out splint on your code snippet I get the following warning:
main.c:9:2: Possible out-of-bounds store: pnum[i]
Presumably this warning would have assisted you.
Our CheckPointer tool can help find memory management errors. It works with GCC 3/4 and Microsoft dialects of C.
Many dynamic checkers only catch accesses outside of an object, and then only if the object is heap allocated. CheckPointer will find memory access errors inside a heap-allocated object; it is illegal to access off the end of a field in a struct regardless of the field type; most dynamic checkers cannot detect such errors. It will also find accesses off the edge of locals.
I have one C code app. which i was building using MS-VS2005. I had one output data buffer which was being allocated dynamically using malloc.
For some test cases, the memory size which was being malloc'd was falling short than the the actual output size in bytes which was generated. That larger sized output was written into the smaller sized buffer causing buffer overflow. As a result of which the test-run was crashing with MSVS-2005 showing up a window "Heap corruption ...."
I knew it had to do with some dynamic memory allocation, but i took long time to actually find the root cause, as i did not doubt the memory allocation because i was allocating large enough size necessary for the output. But one particular test case was generating more output than what i had calculated, hence the resulting crash.
My question is:
1.) What tools i can use to detect such dynamic memory buffer over-flow conditions. Can they also help detect any buffer overflow conditions(irrespective of whether the buffer/array is on heap, stack, global memory area)?
2.) Will memory leak tools(like say Purify) or code analysis tools like lint, klocworks would have helped in particular case? I believe they have to be run time analysis tools.
Thank you.
-AD.
One solution, which I first encountered in the book Writing Solid Code, is to "wrap" the malloc() API with diagnostic code.
First, the diagnostic malloc() arranges to allocate additional bytes for a trailing sentinel. For example, an additional four bytes following the allocated memory are reserved and contain the characters 'FINE'.
Later, when the pointer from malloc() is passed to free(), a corresponding diagnostic version of free() is called. Before calling the standard implementation of free() and relinquishing the memory, the trailing sentinel is verified; it should be unmodified. If the sentinel is modified, then the block pointer has been misused at some point subsequent to being returned from the diagnostic malloc().
There are advantages of using a memory protection guard page rather than a sentinel pattern for detecting buffer overflows. In particular, with a pattern-based method, the illegal memory access is detected only after the fact. Only illegal writes are detected by the sentinel pattern method. The memory protection method catches both illegal reads and writes, and they are detected immediately as they occur.
Diagnostic wrapper functions for malloc() can also address other misuses of malloc(), such as multiple calls to free() for same memory block. Also, realloc() can be modified to always move blocks when executed in a debugging environment, to test the callers of realloc().
In particular, the diagnostic wrappers may record all of the blocks allocated and freed, and report on memory leaks when the program exits. Memory leaks are blocks which are allocated by not freed during the program execution.
When wrapping the malloc() API, one must wrap all of the related functions, including calloc(), realloc(), and strdup().
The typical way of wrapping these functions is via preprocessor macros:
#define malloc(s) diagnostic_malloc(s, __FILE__, __LINE__)
/* etc.... */
If the need arises to code a call to the standard implementation (for example, the allocated block will be passed to a third-party, binary-only library which expects to free the block using the standard free() implementation, the original function names can be accessed rather than the preprocessor macro by using (malloc)(s) -- that is, place parentheses around the function name.
Something you can try is allocate enough pages + 1 using VirtualAlloc, use VirtualProtect with PAGE_READONLY | PAGE_GUARD flags on the last page, then align the suspected allocation so the end of the object is near the beginning of the protected page. If all goes well you should get an access violation when the guard page is accessed. It helps if you know approximately which allocation is overwritten. Otherwise it requires overriding all allocations which may require a lot of extra memory (at least 2 pages per allocation). A variation on this technique that I'm hereby christening as "statistical page guard" is to only randomly allocate memory for a relatively small percentage of allocations in that manner to avoid large bloat for small objects. Over a large number of execution runs you should be able to hit the error. The random number generator would have to be seeded off something like time in this case. Similarly you can allocate the guard page in front of the object if you suspect an overwrite at a lower address (can't do both at the same time but possible to randomly mix up as well).
An update: it turns out the gflags.exe (used to be pageheap.exe) microsoft utility already supports "statistical page guard" so i reinvented the wheel :) All you need to do is run gflags.exe /p /enable [/random 0-100] YourApplication.exe and run your app. If you are using a custom heap or custom guards on your heap allocations then you can simply switch to using HeapAlloc at least for catching bugs and then switch back. Gflags.exe is a part of Support Tools package and can be downloaded from the microsoft download center, just do a search there.
PC-Lint can catch some forms of malloc/new size problems, but I'm not sure if it would have found yours.
VS2005 has good buffer overflow checking for stack objects in debug mode (runs at end of function). And it does periodic checking of the heap.
As for it helping to track down where the problems occurred, this is where I tend to start using macro's to dump all allocations to match against the corrupted memory later (when it's detected).
Painful process, so I'm keen to learn better ways also.
Consider our Memory Safety Check. I think it will catch all the errors you describe. Yes, it is runtime checking of every access, with some considerable overhead (not as bad as valgrind we think) with the benefit of diagnosing the first program action that is errorneous.