I am working on a memory leak tool, the thing is: this tool should catch memory leaks only from test program, but what actually happens is, i have created a timer using API timer_create (POSIX), and this is somehow causing a leak of 144+56 bytes.
Any idea, as to how to stop it? How can i make sure, that all malloc requests from timer_create are not logged?
I am using the timer thread function method, and not signal. SIGEV_THREAD
I don't see any N in your reported memory leakage, just what appears to be a small constant, so my initial guess is that this is purely one-time overhead of setting up the timer thread system and not an actual memory leak. Try running your program with strace and make sure the timer is destroyed. If so, whatever internal memory is left is a matter of the implementation's quality and not a potential error in your program.
By the way, another good test approach: create 10 or 100 timers, then destroy them all, and compare the amount of memory "leaked". If it's the same as with one, I would say there's no issue.
Related
I've a question concerning the GLib.
I would like to use the GLib in a server context but I'm not aware on how the memory is managed:
https://developer.gnome.org/glib/stable/glib-Memory-Allocation.html
If any call to allocate memory fails, the application is terminated. This also means that there is no need to check if the call succeeded.
If I look at the source code, if g_malloc failed, it will call g_error:
g_error()
define g_error(...)
A convenience function/macro to log an error message.
Error messages are always fatal, resulting in a call to abort() to terminate the application.[...]
But in my case, as I'm developing a server application, I don't want the application exit, I would prefer, as the traditional malloc function, the GLib functions returns NULL or something to indicate an error happened.
So, my question is, there is a manner to handle out of memory?
Is the GLib not recommended for server purpose applications?
If I look at the man of abort I can see that I can handle the signal but I'll make the management of out-of-memory errors a little bit painful...
The abort() function causes abnormal program termination to occur, unless
the signal SIGABRT is being caught and the signal handler does not
return.
Thanks for you help!
It's very difficult to recover from lack of memory. The reason for that is that it can be considered a terminal state, in the sense that lack of memory will persist for some time before it goes away. Even reacting to the lack of memory (like informing the user) might require more memory, for example, to build and send a message. A related problem is that there are operating systems (linux at least) that may be over optimistic about allocating memory. When the kernel realizes that memory is missing, it may kill the application, even if your code is handling the failures.
So either you have a much stricter grasp of your whole system than average, or you won't be able to successfully handle out of memory errors, and, in this case, it doesn't matter what the helper library is doing.
If you really want to control memory allocation while still using glib, you have partial ways to do that. Don't use any glib allocation function and use some from other library. Glib provides functions that receive a "free function" when necessary. For example:
https://developer.gnome.org/glib/2.31/glib-Hash-Tables.html#g-hash-table-new-full
The hash table constructor accepts functions for destroying both keys and values. In your case, the data will be allocated using custom allocation functions, while the hash data structures will be allocated with glib functions.
Alternatively you could use g_try_* macros to allocate memory, so you still use glib allocator, but it won't abort on error. Again, this only partially solves the problem. Internally, glib will implicitly call functions that may abort and it assumes it will never return on error.
About the general question: does it make sense for a server to crash when it's out of memory ? The obvious answer is no, but I can't estimate how theoretical this answer is. I can only expect that the server system be properly sized for its operation and reject as invalid any input that could potentially exceed its capacities, and for that, it doesn't matter which libraries it might use.
I'm probably editorializing a bit here, but the modern tendency to use virtual/logical memory (both names have been used, although "logical" is more distinct) does dramatically complicate knowing when memory is exhausted, although I think one can restore the old, real-(RAM + swap) model (I'll call this the physical model) in Linux with the following in /etc/sysctl.d/10-no-overcommit.conf:
vm.overcommit_memory = 2
vm.overcommit_ratio = 100
This restores the ability to have the philosophy that if a program's malloc just failed, that program has a good chance of having been the actual cause of memory exhaustion, and can then back away from the construction of the current object, freeing memory along the way, possibly grumbling at the user for having asked for something crazy that needed too much RAM, and awaiting the next request. In this model, most OOM conditions resolve almost instantly - the program either copes and presumably returns RAM, or gets killed immediately on the following SEGV when it tries to use the 0 returned by malloc.
With the virtual/logical memory models that linux tends to default to in 2013, this doesn't work, since a program won't find memory isn't available at malloc, but instead upon attempting to access memory later at which point the kernel finally realizes there's nowhere in RAM for it. This amounts to disaster, since any program on the system can die, rather than the one the ran the host out of RAM. One can understand why some GLib folks don't even care about trying to fix this problem, because with the logical memory model, it can't be fixed.
The original point of logical memory was to allow huge programs using more than half the memory of the host to still be able to fork and exec supporting programs. It was typically enabled only on hosts with that particular usage pattern. Now in 2013 when a home workstation can have 24+ GiB of RAM, there's really no excuse to have logical memory enabled at all 99% of the time. It should probably be disabled by default on hosts with >4 GiB of RAM at boot.
Anyway. So if you want to take the old-school physical model approach, make sure your computer has it enabled, or there's no point to testing your malloc and realloc calls.
If you are in that model, remember that GLib wasn't really guided by the same philosophy (see http://code.google.com/p/chromium/issues/detail?id=51286#c27 for just how madly astray some of them are). Any library based on GLib might be infected with the same attitude as well. However, there may be some interesting things one can do with GLib in the physical memory model by emplacing your own memory handlers with g_mem_set_vtable(), since you might be able to poke around in program globals and reduce usage in a cache or the like to free up space, then retry the underlying malloc. However, that's limited in its own way by not knowing which object was under construction at the point your special handler is invoked.
I've written a debugger using ptrace(2) that is mainly for auditing system calls and redirecting standard IO of a child process. I would also like to detect memory leaks using this debugger.
I thought that it might be as easy as counting references to the system call brk(2), but it isn't. Unfortunately (or fortunately), Linux seems to call brk(2) at the end of the program regardless of whether or not the memory was properly freed.
I've seen this in a program that calls malloc(3) and free(3) and a program that just calls malloc(3) - they both have equal counts of brk(2) calls by the time the program has called exit_group(2), which is happens on return (perhaps I could be interpreting those results incorrectly?).
Or, perhaps exit_group(2) isn't equivalent to 'return' from main, and I should be setting a different break point for auditing the call count of brk(2).
I found a similar question here, but I still haven't found an answer.
I understand that Valgrind is a perfect tool for this, but it would cause considerable overhead.
Does anyone have helpful information on detecting memory leaks with ptrace(2)? Is possible with ptrace(2)? Is there a more practical way? Is there an API for memory debugging child processes?
Edit:
If there's other functions involved with allocate memory, I would count those too. On the page for malloc, it says that mmap(2) is also used in memory allocation. So, I would be counting that too.
Use gdb's heap extension. It will do what you want. IF you want to use it programatically, just pipe the results to your application to do post processing:
https://fedorahosted.org/gdb-heap/
I have read this-
Memory that is allocated by malloc(for example) and that is not freed
using free() function is released when the program terminates.And
that it is done by the opearting system. So when does having or not
having a garbage collector come into picture?
Or is it that not all operating systems do this automatic release of memory on program termination?
That claim about malloc and free is correct for all modern computing operating systems. But the statement as a whole reflects a complete misunderstanding of the purpose of garbage collection.
The reason you call free is not to clean things up for after your program terminates. The reason you call free is to permit the memory to be re-used during the subsequent execution of a long-running program.
Consider a message server that handles a hundred messages per second. You call malloc when you receive a new message. And then you have to do lots of things with it. You may have to log it. You may have to send it to other clients. You may have to write it to a database. When you are done, if you don't free it, after a few days you'll have millions of messages stuck in memory. So you have to call free.
But when do you call free? When one client is done sending a message, another client might still be using it. And maybe the database still needs it.
The purpose of garbage collection is to ensure that object's used memory is released (so it can be re-used to hold a new message during the application's lifetime) without having to burden the application programmer with the duty (and risks) associated with tracking exactly when the object is no longer required by any code that might be using it.
If an application doesn't run for very long or doesn't have any objects whose lifetimes are difficult to figure out, then garbage collection doesn't do very much good. And there are other techniques (such as reference-counted pointers) that can provide many of the same benefits as garbage collection. But there is a real problem that garbage collection does solve.
Most modern operating systems will indeed free everything you're allocated on program termination. However, a garbage collector will free unused memory before program termination. This allows your program to skip the frees, but still manage to keep allocating memory indefinitely, as long as it lets go to references to memory that isn't being used anymore, and as long as your total working set size doesn't exceed physical memory limits.
All OS do free memory when the program quits. Memory leaks are 'only' a problem because they waste memory on the machine, or cause the program to crash. So you'd garbage collect to prevent these things from happening, but without worrying about actually freeing your own pointers when you're done with them.
They're two solutions to the same problem, really. But, it's because the problem happens during runtime that you're worried about it.
Imagine a long running process like a web server that malloc()s a bunch of data structures for every connection it services. If it never free()s any of that memory, the process memory usage will continually grow, possibly consuming everything available on the system.
I often catch myself doing the following (in non-critical components):
some_small_struct *ptr=(some_small_struct *) malloc(sizeof(some_small_struct));
ptr->some_member= ...;
In words, I allocate dynamically memory for a small structure and I use it directly without checking the malloc'ed pointer. I understand there is always a chance that the program won't get the memory it asks for (duh!) but consider the following:
If the program can't even get some memory for a small structure off the
heap, maybe there are much bigger problems looming and it doesn't matter after all.
Furthermore, what if handling the null pointer exacerbates the precarious situation even more?? (e.g. trying to log the condition calls even more non-existing resources etc.)
Is my reasoning sane (enough) ?
Updated:
A "safe_malloc" function can be useful when debugging and might be useful otherwise
+X access can hide the root cause of a NULL pointer
On Linux, "optimistic memory allocation" can shadow loomin OOM (Out-Of-Memory) conditions
Depends on the platform. For instance, on Linux (by default) it does not make much sense to check for NULL:
http://linux.die.net/man/3/malloc
By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available. This is a really bad bug. In case it turns out that the system is out of memory, one or more processes will be killed by the infamous OOM killer.
In the case of C, it depends on the platform. If you are on an embedded platform with very little memory, you should alweays check, thouggh what you do if it does fail is more difficult to say. On a modern 32-bit OS with virtual memory, the system will probably become unresponsive and crash before it admits to running out of memory. In this case, the call to malloc never returns, so the utility of checking its value becomes moot.
In the case of C++, you should be using new instead of malloc, in which case an exception will be raised on exhaustion, so there is no point in checking the return value.
I would say No.
Using a NULL pointer is going to crash the program (probably).
But detecting it and doing something intelligent will be OK and you may be able to recover from the low memory situation.
If you are doing a big operation set some global error flag and start unwinding the stack and releasing resources. Hopefully one or more of these resources will be your memory hog and your application will get back to normal.
This of course is a C problem and handeled automatically in C++ with the help of exceptions and RAII.
As new will not return NULL there is no point in checking.
Allocations can fail for several reasons. What you do (and can do) about it depends in part on the allocation failure.
Being truly out of memory is catastrophic. Unless you've made a careful plan for this, there's probably nothing you can do. (For example, you could have pre-allocated all the resources you'd need for an emergency save and shutdown.)
But many allocation failures have nothing to do with being out of memory. Fragmentation can cause an allocation to fail because there's not enough contiguous space available even though there's plenty of memory free. The question specifically said a "small structure", so this is probably as bad as true out-of-memory condition. (But code is ever-changing. What's a small structure today might be a monster tomorrow. And if it's so small, do you really need memory from the heap or can you get it from the stack?)
In a multi-threaded world, allocation failures are often transient conditions. Your modest allocation might fail this microsecond, but perhaps a memory-hogging thread is about to release a big buffer. So a recovery strategy might involve a delay and retry.
How (and if) you handle allocation failure can also depend on the type of application. If you're writing a complex document editor, and a crash means losing user's work, then it's worth expending more effort to handle these failures. If your application is transactional, and each change is incrementally applied to persistent storage, then a crash is only a minor inconvenience to the user. Even so, logging should be considered. If you're application is routinely getting allocation failures, you probably have a bug, and you'll need the logs to know about it and track it down.
Lastly, you have to think about testing. Allocation failures are rare, so the chance that recovery code has been exercised in your testing is vanishingly small--unless you've taken steps to ensure test coverage by artificially forcing failures. If you aren't going to test your recovery code, then it's probably not worth writing it.
at the very least I would put an assert(ptr != NULL) in there so you get a meaningful error.
Furthermore, what if handling the null pointer exacerbates the precarious situation even more??
I do not see why it can exacerbate the situation.
Anyway, when writing code for windows ptr->some_member will throw access violation so you will immediately see the problem, therefore I see no reason to check the return value, unless your program has some opportunity to free the memory.
For platforms that do not handle null-pointers in a good way(throwing exception) it is dangerous to ignore such points.
Assuming that you are running on a Linux/MaxOs/Windows or other virtual memory system, then... the only reason to check the return value from malloc is if you have a strategy for freeing up enough memory to allow the program to continue running. An informative message will help in diagnosing the problem, but only if your program caused the out-of-memory situation.
Usually it is not your program and the only thing that your program can to do help is to exit as quickly as possible.
assert(ptr != NULL);
will do all of these things. My usual strategy is to have a layer around malloc that has
this in it.
void *my_malloc(size_t size)
{
void *ptr = malloc ( size );
assert(ptr != NULL);
return *ptr;
}
Then you call my_malloc instead of malloc. During development I use a memory allocation library that is conducive to debugging. After that if it runs out of memory - I get a message.
Yes, having insufficient memeory will almost certatinly presage other failures coming soon. But how sure are you that no corrupt output will occur between the failure to allocate and the final crash?
How sure are you for every program, every time you make an edit.
Catch your errors so you can know you crashed on time.
It is possible to allocate a largish chunk of memory at startup that you can free when you hit an out of memory condition and use that to shut down gracefully.
I always feel it is important and best to handle the return of malloc or any other system call for that matter. Though in modern systems (apart from embedded ones) it's a rare scenario unless and until your code uses too much memory, it's always safer.
Continuing the code after a system call failure can lead to corruption, crash and what not apart from making your program look bad.
Also, in linux, memory allocated to a process is limited. Try creating 1000 threads in a process and allocate some memory in each one of them, then you can easily simulate the low memory condition. : )
Always better to check for sys call return values!
I am working on a parallel app (C, pthread). I traced the system calls because at some point I have bad parallel performances. My traces shown that my program calls mprotect() many many times ... enough to significantly slow down my program.
I do allocate a lot of memory (with malloc()) but there is only a reasonable number of calls to brk() to increase the heap size. So why so many calls to mprotect() ?!
Are you creating and destroying lots of threads?
Most pthread implementations will add a "guard page" when allocating a threads stack. It's an access protected memory page used to detect stack overflows. I'd expect at least one call to mprotect each time a thread is created or terminated to (un)protect the guard page. If this is the case, there are several obvious strategies:
Set the guard page size to zero using pthread_attr_setguardsize() before creating threads.
Use a thread-pool (of as many threads as processors say). Once a thread is done with a task, return it to the pool to get a new task rather than terminate and create a new thread.
Another explanation might be that you're on a platform where a thread's stack will be grown if overflow is detected. I don't think this is implemented on Linux with GCC/Glibc as yet, but there have been some proposals along these lines recently. If you use a lot of stack space whilst processing, you might explicitely increase the initial/minimum stack size using pthread_attr_setstacksize.
Or it might be something else entirely!
If you can, run your program under a debug libc and break on mprotect(). Look at the call stack, see what your code is doing that's leading to the mprotect() calls.
glibc library that has ptmalloc2 for its malloc uses mprotect() internally for micromanagement of heap for threads other than main thread (for main thread, sbrk() is used instead.) malloc() firstly allocates large chunk of memory with mmap() for the thread if a heap area seems to have contention, and then it changes the protection bits of unnecessary portion to make it accessible with mprotect(). Later, when it needs to grow the heap, it changes the protection to read/writable with mprotect() again. Those mprotect() calls are for heap growth and shrink in multithreaded applications.
http://www.blackhat.com/presentations/bh-usa-07/Ferguson/Whitepaper/bh-usa-07-ferguson-WP.pdf
explains this in a bit more detailed way.
The 'valgrind' suite has a tool called 'callgrind' that will tell you what is calling what. If you run the application under 'callgrind', you can then view the resulting profile data with 'kcachegrind' (it can analyze profiles made by 'cachegrind' or 'callgrind'). Then just double-click on 'mprotect' in the left pane and it will show you what code is calling it and how many times.