I have a number of data structures (trees, queues, lists), created using dynamic allocation routines (malloc, calloc). Under some critical conditions, the program should terminate. Traversing all objects to free their memory takes long time.
Is it safe to avoid traversing all data structures just before program stops? If yes, does it apply to all operating systems and environments (e.g. multiple threads)?
All the memory dynamically allocated by a process is released back to the OS on process termination, doesn't matter intentionally or via a crash. Same happens with files and sockets - ref counts inside the kernel get decremented and resources get released if there are no more references.
An exception to the above might be the shared memory.
When a program (i.e., a process) terminates, all local and heap memories are automatically reclaimed. Note that these memory regions are specific to a process. So, you may skip the traverse and deallocation just before the program termination. However, if the program uses a shared/global memory, then you need to explicitly reclaim that. Finally, it applies, at least, Linux/Unix and Windows. I believe it applies to all modern operating systems.
Short answer: yes. In any modern OS, memory is private to each process, and once the process exits, all memory is reclaimed by the OS (unless the OS itself is broken).
You don't have to free() all your dynamically-allocated memory before terminating the program. The operating system releases all the memory that was owned by the process anyway. It also closes any network connections that you had open.
Related
Is a thread dynamically allocated memory?
I have been researching and have a fair understanding of threads and how they are used. I have specifically looked at the POSIX API for threads.
I am trying to understand thread creation and how it differs from a simple malloc call.
I understand that threads share certain memory segments with the parent process, but it has it's own stack.
Any resources I can read through on this topic is appreciated. Thanks!
Thread creation and a malloc() call are completely different concepts. A malloc() call dynamically allocates the requested byte chunk of memory from the heap for the use of the program.
Whereas a thread can be considered as a 'light-weight process'. The thread is an entity within a process and every process will have atleast one thread to help complete its execution. The threads of a process will share the process virtual address and all the resources of the process. When you create new threads of a process, these new threads will have their own user stack, they will be scheduled independently to be executed by the scheduler. Also for the thread to run concurrently they will have their context which will store the state of the thread just before preemption i.e the status of all the registers.
Is a thread dynamically allocated memory?
No, nothing of the sort. Threads have memory uniquely associated with them -- at least a stack -- but such memory is not the thread itself.
I am trying to understand thread creation and how it differs from a simple malloc call.
New thread creation is not even the same kind of thing as memory allocation. The two are not at all comparable.
Threading implementations that have direct OS support (not all do) are unlikely to rely on the C library to obtain memory for their stack, kernel data structures, or any other thread-implementation-associated data. On the other hand, those that do not have OS support, such as Linux's old "green" threads, are more likely to allocate memory via the C library. Even threading implementations without direct OS support have the option of using a system call to obtain the memory they need, just as malloc() itself must do. In any case, the memory obtained is not itself the thread.
Note also that the difference between threading systems with and without OS support is orthogonal to the threading API. For example, Linux's green threads and the now-ubiquitous, kernel-supported NPTL threads both implement the POSIX thread API.
In my program written with C and C++, I will new an object to fulfill the task, then delete the object.
At the moment after new object but before delete object, if the user presses ctrl+c to break the process, that will cause delete not to be called and a memory leak occurs.
What should I do to avoid this situation?
Also, if the memory was reclaimed by the OS, what about the opened files? Are they closed by the OS or should I close them manualy?
In a virtual-memory-based system, all memory is returned to the OS when a process is terminated, regardless of whether it was freed explicitly in the application code. The same might not be true of other resources, however, which you may want to free cleanly. In which case, you need to provide a custom signal handler for the SIGINT signal (which is received on Ctrl+C), see e.g. http://linux.die.net/man/2/sigaction.
Pressing CtrlC will send a SIGINT to the process, which by default does a mostly-orderly shutdown, including tearing down the memory manager and releasing all allocated heap and stack. If you need to perform other tasks then you will need to install a SIGINT handler and perform those tasks yourself.
If you allocated any SYSV Shared Memory Segments using shmget(2) then you must clean up after yourself with shmctl(2).
If you allocated any POSIX Shared Memory Segments using shm_open(3) then you must clean up after yourself with shm_unlink(3).
Both SYSV and POSIX shared memory segments persist past process termination. You can see what persists using the ipcs(1) tool.
Of course, if you haven't used any SYSV or POSIX shared memory segments, then this is all just noise. :)
You are subscribing to a rather common misconception that heap blocks that are not freed, but still accessible at the time a program exists are leaks. This is not true. Leaked blocks are those which no pointer still references, hence they can't be freed.
Through the years of playing with (and breaking) lots of perfectly good kernels, I have never managed to sufficiently break a virtual memory manager to the point where it no longer reclaimed the entire address space of a process once it exited. Unless you are working with a kernel clearly marked as 'new and experimental', you will have better luck winning the lottery than finding a system that doesn't employ an effective virtual memory manager.
Don't put cruft in your code just to get a perfect score in Valgrind. If you have no real clean up tasks to do other than freeing memory that still has valid references, you don't need to bother. If someone throws a kill -9 to your program, you won't be able to handle it and will see the old behavior repeat.
If you have file descriptors to clean up, shared locks to relinquish, streams to flush or whatever else must happen so other processes don't miss you when you're gone, by all means take care of that. Just don't go adding code that does nothing to solve a non-problem, it just seems silly to do so.
Note
This was originally going to be a comment, but is far too long and SO frowns on writing a novel one comment at a time.
When CTRL+C is pressed in a Linux console, the SIGINT signal is sent to the application which, if the signal has no handler, will terminate the program, returning all memory to the OS. This of course would make it pointless to do any freeing of memory, since all memory will freed once the program exists. However, if you would like to handle the CTRL+C SIGINT signal (maybe to write out some last data to a file or do some other cleanup), you can use the function signal() to install a function to be called when the signal is received. Check out the man page for this function if you want to learn more.
If the process quits, a memory leak will NOT normally occur.
Most of the memory you allocate will be freed on Ctrl+C. If you see memory usage not return to its prior level, it is almost certainly caused by buffered filesystem blocks.
However, you should definitely clean things up, in particular if you have used any other types of resources:
Files created in temporary directories won't be deleted. This includes /dev/shm, leaving such a file could be considered a "memory leak".
System V or posix shared memory segments won't get thrown away when your process quits. If this bothers you, clean them up specifically. Alternatively, clean them up on a subsequent run.
Normally a leak (of a persistent or semi-persistent object e.g. file) doesn't matter if a subsequent run doesn't leak more memory. So cleaning up on a future run is good enough.
Imagine a process running every 5 minutes from "cron", if it crashes on each run and leaves some mess, it's still ok provided each run cleans up the mess from the previous crash.
The OS will reclaim the memory allocated by the process when the process exits as a result of Ctrl-C or any other means.
I experience some memory allocation problems and try to detect possible reasons for these problems.
There are many possible reasons, and lots of hours must be spent to check each of them.
One of the possible reasons is that there is a memory buffer, that is allocated within a thread, and this buffer is used after the thread terminates.
So, if there is a chance that thread termination causes memory deallocation, then many hours of debugging may be avoided.
Thank you very much in advance.
I don't think it does, although it of course might depend on your particular details.
Generally, memory allocation from the operating system's point of view is a per-process activity, while threads exist inside the process. So if one thread allocates memory and then dies, the operating system doesn't clean that up since the process is still alive. Memory is shared inside the process, so the OS can't know that the memory no longer is used and can be cleaned up.
No, threads that 'die' do not deallocate any memory.
When a thread ends, the thread itself vanishes from memory, like a function does once it's done executing. It will take all the 'stack' objects with it, but all the memory you allocated yourself (i.e. malloc) will still be there.
As such, before you end your thread, you should make sure that all dynamic memory that was used by the thread and is not needed any more is freed properly.
Anything on the thread's stack (a local variable, for example) becomes invalid when the thread ends. However, if the data is in the heap, then the memory is still valid as long as the process is running. Of course, you'll need to save the pointer to that heap allocation somewhere outside that thread.
Memory allocated by a thread behaves like memory allocated by a method call:
variables on the stack will be dealocated when the method returns (thread terminates)
variables on the heap will continue to be allocated unless explicitly deallocated.
In addition to all answers, I'd like to make a note that pthread has a TLS keys which are registered with pthread_key_create which accepts key ID and destructor functions. On pthread_exit a static pthread_key_clean_all() is called that iterates through the keys and invokes assigned destructors that may perform memory deallocation (by application design).
So, to understand that - search in your code all pthread_key_create invocations, check if a destructor assigned and the put breakpoints to all of them to check what and in which order is destroyed.
Suppose a program has memory leaks.
1) When a process dies (normally or segmentation fault), are those leaked memory freed?
2) What about other resources a process holds?
With most modern operating systems (linux, windows from around NT 3.5), yes.
Stack and heap memory is freed and file descriptors are closed on all modern system, I think.
On POSIX systems there are a number of resources that are not freed when a process exits, shared semaphores, message queues and memory segments. These are meant to be persistent between processes, so they simply can't. It is the responsibility of the application to free them.
It could do that e.g with on_exit handlers, but usually there is a simpler way. For memory segments you would typically use shm_unlink after all processes have opened such a segment. The segment then ceases to exist when the last process (and its file descriptor to the segment) is closed.
1) Yes, the memory is freed.
2) Different process model? I don't know what you mean by that, but once a program dies, all the memory that it malloc'd or new'd is then returned to the OS and will be reallocated to another program later.
3) Once a program exits, all allocated memory is returned to the OS, however until the process is wait()ed by another process, there is a small amount of data such as the exit status waiting around for someone to collect it. On linux, I believe, a normal process from bash/init will be waited and cleaned up automatically.
You can safely assume with modern linux systems that the memory will be freed... However... Its not a guaruntee, and certainly not best practice.
In my program written with C and C++, I will new an object to fulfill the task, then delete the object.
At the moment after new object but before delete object, if the user presses ctrl+c to break the process, that will cause delete not to be called and a memory leak occurs.
What should I do to avoid this situation?
Also, if the memory was reclaimed by the OS, what about the opened files? Are they closed by the OS or should I close them manualy?
In a virtual-memory-based system, all memory is returned to the OS when a process is terminated, regardless of whether it was freed explicitly in the application code. The same might not be true of other resources, however, which you may want to free cleanly. In which case, you need to provide a custom signal handler for the SIGINT signal (which is received on Ctrl+C), see e.g. http://linux.die.net/man/2/sigaction.
Pressing CtrlC will send a SIGINT to the process, which by default does a mostly-orderly shutdown, including tearing down the memory manager and releasing all allocated heap and stack. If you need to perform other tasks then you will need to install a SIGINT handler and perform those tasks yourself.
If you allocated any SYSV Shared Memory Segments using shmget(2) then you must clean up after yourself with shmctl(2).
If you allocated any POSIX Shared Memory Segments using shm_open(3) then you must clean up after yourself with shm_unlink(3).
Both SYSV and POSIX shared memory segments persist past process termination. You can see what persists using the ipcs(1) tool.
Of course, if you haven't used any SYSV or POSIX shared memory segments, then this is all just noise. :)
You are subscribing to a rather common misconception that heap blocks that are not freed, but still accessible at the time a program exists are leaks. This is not true. Leaked blocks are those which no pointer still references, hence they can't be freed.
Through the years of playing with (and breaking) lots of perfectly good kernels, I have never managed to sufficiently break a virtual memory manager to the point where it no longer reclaimed the entire address space of a process once it exited. Unless you are working with a kernel clearly marked as 'new and experimental', you will have better luck winning the lottery than finding a system that doesn't employ an effective virtual memory manager.
Don't put cruft in your code just to get a perfect score in Valgrind. If you have no real clean up tasks to do other than freeing memory that still has valid references, you don't need to bother. If someone throws a kill -9 to your program, you won't be able to handle it and will see the old behavior repeat.
If you have file descriptors to clean up, shared locks to relinquish, streams to flush or whatever else must happen so other processes don't miss you when you're gone, by all means take care of that. Just don't go adding code that does nothing to solve a non-problem, it just seems silly to do so.
Note
This was originally going to be a comment, but is far too long and SO frowns on writing a novel one comment at a time.
When CTRL+C is pressed in a Linux console, the SIGINT signal is sent to the application which, if the signal has no handler, will terminate the program, returning all memory to the OS. This of course would make it pointless to do any freeing of memory, since all memory will freed once the program exists. However, if you would like to handle the CTRL+C SIGINT signal (maybe to write out some last data to a file or do some other cleanup), you can use the function signal() to install a function to be called when the signal is received. Check out the man page for this function if you want to learn more.
If the process quits, a memory leak will NOT normally occur.
Most of the memory you allocate will be freed on Ctrl+C. If you see memory usage not return to its prior level, it is almost certainly caused by buffered filesystem blocks.
However, you should definitely clean things up, in particular if you have used any other types of resources:
Files created in temporary directories won't be deleted. This includes /dev/shm, leaving such a file could be considered a "memory leak".
System V or posix shared memory segments won't get thrown away when your process quits. If this bothers you, clean them up specifically. Alternatively, clean them up on a subsequent run.
Normally a leak (of a persistent or semi-persistent object e.g. file) doesn't matter if a subsequent run doesn't leak more memory. So cleaning up on a future run is good enough.
Imagine a process running every 5 minutes from "cron", if it crashes on each run and leaves some mess, it's still ok provided each run cleans up the mess from the previous crash.
The OS will reclaim the memory allocated by the process when the process exits as a result of Ctrl-C or any other means.