I'm getting segfaults when I run my project. Every time I run the program in gdb, the segfaults disappear. This behavior is not random: each time I run it in my shell it segfaults, each time I run it in gdb, the segfaults disappear. (I did recompile using -g).
So before I start adding printfs frantically everywhere in my code, I would like to know a few things:
Is this behavior common?
What's the best way to approach the issue?
I don't know if tests can be scripted since my application is interactive and crashes on a particular user input.
I didn't paste my code here because it'd be way too long. But if anyone is interested in helping out, here it is:
https://github.com/rahmu/Agros
The easiest way to figure it out is to capture core dumps:
$ ulimit -c unlimited
Then run your program. It will generate a core file
Then use gdb:
$ gdb ./program core
And gdb will load and you can run a backtrace to see exactly what operation elicited the segfault.
Does it do a core dump? Is so that load up the core dump in the debugger. Otherwise change the code to get it to do a core dump.
My guess is that it's a concurrency problem causing a reference to be freed out from under a method call that's assuming that the pointer it has will stay valid. The reason that gdb is probably masking this is because GDB only allows 2 threads to actually concurrently run. If you have more than 2 threads running the only 2 will actively run concurrently. GDB also has performance hits which could be masking this specific condition. As mentioned by Ed just make your application core dump and you can open up the core in GDB and check the stack.
Is this behavior common?
Yes. Undefined behaviour is the source of most of these problems, and by definition it is undefined. Recompiling with -g may certainly affect the results. Recompiling at all may change the results, if the compiler uses some pseudo-random genetic algorithm to optimize stuff or something like that.
What's the best way to approach the issue?
An ounce of prevention is worth a ton of cure; learn the common causes of undefined behaviour and pick up good habits to avoid writing them. Once you've found that there is a problem, static analysis of the code is often a good idea; go through and reason to yourself and prove that indices will stay in bounds, data will fit its arrays, invalid pointers won't be dereferenced etc.
Related
We are currently developing a cluster manager software in C. If several nodes connect to the manager, it works perfect, but if we use some tools to simulate 1000 nodes to connect the manager, it will sometimes work in unexpected ways.
How can one debug this kind of bug? It only appears when the load(connection/nodes) is large?
If I use gdb to debug step by step, the app never malfunctions.
How to debug this kind of bug?
In general, you want to use at least these techniques:
Make sure the code compiles and links without warnings. The -Wall is a good start, but -Wextra is better.
Make sure the application has designed-in logging and tracing, which can be turned on or off, and which has sufficient details to debug these kinds of issues, and low overhead.
Make sure the code has good unit-test coverage.
Make sure the tests are sanitizer-clean.
there's also no warning in valgrind check.
It's not clear whether you've simply ran the target application under Valgrind, or whether you also have the unit tests, and the tests are Valgrind-clean. It's also not clear whether you've observed the application mis-behavior under Valgrind or not.
Valgrind used to be the best tool available for heap and unintialized memory problems, but in 2017 this is no longer the case.
Compiler-based Address, Thread and Memory sanitizers catch significantly wider class of errors (e.g. global and stack overflows, and data races), and you should run your unit tests under all of them.
When all of the above still fails to find the problem, you may be able to run the real application instrumented with sanitizers.
Lastly, there are tools like GDB tracing and systemtap -- they are harder to learn, but give you significant power. Overview here.
Sadly the debugger is less useful for debugging concurrency/load issues.
Keep adding logs/printfs, trigger the issue with load testing, then try to narrow it down with more logs/printfs. Repeat.
The faster it is to trigger the bug the faster this will converge. Also prefer the classic "bisection" / "binary search" technique when adding logs - try to narrow down the areas you're looking at by at least half every time.
Many times we don't get correct or complete stack dump during crash. My question is in what all cases we can see this to happen.
Probably it can be because of the function call stack getting corrupted. But How such corruption happens.
My 2nd question is how do we debug such an issue and what approach we can take to find the root cause for the crash.
I understand my questions may not have an exact answer but I would like to know your thoughts.
Thank You...
It is operating system and platform (i.e. processor) specific.
The best way is to use a debugger to find such issues (perhaps a remote one, learn about gdbserver)
I would suggest to debug most of your code on a desktop Linux system (because you have lots of useful tools: valgrind, gcc -fsanitize=address, gdb, etc...)
Of course, the call stack can be corrupted to the point of being unusable. Try to memset the stack segment, then return from the function doing that (no matter what tool or trick you would use, the stack is then desperately corrupted on most platforms)!
You might be interested by GNU glibc backtrace function, GCC __builtin_return_address, libbbacktrace by Ian Taylor in GCC
You might also enable core dumps and analyze them post mortem (perhaps using a cross-debugger). See core(5), proc(5), setrlimit(2)
I got this problem on different C projects when using gdb.
If I run my program without it, it crashes consistently at a given event probably because of a invalid read of the memory. I try debugging it with gdb but when I do so, the crash seems to never occur !
Any idea why this could happen ?
I'm using mingw toolchain on Windows.
Yes, it sounds like a race condition or heap corruption or something else that is usually responsible for Heisenbugs. The problem is that your code is likely not correct at some place, but that the debugger will have to behave even if the debugged application does funny things. This way problems tend to disappear under the debugger. And for race conditions they often won't appear in the first place because some debuggers can only handle one thread at a time and uniformly all debuggers will cause the code to run slower, which may already make race conditions go away.
Try Valgrind on the application. Since you are using MinGW, chances are that your application will compile in an environment where Valgrind can run (even though it doesn't run directly on Windows). I've been using Valgrind for about three years now and it has solved a lot of mysteries quickly. The first thing when I get a crash report on the code I'm working with (which runs on AIX, Solaris, BSDs, Linux, Windows) I'm going to make one test run of the code under Valgrind in x64 and x86 Linux respectively.
Valgrind, and in your particular case its default tool Memcheck, is going to emulate through the code. Whenever you allocate memory it will mark all bytes in that memory as "tainted" until you actually initialize it explicitly. The tainted status of memory bytes will get inherited by memcpy-ing uninitialized memory and will lead to a report from Valgrind as soon as an uninitialized byte is used to make a decision (if, for, while ...). Also, it keeps track of orphaned memory blocks and will report leaks at the end of the run. But that's not all, more tools are part of the Valgrind family and test various aspects of your code, including race conditions between threads (Helgrind, DRD).
Assuming Linux now: make sure that you have all the debug symbols of your supporting libraries installed. Usually those come in the *-debug version of packages or in *-devel. Also, make sure to turn off optimization in your code and include debug symbols. For GCC that's -ggdb -g3 -O0.
Another hint: I've had it that pointer aliasing has caused some grief. Although Valgrind was able to help me track it down, I actually had to do the last step and verify the created code in its disassembly. It turned out that at -O3 the GCC optimizer got ahead of itself and turned a loop copying bytes into a sequence of instructions to copy 8 bytes at once, but assumed alignment. The last part was the problem. The assumption about alignment was wrong. Ever since, we've resorted to building at -O2 - which, as you will see in this Gentoo Wiki article, is not the worst idea. To quote the relevant partÖ
-O3: This is the highest level of optimization possible, and also the riskiest. It will take a longer time to compile your code with this
option, and in fact it should not be used system-wide with gcc 4.x.
The behavior of gcc has changed significantly since version 3.x. In
3.x, -O3 has been shown to lead to marginally faster execution times over -O2, but this is no longer the case with gcc 4.x. Compiling all
your packages with -O3 will result in larger binaries that require
more memory, and will significantly increase the odds of compilation
failure or unexpected program behavior (including errors). The
downsides outweigh the benefits; remember the principle of diminishing
returns. Using -O3 is not recommended for gcc 4.x.
Since you are using GCC in MinGW, I reckon this could well apply to your case as well.
Any idea why this could happen ?
There are several usual reasons:
Your application has multiple threads, has a race condition, and running under GDB affects timing in such a way that the crash no longer happens
Your application has a bug that is affected by memory layout (often reading of uninitialized memory), and the layout changes when running under GDB.
One way to approach this is to let the application trap whatever unhandled exception it is being killed by, print a message, and spin forever. Once in that state, you should be able to attach GDB to the process, and debug from there.
Although it's a bit late, one can read this question's answer in order to be able to set up a system to catch a coredump without using gdb. He may then load the core file using
gdb <path_to_core_file> <path_to_executable_file>
and then issue
thread apply all bt
in gdb.
This will show stack traces for all threads that were running when the application crashed, and one may be able to locate the last function and the corresponding thread that caused the illegal access.
Your application is probably receiving signals and gdb might not pass them on depending on its configuration. You can check this with the info signals or info handle command. It might also help to post a stack trace of the crashed process. The crashed process should generate a core file (if it hasn't been disabled) which can be analyzed with gdb.
I am experimenting with the c-language right at the moment, yet i have some trouble with memory allocation. After some time i have to restart my computer because my memory runs full. Is there a way to let the compiler tell me which arrays do not get deallocated after the program has run?
Thx for answers
you can use valgrind to do that.
http://tldp.org/HOWTO/Valgrind-HOWTO/
http://valgrind.org/
use it on your compiled program with --leak-check=yes
You didn't tell us anything about your compiler, OS, platform... so the rest could only be wild guesses.
This sounds much that you have dead processes or something like that that keep eating your memory in the background. On linux you have top (and inside top press M) to inspect the processes running on your system and how much memory, time etc they consume. Do that to see what is happening on your machine and don't reboot it blindly without knowing the reason.
There are equivalent tools on all other operating systems that let you inspect the current state of processes.
You have tools that can tell you about memory leaks. Compilers i am afraid may not be useful for tha tpurpose.
You can also use DevPartner or Valgrind to analyse your memory leaks in case you are suspecting them. But for your system to be restarted because of memory issues how long do you run the application before you perform a restart.
How did you get to know that this is a memory related issue.
You better check your source code first, if you are under Linux, using 'splint' to your source and that will display you a lot, try to fix those warnings or errors, if everything gets done, recompile your source and try 'valgrind' to the exacutable.
You can see the reference of splint through its official website and so as valgrind.
splint: www.splint.org
valgrind: valgrind.org
Good luck~~~
HI, i am recently in a project in linux written in C.
This app has several processes and they share a block of shared memory...When the app run for about several hrs, a process collapsed without any footprints so it's very diffficult to know what the problem was or where i can start to review the codes....
well, it could be memory overflown or pointer malused...but i dunno exactly...
Do you have any tools or any methods to detect the problems...
It will very appreciated if it get resolved. thanx for your advice...
Before you start the program, enable core dumps:
ulimit -c unlimited
(and make sure the working directory of the process is writeable by the process)
After the process crashes, it should leave behind a core file, which you can then examine with gdb:
gdb /some/bin/executable core
Alternatively, you can run the process under gdb when you start it - gdb will wake up when the process crashes.
You could also run gdb in gdb-many-windows if you are running emacs. which give you better debugging options that lets you examine things like the stack, etc. This is much like Visual Studio IDE.
Here is a useful link
http://emacs-fu.blogspot.com/2009/02/fancy-debugging-with-gdb.html
Valgrind is where you need to go next. Chances are that you have a memory misuse problem which is benign -- until it isn't. Run the programs under valgrind and see what it says.
I agree with bmargulies -- Valgrind is absolutely the best tool out there to automatically detect incorrect memory usage. Almost all Linux distributions should have it, so just emerge valgrind or apt-get install valgrind or whatever your distro uses.
However, Valgrind is hardly the least cryptic thing in existence, and it usually only helps you tell where the program eventually ended up accessing memory incorrectly -- if you stored an incorrect array index in a variable and then accessed it later, then you will still have to figure that out. Especially when paired with a powerful debugger like GDB, however (the backtrace or bt command is your friend), Valgrind is an incredibly useful tool.
Just remember to compile with the -g flag (if you are using GCC, at least), or Valgrind and GDB will not be able to tell you where in the source the memory abuse occurred.