When an application causes a serious segment-fault issue, which is hard to find or track. I can use a debug version and generate a core dump file when issue happens. And debug this app with core-dump file.
But how to track down exceptional bugs in application when released? There seems to be no core-dump file in release version. Although log is an option, it is useless when there is a hard to track bugs happens.
So my question is how to track down those hard to track bugs in release version? Any suggestions or technology out there available?
Following reference may help the discussion.
[1] Core dump in Linux
[2] generate a core dump in linux
[3] Solaris Core dump analysis
You can compile a release version with gcc -g -O2 ...
The lack of core dump is related to your user's setting of resource limits (unless the application is explicitly calling setrlimit or is setuid; then you should offer a way to avoid that call). You might teach your users how to get core dumps (with the appropriate bash ulimit builtin).
(and there is some obscure way to put the debugging information outside of the executable)
The distributions provide -dbg packages that provide debugging symbols for programs. They are built along with the binary packages and can provide your users the ability to generate meaningful backtraces from core dumps. If you build your packages using the same utilities, you can get these -dbg packages for your own software "nearly free".
I suggest to use a crash reporting system, in my experience we use google's break-pad project for our windows client program, of course you can write your own.
Google break-pad is an open-source multi-platform crash reporting system, it can make mini or full memory dump when exception or crash happen, then you can config it to upload the dump file and any additional files to a specific ftp server or http server, very help to find bug.
Here is the link:
Google Break-pad
Ask the "customer" for a description of what he or she did to make it crash, and try to replicate it yourself with your own version that has debug information.
The hard part is getting correct information from the customer. Often they will say they did nothing special or nothing different than before. If possible, go see the person having the problem, and ask them to do what they do to make the program crash, writing down every step.
Related
I'm debugging a program written in plain C (no C++, MFC, .NET, etc.) to the WIN32API. It must compile in both VS2005 (to run under Win 2K/XP) and VS2010 (to run under Win7.) I've been unable to duplicate a bug that my customer seems able to duplicate fairly reliably, so I'm looking for ways to have my program "debug itself" as-it-were. It is monitoring all of the key values that are changing, but what I'd really like to see is a stack dump when a value changes. Oh, I cannot run a "true" debug build (using the debug libraries) without installing the compiler on the customer's machine and that is not an option, so this must be built into my release build.
Is there any way to do this other than just adding my own function entry/exit calls to my own stack monitor? I'd especially like to be able to set a hardware breakpoint when a specific memory address changes unexpectedly (so I'd need to be able to disable/enable it around the few EXPECTED change locations.) Is this possible? In a Windows program?
I'd prefer something that doesn't require changing several thousand lines of code, if possible. And yes, I'm very underprivileged when it comes to development tools -- I consider myself lucky to have a pro version of the Visual Studio IDEs.
--edit--
In addition to the excellent answers provided below, I've found some info about using hardware breakpoints in your own code at http://www.codereversing.com/blog/?p=76. I think it was written with the idea of hacking other programs, but it looks like it might work find for my needs, allowing me to create a mini dump when an unexpected location writes to a variable. That would be cool and really useful, especially if I can generalize it. Thanks for the answers, now I'm off to see what I can create using all this new information!
You can use MiniDumpWriteDump function which creates a dump, which can be used for post-mortem debugging. In the case application crashes, you can call MiniDumpWriteDump from unhandled exception handler set by SetUnhandledExceptionFilter. If the bug you are talking about is not crash, you can call MiniDumpWriteDump from any place of the program, when some unexpected situation is detected. More about crash dumps and post-mortem debugging here: http://www.codeproject.com/Articles/1934/Post-Mortem-Debugging-Your-Application-with-Minidu
The main idea in this technique is that mini dump files produced on a client site are sent to developer, they can be debugged - threads, stack and variables information is available (with obvious restrictions caused by code optimizations).
There are a bunch of Win32 functions in dbghelp32.dll that can be used to produce a stack trace for a given thread: for an example of this see this code.
You can also look up the StackWalk64() and related functions on MSDN.
To get useful information out, you should turn on PDB file generation in the compiler for your release build: if you set up your installer so that on the customer's computer the PDB files are in the same place as the DLL, then you can get an intelligible stack trace out with function names, etc. Without that, you'll just get DLL names and hex addresses for functions.
I'm not sure how practical it would be to set up hardware breakpoints: you could write some sort of debugger that uses the Win32 debugging API, but that's probably more trouble than its worth.
If you can add limited instrumentation to raise an identifiable exception when the symptom recurs, you can use Process Dumper to generate a full process dump on any instance of that exception.
I find I cite this tool very frequently, it's a real godsend for hard-to-debug production problems but seems little-known.
I am an intermediate C programmer who is learning his way around. I recently learned using GDB (forced to actually) and also am doing some code using Ptrace.
I was browsing through some code on ptrace and came across "DEBUG INFO".
I want to know what this means and when should I include this in my code?
What are debug info files?
I couldnt find much from google and wiki related to this. There is no wiki for that tag too.
May be if you can point me to the right direction I can repay by writing one.
DEBUG_INFO you are referring to is probably just an option (and define) of linux kernel. Enabling it when configuring kernel tells compiler to include debug information in resulting binary. This debug information itself is just additional data, that allows debuggers (and other tools like strace, addr2line, gprof, ...) relate binary code it's executing to source it was compiled from (source lines, variable names, etc, etc). You can find more details on debug information in general starting at DWARF (popular debug data format).
check for what is a core dump .you might get some directions.
The whole credit goes to Peter for the answer.
Initially I went through "What is core dump"
Then I came across the following link which actually tells the difference between
Debug Build
Release build
which is: debug build includes debug information while a release built doesn't have these and is Optimized code.
Also this answer can help you get more information.
I have written an application in C, which runs as a Windows service. Most users can run the app without any problems, but a significant minority experience crashes caused by an Access Violation, so I know I have a bug somewhere. I have tried setting up virtual machines to mirror the users' configurations as closely as possible, but cannot reproduce the issue.
My background is in Java - when a Java app crashes it will produce a stack trace showing exactly where the problem occurred, but native applications aren't so helpful. What techniques are normally used by C developers for tracking down this type of problem? I have no physical access to the users' machines that experience the crash, but I could send then additional tools to install, to capture information. I also have Windows error reports showing Exception Code/Offset etc but these don't mean much to me. I have compiled my application using gcc - are there some compiler options I can use to generate more information in the event of a crash?
You could try asking the users to run ProcDump to capture a core dump when the program crashes. Unlike using something like Visual Studio it's a single, simple command-line utility so there should be no problem getting the users to run it.
On most modern operating systems your app can install a crash handler that'll walk the stack(s) in the event of a crash. I have no experience doing this on Windows, but this article walks through how to do it.
I have learned C and I would like to start to improve open source software. I would like to hack away one irritating bug in GTK+. To see the bug I need to use Gedit.
How can I download the sources of GTK+ and gedit and compile both of them so that I can see where the bug is? And I have never used any debugger in Linux so is there somewhere a tutorial for that?
You can get information about downloading and compiling of gedit here:
http://projects.gnome.org/gedit/developers.html
My Debugger of choice is GDB:
http://www.gnu.org/software/gdb/
GDB is a console application that may be hard to handle for a beginner. Try an IDE like Eclipse that provides a GUI to GDB.
Almost all Linux debuggers are front-ends for or adaptations of the gdb debugger. You should therefore learn how to use this first, preferably by starting on
small programs of your own construction, not giant FOSS codebases. The gdb manual,
available from here is actually a pretty good tutorial.
The information on building and installing GTK+ should be here:
http://www.gtk.org/development.html
The sources should be here:
http://www.gtk.org/download-linux.html
You can check out gdb:
http://www.gnu.org/software/gdb/ That's a pretty standard linux debugger. I would spend time with it on something simple first, or get an IDE that uses it. Learning gdb can be worth the time though.
gdb is a dinosaur which should long since have been made extinct. Debugging is one area where (gasp!) Windows beats Unix. Having got that off my chest, I advise you to start with the Data Display Debugger (DDD) graphical front end to gdb. Yes, the GUI is quaint, but you will be far more productive, quicker, than if you start with gdb.
Also don't overlook valgrind for finding and diagnosing memory errors. The KDE project actually mandates use of valgrind; I'm not sure about Gnome.
It is possible that you won't see the bug if you download the latest gedit and gtk+ sources. It might be fixed in the latest sources or the latest sources might be different enough to not trigger your bug.
What Linux distribution are you running?
First of all I'd suggest consulting your distribution's bug database to see if someone has filed a similar bug. If you don't find anything, I'd suggest using your distribution's tools to obtain the source that corresponds to the binary that you have installed on your sytem (e.g. apt-get source libgtk-2.0 on Debian or Ubuntu).
Also, your distribution might also make a debug package available (e.g. libgtk2.0-0-dbg on Debian) which will let you run a debugger on the binary that you already have without requiring the source. While this is no substitute for having the sources, it can be useful for running valgrind or making sense out of a core file.
I'm working on a free software (bsd license) project with others. We're searching for a system that check out our source code (svn) and build it also as test it (unit tests with Check / other tools).
It should have a webbased interface and generate reports.
I hope we don't have to write such a system from null by ourselves...
You surely do not have to code this yourself - there are a lot of continuous integration systems which are able to check out source code from systems such as SVN and they are generally easy to extend with your own tasks, so running custom test scripts/programs should not be a problem.
While these CI systems are probably not written in C, this does not matter, since they just need to be able to access and compile your source code, for which they will use an external compiler anyways.
Just to list some of the well known CI tools:
CruiseControl
Hudson
TeamCity
You might also be interested in other questions on Stack Overflow tagged as continuous-integration. :)
I don't think that there's a buildsystem that is capable of doing all this tasks - but what about combining them?
SCons is a nice buildsystem that runs on every machine that has Python. It can even build directly from SVN. For automatic building you can try Buildbot.
Check out buildbot
My vote would be CruiseControl.NET, it has everything you are asking for. It is open source so the costs are low, and it has a very active user community on google groups to help you with your problems as you grow accustomed to it. Also, although .NET based, using MONO it is very nice on Linux and Mac build servers as well so you have everything covered.