I'm trying to use gdb to debug a C program which all works ok until I pass a pointer to a function (to a third party library), after which gdb loses focus and the program runs without hitting my breakpoint in my call back function.
For example I am calling pcap_loop from the libpcap library which expects a pointer to my call back function got_packet.
pcap_loop(handle, num_packets, got_packet, NULL);
As soon as i step in or over this line with gdb my break point in got_packet is never hit.
Why ?
Any ideas?
There are some circumstances in which GDB breakpoints can get skipped, especially when debugging code compiled with optimization enabled, but the most likely answer is that the library function isn't doing what you expect.
GDB can't step into libraries that don't have debug information, so instead it just appears to "lose focus", as you put it, sets a temporary breakpoint on the return point and waits for the function to finish. That still shouldn't stop the breakpoint inside the call-back from triggering.
If you are using a library that came from your OS repository then you might find that there is a "debug" package you can install. This would allow you to step into the library code and possibly work out what the problem is that way.
However, it's usually far simpler to add printf debugging, or "break" the got_packet function in some way that will cause a signal (*(int *)0 = 1), and satisfy yourself that way whether it is being called or not.
If you can show that the function really is being called, and that GDB really isn't catching the breakpoint, then you have a GDB bug (or maybe a kernel bug).
Related
I am debugging an application to fix a segmentation fault that I suspect to be caused by a race condition.
I'd like to put some print statements in the code, but I know for experience that adding calls to printf is not recommended since this could change the behavior of the threads and in some case hide the bug.
Looking at other options, I have seen that with gdb it is possible to use break points to print something and then automatically continue the execution:
break foo
commands
silent
printf "Called foo: x is %d\n",x
cont
end
Is this any better then putting a printf in my code?
I know that gdb has a also Tracepoints but they only work with gdbserver and this is an additional level of complication that I would prefer to avoid at the moment.
Additional information: the application is written in C and it runs on Linux.
Is this any better then putting a printf in my code?
No, it's much worse. Every breakpoint that is hit in GDB triggers the following chain of events:
context switch from running thread to GDB
GDB stops all other threads (assuming default all-stop mode)
GDB evaluates breakpoint commands
GDB resumes all threads (this is itself a complicated multi-step process, which I would not go into here).
This is at least an order of magnitude more expensive and disruptive than a simple printf call, and is very likely to hide whatever race you were trying to debug.
The bottom line is that GDB is in general completely unsuitable for debugging data races.
I second the ThreadSanitizer recommendation by Christopher Ian Stern.
The only problem with this bug is that I am doing the debug on a production machine where I cannot install other SW.
First, ThreadSanitizer instruments your existing program. It has a runtime library, but that could be statically linked into your binary. There is nothing that you need to install on your production machine.
Second, ThreadSanitizer detects data races even when they do not cause visible behavior changes. It may well turn out that you don't need to run on your production machine at all: simply running your tests (you do have tests, right?) on your development machine may prove to be sufficient.
Since you are on Linux, I would recomend ThreadSanitizer. That is using a recent version of gcc or clang and passing -fsanitize=thread to the build. This isn't a printf repacment but should tell you explicitly about any race conditions in your code. Even after you solve this problem if you are working with multithreaded code, you will want to have this tool available. Alternately, or in addition, I have had good results with Valgrind's http://valgrind.org Data Race Detector, but I would start with ThreadSanitizer.
Is it possible to control gdb from the debugged program? I'm hoping for a library that can help with this, ideally with an API such as gdb_sendcmd("bt"), but I can live with something like connecting to local gdb via a socket.
Primary use case is programmatically adding a data breakpoint to monitor when a certain memory address next gets modified. Target language is, naturally, C; anything applicable to it can be reused with C++ and Objective-C.
Answer from Employed Russian has solved my direct problem, but I'd still love to know how I can run GDB commands programmatically from the debugged program. It may speed up some debugging if I could simply add code to the project instead of writing extra startup commands for GDB that would create breakpoints with attached commands.
So if there is a way to run commands programmatically, I'd still love to hear it ;)
The Python interface exported by GDB allows you to do many things. Maybe something like this would fit your requirements:
import gdb
CMD_FCT = "gdb_run"
CMD_NAME = "str"
class GdbRunBreakpoint(gdb.Breakpoint):
def __init__(self):
gdb.Breakpoint.__init__(self, CMD_FCT, internal=1)
self.silent = True
def stop(self):
cmd = gdb.parse_and_eval(CMD_NAME).string()
gdb.execute(cmd)
return False
GdbRunBreakpoint()
(just write than in a file, and source it from your .gdbinit file)
and on the application side:
void gdb_run(char *str) {}
int main () {
gdb_run("where");
}
I think the code is straight forward, but as I mentioned in https://stackoverflow.com/a/8884512/341106, not everything is allowed in the stop callback, GDB is in an intermediate state, but many things will work as expected.
EDIT: needless to say, this won't work if you app is not compiled with debug symbols!
Is it possible to control gdb from the debugged program?
No. If the program could do that, and (say) disabled all breakpoints, how would you debug it?
Primary use case is programmatically adding a data breakpoint to monitor when a certain memory address next gets modified.
This often comes up in a context like this: on Nth invocation of foo(), a (local or global) variable bar gets unexpectedly clobbered, and you want to find the culprit.
If that is indeed your problem, then set a breakpoint on the line where you know the value of bar is still good (just after it has been initialized). Set ignore count (using ignore command) on that breakpoint to N-1, and then, when the breakpoint is hit, set the watchpoint on bar.
You can even attach commands to the breakpoint:
commands 1 # assuming this was the first breakpoint
watch bar
continue
end
so the watchpoint is attached automatically.
I'm developing in C++(g++) with a non-opensource lib.
every time I run the program, the lib will crash (it double-free some memory).
it's ok for my program now. but it's bad for profiling. I use -pg to profiling the program. As a result of the crash, no 'gmon.out' is generated. so I cannot profile it at all.
Question:
How to profiling a 'crashy' program (with gprof).
PS. valgrind is ok to analysis a crashy program.
regards!
There's a function you can call from your program to dump profile data (the same one that's automatically installed as an atexit handler when you link with -pg), but I don't know what it's called offhand.
The easyist thing to do it, just insert an exit(0); call at a suitable point in your program. Alternatively, you can set a breakpoint and use call exit(0) in GDB (except that debugging the program will affect the profile data if you stop it in the middle).
I'm doing a project that makes unix system calls. To be specific, my project heavily relies of calls to getcontext(), makecontext(), swapcontext() and setcontext(). I tried to debug my code using gdb. I stepped into the code line by line and examined the control but once a call to swapcontext() is made, it doesn't step into the code anymore. Instead, debugging almost stops there and the rest of the program is run automatically instead of line by line. I guess gdb does not go into context calls? Is there any way to get around this? Is there a debugger that i can use for this?
Thanks
setcontext and swapcontext calls change the program's stack, and gdb gets confused. I do not know whether some other debugger can handle it nicely.
gdb steps through one thread and calls this the current thread. Other threads will run as you are doing this. If you set a breakpoint that gets hit in a thread other than the current thread then gdb will change the current thread to that thread. Stepping is now relative to the new current thread.
Stepping with gdb over calls of swapcontext() with 'step' or 'next' does not work because not only the stackpointer changes but also the call returns to a different code line (which is the desired effect of swapcontext()). As gdb puts a breakpoint in the next code line which will not be executed until another swapcontext() returns to this place the execution will not break.
You need to foresee the line to which swapcontext() will return and set a breakpoint there. For a new (unused) context this will be the line you specified as entry function. For used contexts it will probably one of the lines after a swapcontext() there...
You can repeatedly use GDB's stepi command to first step into and then step through the swapcontext() function. You must step several dozen times, including a few steps over kernel system calls—I presume to save the floating point state?—and you'll eventually come out in the user thread you're swapping to. It's a tad time-consuming, but it works.
As much as you may not like this answer, the best bet is to step through your code in small chunks by hand. Threaded programs don't play very well with debuggers like GDB and Valgrind (at least in my experience) and most bugs can be determined by a careful step-by-step manual analysis of the code.
I'm trying to use check unit testing framework for my C application. But I can't use the debugger (gdb) with it because of two points:
first, check use some complex macros (START_TEST and END_TEST) and the debugger has trouble to put a breakpoint in my code between these two macros (in fact, I can put a software breakpoint but It is never seen by gdb)
second, check define some sort of exceptions by redefining behavior of interruption. Hence, when I try to put a hardware breakpoint, the test failed and exit because check consider the hardware breakpoint as a failure of my test.
Does anyone has already met this problem and has a solution?
Look at the no-fork mode:
Check normally forks to create a separate address space. This allows a signal or early exit to be caught and reported, rather than taking down the entire test program, and is normally very useful. However, when you are trying to debug why the segmentation fault or other program error occurred, forking makes it difficult to use debugging tools.
Actually, you CAN use fork-mode too.
gdb has two interesting options related to fork behaviour:
- detach-on-fork (set this to false)
- follow-on-fork (either parent or child; I always take child)
This will make gdb follow the child process.
When the child process has ended, you have to manually switch back to the parent process by using the inferior command.
I read this and he suggests a very simple solution:
gdb > set environment CK_FORK=no
that worked for me. I could then set a breakpoint in functions the test cases call (that is, the functions under test), and it broke at the right place.
Try TAP (Test Anything Protocol) … it's a lot easier to implement, ship and debug. It's also very easy to make it valgrind-aware and tends to play nicer with gdb.