I'm trying to show the variables in all stack frames, but haven't found a good way to do this in one shot like I can with bt full. The 'frame variable' only shows the variables in the current frame and there didn't seem to be any equivalent in the LLDB mapping guide.
bt all is the short alias, thread backtrace all is the full command.
In the olden days, in gdb you needed to do thread apply all backtrace iirc, or t a a bt for short. They must have added bt full since then. I'm not sure if I'm a fan of bt full, honestly. Is it asking for a complete backtrace of a single thread? Or a complete backtrace of all threads? My first impression is that this would only give you a complete backtrace of one thread.
Anyway, not a big fan of gdb's bt full naming, not sure I'd want to add it as a recognized command in lldb, I think lldb's bt all shortcut is a good choice.
Related
Sorry for the nondescript title, couldn't think of anything better.
When I run my program through GDB to find a Segmentation Fault, I get thousands of lines of this garbage:
(gdb) step
_IO_new_file_xsputn (f=0x76fa0b40 <_IO_2_1_stdout_>, data=0x7efff284, n=7) at fileops.c
1279 fileops.c: No such file or directory.
(gdb) step
1273 in fileops.c
(gdb) step
1279 in fileops.c
(gdb) step
1286 in fileops.c
(gdb) step
1288 in fileops.c
(gdb) step
1289 in fileops.c
(gdb) step
1292 in fileops.c
(gdb) step
1294 in fileops.c
(gdb) step
1292 in fileops.c
...and on and on and on. It makes debugging very difficult and tedious because:
I must type step (or press Enter) repeatedly.
I can hardly ever even see the output of my program, as it's hidden in pointless information.
If I'm pressing Enter repeatedly, I often miss the Seg Fault I'm looking for.
How can I tell gdb to quit printing the lines like this?
1273 in fileops.c
Is there some config file or command-line option I can use?
Are you sure you need to step through glibc code in fileops.c ?
If you want only to debug your own code you should finish current frame to go out of _IO_new_file_xsputn function after you have stepped into it and continue debugging your code.
How can I tell gdb to quit printing the lines like this?
By using the step command, you asked GDB to print exactly this info.
If you don't want that info, you should stop asking for it.
I often miss the Seg Fault I'm looking for.
If you simply run, GDB will automatically stop when you get a SIGSEGV, you can't miss that.
What you probably want is to stop in your code some time before the SIGSEGV, and step from there. There are several techniques to achieve this, reverse debugging is one of them. Using replay debugging is another.
It's hard to offer more advice, because you didn't show what you are actually trying to do.
That's how step works - it steps through the code.
What about a breakpoint? If you set one the debugger will stop there and let you step through the code manually.
For something like a segmentation fault or similar signals, which abort execution, you can run and backtrace when the program has stopped to gain information about the location execution has stopped at. Then set a breakpoint there and the program will stop at the offending instruction.
Sometimes in GDB I want to see the control flow that got the program to where it is now. Simply put, how do I make GDB print the last x lines executed?
This is yet another use case for Reverse Debugging.
You should start process record and replay at some point:
(gdb) record
When you want to see last executed lines you can go backwards like
this:
(gdb) reverse-step 3
or
(gdb) reverse-next 3
Use this answer https://stackoverflow.com/a/1545732/72178 to actually
print the next N executed lines.
You simply cannot do that (easily) in gdb, because the execution trace of any program is not kept (and keeping it would be very expensive : it would slow down a lot the execution, and it would use a lot of resources - memory & disk space).
You can however use the backtrace or bt command of gdb to show the call stack, that is the current instruction pointer in the current function, the calling function, the calling function of the calling function, and so forth
BTW, if you really wanted it, you might script recent gdb using Python or Guile to give you such information. You certainly could keep the entire trace (by scripting around the step & backtrace functionalities).
I'm trying to test a scheduler that I wrote. I schedule two processes - both are infinite while loops (just while(1) statements). When I run the program sometimes it segfaults after like ten seconds (sometimes 5 sec, sometimes 15 or more). Sometimes it doesn't segfault at all and runs as expected. I have a log file which shows me that both processes are scheduled as expected before the segfault occurs. I'm trying to debug the errors using gdb but it's not being very helpful. Here's what I got with backtrace:
#0 0x00007ffff7ff1000 in ?? ()
#1 0x000000000000002b in ?? ()
#2 0x00007ffff78b984a in new_do_write () from /lib64/libc.so.6
#3 0x000000000061e3d0 in ?? ()
#4 0x0000000000000000 in ?? ()
I don't really understand #2.
I think this may be a stack overflow related error. However, I only malloc twice in the whole process - both times when I'm setting up the two processes, I malloc a pcb block in the pcb table I wrote. Has anyone run into similar issues before? Could this be something with how I'm setting/swapping the contexts in the scheduler? Why does it segfault sometimes, and sometimes not?
You didn't tell how you obtained the stack trace that you show in the question.
It is very likely that the stack trace is bogus not because the stack is corrupt, but because you've invoked GDB incorrectly, e.g. specified wrong executable when attaching the process or examining core dump.
One common mistake is to build the executable with -O2 (let's call this executable E1), then rebuild it with -g (let's call this E2) and try to analyze core of live process that is running E1 giving GDB E2 as the symbol file.
Don't do that, it doesn't work and isn't expected to work.
Since your stack seems corrupted, you're probably correct that you have a stack buffer overflow somewhere. Without the code, it's a little difficult to tell.
But this has nothing to do with your malloc calls. Overflowing dynamically allocated buffers would corrupt the heap, not the stack.
Whay you'll probably need to be looking at is local variables that aren't big enough for the data you're trying to copy in to them, like:
char xyzzy[5];
strcpy (xyzzy, "this is a bad idea";
Or passing a buffer (again, most likely on the stack) to a system call that writes more data to it than you provide for.
They're the most likely causes though theoretically, of course, any undefined behaviour on your part could cause this. If the solution is not evident based on this answer, you'll probably need to post the code that caused it. Try to ensure you trim it down as much as possible when you do that so that it's the shortest complete program that exhibits the bug.
Often you'll find by doing that, the problem becomes evident :-)
We have an embedded version of Linux kernel running on a MIPs core. The Programme we have written runs a particular test suite. During one of the stress tests (runs for about 12hrs) we get a seg fault. This in turn generates a core dump.
Unfortunately the core dump is not very useful. The crash is in some system library that is dynamically linked (probably pthread or glibc). The backtrace in the core dump is not helpful because it only shows the crash point and no other callers (our user space app is built with -g -O0, but still no back trace info):
Cannot access memory at address 0x2aab1004
(gdb) bt
#0 0x2ab05d18 in ?? ()
warning: GDB can't find the start of the function at 0x2ab05d18.
GDB is unable to find the start of the function at 0x2ab05d18
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
This problem is most likely caused by an invalid program counter or
stack pointer.
However, if you think GDB should simply search farther back
from 0x2ab05d18 for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
Another unfortunate-ness is that we cannot run gdb/gdbserver. gdb/gdbserver keeps breaking on __nptl_create_event. Seeing that the test creates threads, timers and destroys then every 5s it is almost impossible to sit for a long time hitting continue on them.
EDIT:
Another note, backtrace and backtrace_symbols is not supported on our toolchain.
Hence:
Is there a way of trapping seg fault and generate more backtrace data, stack pointers, call stack, etc.?
Is there a way of getting more data from a core dump that crashed in a .so file?
Thanks.
GDB can't find the start of the function at 0x2ab05d18
What is at that address at the time of the crash?
Do info shared, and find out if there is a library that contains that address.
The most likely cause of your troubles: did you run strip libpthread.so.0 before uploading it to your target? Don't do that: GDB requires libpthread.so.0 to not be stripped. If your toolchain contains libpthread.so.0 with debug symbols (and thus too large for the target), run strip -g on it, not a full strip.
Update:
info shared produced Cannot access memory at address 0x2ab05d18
This means that GDB can not access the shared library list (which would then explain the missing stack trace). The most usual cause: the binary that actually produced the core does not match the binary you gave to GDB. A less common cause: your core dump was truncated (perhaps due to ulimit -c being set too low).
If all else fails run the command using the debugger!
Just put "gdb" in form of your normal start command and enter "c"ontinue to get the process running. When the task segfaults it will return to the interactive gdb prompt rather than core dump. You should then be able to get more meaningful stack traces etc.
Another option is to use "truss" if it is available. This will tell you which system calls were being used at the time of the abend.
I'm doing a project that makes unix system calls. To be specific, my project heavily relies of calls to getcontext(), makecontext(), swapcontext() and setcontext(). I tried to debug my code using gdb. I stepped into the code line by line and examined the control but once a call to swapcontext() is made, it doesn't step into the code anymore. Instead, debugging almost stops there and the rest of the program is run automatically instead of line by line. I guess gdb does not go into context calls? Is there any way to get around this? Is there a debugger that i can use for this?
Thanks
setcontext and swapcontext calls change the program's stack, and gdb gets confused. I do not know whether some other debugger can handle it nicely.
gdb steps through one thread and calls this the current thread. Other threads will run as you are doing this. If you set a breakpoint that gets hit in a thread other than the current thread then gdb will change the current thread to that thread. Stepping is now relative to the new current thread.
Stepping with gdb over calls of swapcontext() with 'step' or 'next' does not work because not only the stackpointer changes but also the call returns to a different code line (which is the desired effect of swapcontext()). As gdb puts a breakpoint in the next code line which will not be executed until another swapcontext() returns to this place the execution will not break.
You need to foresee the line to which swapcontext() will return and set a breakpoint there. For a new (unused) context this will be the line you specified as entry function. For used contexts it will probably one of the lines after a swapcontext() there...
You can repeatedly use GDB's stepi command to first step into and then step through the swapcontext() function. You must step several dozen times, including a few steps over kernel system calls—I presume to save the floating point state?—and you'll eventually come out in the user thread you're swapping to. It's a tad time-consuming, but it works.
As much as you may not like this answer, the best bet is to step through your code in small chunks by hand. Threaded programs don't play very well with debuggers like GDB and Valgrind (at least in my experience) and most bugs can be determined by a careful step-by-step manual analysis of the code.