gdb bt gives only ??, how can I debug? - c

/var/log/message:
segfault at 0 ip 00007fcd16e5853a sp 00007ffd98e37e58 error 4 in libc-2.24.so[7fcd16dc9000+195000]
addr2line -e a.out 00007fcd16e5853a
??:0
gdb bt
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fcd16e5853a in ?? ()
(gdb) bt
#0 0x00007fcd16e5853a in ?? ()
#1 0x000055f2f45fe95b in ?? ()
#2 0x000055f200000080 in ?? ()
#3 0x00007fcd068c2040 in ?? ()
#4 0x000055f2f6109c48 in ?? ()
#5 0x0000000000000000 in ?? ()
build with gcc -Wall -O0 -g
How can I debug this, are there more methods?

gdb bt
Surely that is not the command you actually executed.
Most likely you did something like this:
gdb /path/to/core
(gdb) bt
Don't do that. Do this instead:
gdb /path/to/a.out /path/to/core
(gdb) bt
If you already did invoke GDB correctly, other likely reasons why bt did not work:
You are analyzing the core on a different machine from the one on which it was produced. See this answer.
You rebuilt a.out with different flags. Use the exact binary that crashed.
You have updated libc after the core was produced. Restore it to the version that was current as of when the core was produced.
P.S. This command
addr2line -e a.out 00007fcd16e5853a
makes no sense: the error message told you that the address 00007fcd16e5853a is in libc-2.24.so. The a.out has nothing to do with that address.
The command you want to use is:
addr2line -fe /path/to/libc-2.24.so 195000
P.P.S.
segfault at 0 ip 00007fcd16e5853a ...
This means: NULL pointer dereference inside libc. The most probable cause: not checking for error return, e.g. something like:
FILE *fp = fopen("/some/file", "r");
fscanf(fp, buffer, sizeof(buffer)); // Oops: didn't check for NULL.

Related

Segfault with GtkFileChooserDialogNative

I simply try to open a file through a dialog using Gtk3. This is the code I am using, no modifications at all:
https://docs.gtk.org/gtk3/class.FileChooserNative.html#typical-usage-gtkfilechoosernative-typical-usage
The window opens fine but when I try to open the file, it segfaults. According to gdb:
Thread 1 "a.out" received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
76 VPCMPEQ (%rdi), %ymm0, %ymm1
Using debuginfod of course. What am I doing wrong?
EDIT: Some backtrace from gdb:
(gdb) bt
#0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
#1 0x00007ffff7e1b948 in __GI__IO_puts (str=0x0) at ioputs.c:35
#2 0x00005555555551d6 in main (argc=1, argv=0x7fffffffdb28) at main.c:6

How can this code corrupt my stack trace?

This simple test program,
$ cat crash.c
int main() {
int x = 0;
*(&x + 5) = 10;
return 0;
}
Compiled with GCC 7.4.0,
$ gcc -O0 -g crash.c
Has an unexpected stack trace
$ ./a.out
Segmentation fault (core dumped)
$ gdb ./a.out /tmp/wk_cores/core-pid_19675.dump
Reading symbols from ./a.out...done.
[New LWP 19675]
Core was generated by `./a.out'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f450000000a in ?? ()
(gdb) bt
#0 0x00007f450000000a in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x00007fffd6f97598 in ?? ()
#3 0x0000000100008000 in ?? ()
#4 0x00005632be83d66a in frame_dummy ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
I don't understand why the stack doesn't show the invalid store to privileged memory? Can someone help me understand this?
I don't understand why the stack doesn't show the invalid store to privileged memory?
Because you didn't store anything to privileged memory.
To do that, you need to write way outside of stack, something like:
*(&x + 0x10000) = 5;
As is, your program does exhibit undefined behavior, but it doesn't write to "privileged" memory, just to memory that is writable, but that you should't write to.
x is the latest variable in your stack. So if you write at x+5, no matter how far you go, you always write in stack memory after the region of the current allocated stack. Therefore it always fail.

Inspecting caller frames with gdb

Suppose I have:
#include <stdlib.h>
int main()
{
int a = 2, b = 3;
if (a!=b)
abort();
}
Compiled with:
gcc -g c.c
Running this, I'll get a coredump (due to the SIGABRT raised by abort()), which I can debug with:
gdb a.out core
How can I get gdb to print the values of a and b from this context?
Here's the another way to specifically get a and b values by moving to the interested frame and then info locals would give you the values.
a.out was compiled with your code. (frame 2 is what you are interested in i.e., main()).
$ gdb ./a.out core
[ removed some not-so-interesting info here ]
Reading symbols from ./a.out...done.
[New LWP 14732]
Core was generated by `./a.out'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007fac16269f5d in __GI_abort () at abort.c:90
#2 0x00005592862f266d in main () at f.c:7
(gdb) frame 2
#2 0x00005592862f266d in main () at f.c:7
7 abort();
(gdb) info locals
a = 2
b = 3
(gdb) q
You can also use print once frame 2:
(gdb) print a
$1 = 2
(gdb) print b
$2 = 3
Did you compile with debug symbols -g? The command should be bt for backtrace, you can also use bt full for a full backtrace.
More infos: https://sourceware.org/gdb/onlinedocs/gdb/Backtrace.html

Why is it that gdb "can't compute CFA" when using a separate debug symbols file?

I'm trying to invoke gdb with a stripped executable and a separate debug symbols file, on a core dump generated from running the stripped executable.
But when I use the separate debug symbols file, gdb is unable to give information on local variables for me.
Here is a log showing entirely how I produce my 3 ELF files and the core file and then run them through gdb 3 times.
First I just run gdb with the stripped executable and of course can't see any file names or line numbers, and can't inspect variables.
Then I run gdb using the stripped executable and grabbing the debug symbols from the original unstripped executable. This works pretty well but does give a disturbing and apparently unwarranted warning about the core and executable possibly mismatching.
Finally I run gdb with the stripped executable and the separate debug file. This still gives filenames and line numbers, but I can't inspect local variables and I get a "can't compute CFA for this frame" error.
Here is the log:
2016-09-16 16:01:45 barry#somehost ~/proj/segfault/segfault
$ cat segfault.c
#include <stdio.h>
int main(int argc, char **argv) {
char *badpointer = (char *)0x2398723;
printf("badpointer: %s\n", badpointer);
return 0;
}
2016-09-16 16:03:31 barry#somehost ~/proj/segfault/segfault
$ gcc -g -o segfault segfault.c
2016-09-16 16:03:37 barry#somehost ~/proj/segfault/segfault
$ objcopy --strip-debug segfault segfault.stripped
2016-09-16 16:03:40 barry#somehost ~/proj/segfault/segfault
$ objcopy --only-keep-debug segfault segfault.debug
2016-09-16 16:03:43 barry#somehost ~/proj/segfault/segfault
$ ./segfault.stripped
Segmentation fault (core dumped)
2016-09-16 16:03:48 barry#somehost ~/proj/segfault/segfault
$ ll /tmp/core.segfault.stripp.11
-rw------- 1 barry bsm-it 188416 2016-09-16 16:03 /tmp/core.segfault.stripp.11
2016-09-16 16:03:51 barry#somehost ~/proj/segfault/segfault
$ gdb ./segfault.stripped /tmp/core.segfault.stripp.11
GNU gdb (GDB) Fedora (7.0.1-50.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/barry/proj/segfault/segfault/segfault.stripped...(no debugging symbols found)...done.
warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2 0x00000000004004f4 in main ()
(gdb) up
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2 0x00000000004004f4 in main ()
(gdb) p argc
No symbol table is loaded. Use the "file" command.
(gdb) q
2016-09-16 16:04:19 barry#somehost ~/proj/segfault/segfault
$ gdb -q -e ./segfault.stripped -s ./segfault -c /tmp/core.segfault.stripp.11
Reading symbols from /home/barry/proj/segfault/segfault/segfault...done.
warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2 0x00000000004004f4 in main (argc=1, argv=0x7fffd1c0a728) at segfault.c:4
(gdb) up
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2 0x00000000004004f4 in main (argc=1, argv=0x7fffd1c0a728) at segfault.c:4
4 printf("badpointer: %s\n", badpointer);
(gdb) p argc
$1 = 1
(gdb) q
2016-09-16 16:04:39 barry#somehost ~/proj/segfault/segfault
$ gdb -q -e ./segfault.stripped -s ./segfault.debug -c /tmp/core.segfault.stripp.11
Reading symbols from /home/barry/proj/segfault/segfault/segfault.debug...done.
warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2 0x00000000004004f4 in main (argc=can't compute CFA for this frame
) at segfault.c:4
(gdb) up
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2 0x00000000004004f4 in main (argc=can't compute CFA for this frame
) at segfault.c:4
4 printf("badpointer: %s\n", badpointer);
(gdb) p argc
can't compute CFA for this frame
(gdb) q
I have some questions about this:
Why does it display the warning "warning: core file may not match specified executable file.", even though I'm using the exact same executable path as was used when the core dump was originally generated?
Why does using the separate debug symbols (-s ./segfault.debug) result in the error "can't compute CFA for this frame" when attempting to inspect local variables?
What is a CFA anyway?
Am I using an incorrect method to product the debug symbol file?
I confirmed that using "objcopy --strip-debug" gives the same result as "strip -g".
Am I using the right options to feed the debug info into gdb?
My intention is that the stripped executables will be installed on a binary-compatible production system and any core dumps generated due to segfaults can be copied back to the devel system where we can feed them into gdb with the debug info and analyse the crash position and stack variables. But as a first step I'm trying to sort out the issues with using separate debug info files on the devel system.
It seems that using a separate debug symbols file causes the "can't compute CFA for this frame" error, even when a core file is not used.
My gcc version:
2016-09-16 16:07:39 barry#somehost ~/proj/segfault/segfault
$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)
I suspect that gdb might be looking for symbols related to the variables in the segfault.debug file when objcopy actually only put them in the segfault.stripped file. If this is the case, perhaps some small adjustment to the options to objcopy could put those symbols in the place gdb is looking?
I commend you for wanting to keep a set of symbol files for everything that is deployed to the production server; in my opinion this is an often overlooked practice, but you will not regret it -- one day it will save you a lot of debugging trouble.
As I have had similar issues in the past, I will try to answer some of your questions, although you have quite an ancient toolchain, if you don't mind me saying so, so I'm not sure how much that really applies here. I'll put up here anyway.
CFA = Canonical Frame Address. This is the base pointer to the stack frame that every local variable is addressed relative to. If you have done some traditional x86 assembly programming, the BP register was used for this. So "can't compute CFA for this frame" basically says "I know of these local variables, but I don't know where they are located on the stack".
There used to be code in GDB that worked only for the DWARF-2 debugging format, and non-conformance triggered this particular error at least. That restriction was lifted some time ago, but that change won't be in your version.
The other thing is there are debug information regarding how variables may be moved around is not always generated. This usually happens in newer compilers though, as they get better at optimizing.
I was able to get rid of my problems by compiling like this:
gcc -g3 -gdwarf-2 -fvar-tracking -fvar-tracking-assignments -o segfault segfault.c
you can try to see if this solves your problem, too.
Regarding the message about the location of the symbol file; it seems that the debugger wants to load it from the system directory. Maybe you have to link the executable to the symbol file with:
objcopy --add-gnu-debuglink=segfault.debug segfault
I found this question while searching for an answer to the following part of the original question:
Why does it display the warning "warning: core file may not match
specified executable file.", even though I'm using the exact same
executable path as was used when the core dump was originally
generated?
There was not an answer to this particular question but through experimentation and research I believe I have found the answer.
Below is a transcript of using gdb to debug a core file. Notice that the "warning: core file may not match specified executable file." error appears when the executable file that caused the core is greater than 15 characters in length.
[~/t]$cat do_abort.c
#include <stdlib.h>
int func4(int f) { if(f) {abort();} return 0;}
int func3(int f) { return func4(f); }
int func2(int f) { return func3(f); }
int func1(int f) { return func2(f); }
int main(void) { return func1(1); }
[~/t]$gcc -g -o 123456789012345 do_abort.c
[~/t]$./123456789012345
Aborted (core dumped)
[~/t]$ll core*
-rw-------. 1 dev wheel 240K Apr 22 03:19 core.42697
[~/t]$gdb -q -c core.42697 123456789012345
Reading symbols from /home/dev/t/123456789012345...done.
[New LWP 42697]
Core was generated by `./123456789012345'.
Program terminated with signal 6, Aborted.
#0 0x00007f0be67631d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 0x00007f0be67631d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f0be67648c8 in __GI_abort () at abort.c:90
#2 0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3 0x000000000040055f in func3 (f=1) at do_abort.c:4
#4 0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5 0x000000000040058d in func1 (f=1) at do_abort.c:6
#6 0x000000000040059d in main () at do_abort.c:7
(gdb) quit
[~/t]$rm core.42697
[~/t]$
[~/t]$mv 123456789012345 1234567890123456
[~/t]$./1234567890123456
Aborted (core dumped)
[~/t]$ll core*
-rw-------. 1 dev wheel 240K Apr 22 03:20 core.42721
[~/t]$gdb -q -c core.42721 1234567890123456
Reading symbols from /home/dev/t/1234567890123456...done.
warning: core file may not match specified executable file.
[New LWP 42721]
Core was generated by `./1234567890123456'.
Program terminated with signal 6, Aborted.
#0 0x00007f5b271fa1d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 0x00007f5b271fa1d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f5b271fb8c8 in __GI_abort () at abort.c:90
#2 0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3 0x000000000040055f in func3 (f=1) at do_abort.c:4
#4 0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5 0x000000000040058d in func1 (f=1) at do_abort.c:6
#6 0x000000000040059d in main () at do_abort.c:7
(gdb) quit
[~/t]$mv 1234567890123456 123456789012345
[~/t]$gdb -q -c core.42721 123456789012345
Reading symbols from /home/dev/t/123456789012345...done.
[New LWP 42721]
Core was generated by `./1234567890123456'.
Program terminated with signal 6, Aborted.
#0 0x00007f5b271fa1d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 0x00007f5b271fa1d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f5b271fb8c8 in __GI_abort () at abort.c:90
#2 0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3 0x000000000040055f in func3 (f=1) at do_abort.c:4
#4 0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5 0x000000000040058d in func1 (f=1) at do_abort.c:6
#6 0x000000000040059d in main () at do_abort.c:7
(gdb) quit
Following through the gdb source code I discovered that the ELF core file structure only reserves sixteen bytes to hold the executable filename, pr_fname[16], including the nul terminator (reference):
35 struct elf_external_linux_prpsinfo32_ugid32
36 {
37 char pr_state; /* Numeric process state. */
38 char pr_sname; /* Char for pr_state. */
39 char pr_zomb; /* Zombie. */
40 char pr_nice; /* Nice val. */
41 char pr_flag[4]; /* Flags. */
42 char pr_uid[4];
43 char pr_gid[4];
44 char pr_pid[4];
45 char pr_ppid[4];
46 char pr_pgrp[4];
47 char pr_sid[4];
48 char pr_fname[16]; /* Filename of executable. */
49 char pr_psargs[80]; /* Initial part of arg list. */
50 };
The "warning: core file may not match specified executable file." warning will be issued by gdb when the name of the executable passed on the command-line to gdb doesn't match the value stored in pr_fname[] in the core file (references here, here, and here).
Using the demonstration I showed at the start of this answer, when the filename is 1234567890123456 the filename stored in the core file as pr_fname[] is 123456789012345 (truncated to 15 characters). If gdb is started using gdb -c core.XXXX 1234567890123456 then the warning will be issued. If gdb is started using gdb -c core.XXXX 123456789012345 then the warning will not be issued.
It should follow that in the example from the original question, if segfault.stripped was renamed to segfault.stripp and gdb was run using gdb ./segfault.stripp /tmp/core.segfault.stripp.11 then the warning should not be issued.

What does this mean in gdb?

Program received signal SIGSEGV, Segmentation fault.
0x08049795 in execute_jobs ()
Current language: auto; currently asm
(gdb) info symbol 0x08049795
execute_jobs + 22 in section .text
(gdb) ptype 0x08049795
type = int
How to get the line number at which the error occurred?
Your binary was not compiled with debugging information. Rebuild with at least -g (or -ggdb, or -ggdb -g3, see GCC manual.)
The exact lines from GDB output:
(gdb) info symbol 0x08049795 execute_jobs + 22 in section .text
means that instruction at address 0x08049795, which is 22 bytes from beginning of function execute_jobs, generated the segmentation fault.
(gdb) ptype 0x08049795 type = int
Here you are asking for type of an integer, and GDB happily replies. Do
(gdb) x/10i 0x08049795
or
(gdb) disassemble execute_jobs
to see actual instructions.
The gdb command "bt" will show you a back trace. Unless you've corrupted the stack this should show the sequence of function calls that lead to the segfault. To get more meaningful information make sure that you've compiled your program with debug information by including -g on the gcc/g++ command line.

Resources