I have a project in C which I am compiling with gcc. It sometimes shows a segmentation fault when run and sometimes it doesn't. When run with gcc $ gdb -q ./build/program, it does not show any errors. After doing a bit of research, I found this question on stack overflow. Setting (gdb) set disable-randomization off does allow me to see the segmentation fault in gdb. However, I have no clue what address space randomization does and where should I look to find the problem. Is there a particular tool or a particular type of construct that I should look into?
I am also including the backtrace here for more context:
(gdb) set disable-randomization off
(gdb) run -n 10 -s 10
Starting program: /path/to/code/build/exact_diag_simulation -n 10 -s 10
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Exact Diagonalization
---------------------
len: 10 nospin: 0 coupling: 0.00e+00
disorder: 1.00e+01 hopping: 1.00e+00
Starting Simulation for Exact Diagonalization...
Run 1 started...[New Thread 0x7f4e34bff6c0 (LWP 12972)]
[New Thread 0x7f4e343fe6c0 (LWP 12973)]
[New Thread 0x7f4e33bfd6c0 (LWP 12974)]
[New Thread 0x7f4e333fc6c0 (LWP 12975)]
[New Thread 0x7f4e32bfb6c0 (LWP 12976)]
[New Thread 0x7f4e323fa6c0 (LWP 12977)]
[New Thread 0x7f4e31bf96c0 (LWP 12978)]
Thread 1 "exact_diag_simu" received signal SIGSEGV, Segmentation fault.
0x00007f4e75934a02 in zgemv_n_SKYLAKEX () from /usr/lib/libblas.so.3
(gdb) bt
#0 0x00007f4e75934a02 in zgemv_n_SKYLAKEX () from /usr/lib/libblas.so.3
#1 0x00007f4e74ff550c in zgemv_ () from /usr/lib/libblas.so.3
#2 0x00007f4e76772142 in zlatrd_ () from /usr/lib/liblapack.so.3
#3 0x00007f4e766f3ba0 in zhetrd_ () from /usr/lib/liblapack.so.3
#4 0x00007f4e766ea105 in zheev_ () from /usr/lib/liblapack.so.3
#5 0x00007f4e77195e07 in LAPACKE_zheev_work () from /usr/lib/liblapacke.so.3
#6 0x00007f4e77195fba in LAPACKE_zheev () from /usr/lib/liblapacke.so.3
#7 0x00005612184a3287 in utils_get_eigh (matrix=0x7f4e3135c010, size=200, eigvals=0x561218f1a0c0)
at src/utils/utils.c:273
#8 0x00005612184a27c4 in run (params=0x7fffa9f5ffd0, create_neighbours=1, gfunc=0x7f4e74ee1010)
at src/exact_diag_simulation.c:155
#9 0x00005612184a2577 in main (argc=5, argv=0x7fffa9f60348) at src/exact_diag_simulation.c:103
I have compiled with the following flags in my makefile:
CFLAGS=-Wall -Wextra -g -fdiagnostics-color=always -fopenmp -ffast-math -fsanitize=address,undefined
LFLAGS=-llapacke -lm -lgsl -lcblas
Related
Suppose I have:
#include <stdlib.h>
int main()
{
int a = 2, b = 3;
if (a!=b)
abort();
}
Compiled with:
gcc -g c.c
Running this, I'll get a coredump (due to the SIGABRT raised by abort()), which I can debug with:
gdb a.out core
How can I get gdb to print the values of a and b from this context?
Here's the another way to specifically get a and b values by moving to the interested frame and then info locals would give you the values.
a.out was compiled with your code. (frame 2 is what you are interested in i.e., main()).
$ gdb ./a.out core
[ removed some not-so-interesting info here ]
Reading symbols from ./a.out...done.
[New LWP 14732]
Core was generated by `./a.out'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007fac16269f5d in __GI_abort () at abort.c:90
#2 0x00005592862f266d in main () at f.c:7
(gdb) frame 2
#2 0x00005592862f266d in main () at f.c:7
7 abort();
(gdb) info locals
a = 2
b = 3
(gdb) q
You can also use print once frame 2:
(gdb) print a
$1 = 2
(gdb) print b
$2 = 3
Did you compile with debug symbols -g? The command should be bt for backtrace, you can also use bt full for a full backtrace.
More infos: https://sourceware.org/gdb/onlinedocs/gdb/Backtrace.html
I'm trying to invoke gdb with a stripped executable and a separate debug symbols file, on a core dump generated from running the stripped executable.
But when I use the separate debug symbols file, gdb is unable to give information on local variables for me.
Here is a log showing entirely how I produce my 3 ELF files and the core file and then run them through gdb 3 times.
First I just run gdb with the stripped executable and of course can't see any file names or line numbers, and can't inspect variables.
Then I run gdb using the stripped executable and grabbing the debug symbols from the original unstripped executable. This works pretty well but does give a disturbing and apparently unwarranted warning about the core and executable possibly mismatching.
Finally I run gdb with the stripped executable and the separate debug file. This still gives filenames and line numbers, but I can't inspect local variables and I get a "can't compute CFA for this frame" error.
Here is the log:
2016-09-16 16:01:45 barry#somehost ~/proj/segfault/segfault
$ cat segfault.c
#include <stdio.h>
int main(int argc, char **argv) {
char *badpointer = (char *)0x2398723;
printf("badpointer: %s\n", badpointer);
return 0;
}
2016-09-16 16:03:31 barry#somehost ~/proj/segfault/segfault
$ gcc -g -o segfault segfault.c
2016-09-16 16:03:37 barry#somehost ~/proj/segfault/segfault
$ objcopy --strip-debug segfault segfault.stripped
2016-09-16 16:03:40 barry#somehost ~/proj/segfault/segfault
$ objcopy --only-keep-debug segfault segfault.debug
2016-09-16 16:03:43 barry#somehost ~/proj/segfault/segfault
$ ./segfault.stripped
Segmentation fault (core dumped)
2016-09-16 16:03:48 barry#somehost ~/proj/segfault/segfault
$ ll /tmp/core.segfault.stripp.11
-rw------- 1 barry bsm-it 188416 2016-09-16 16:03 /tmp/core.segfault.stripp.11
2016-09-16 16:03:51 barry#somehost ~/proj/segfault/segfault
$ gdb ./segfault.stripped /tmp/core.segfault.stripp.11
GNU gdb (GDB) Fedora (7.0.1-50.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/barry/proj/segfault/segfault/segfault.stripped...(no debugging symbols found)...done.
warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2 0x00000000004004f4 in main ()
(gdb) up
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2 0x00000000004004f4 in main ()
(gdb) p argc
No symbol table is loaded. Use the "file" command.
(gdb) q
2016-09-16 16:04:19 barry#somehost ~/proj/segfault/segfault
$ gdb -q -e ./segfault.stripped -s ./segfault -c /tmp/core.segfault.stripp.11
Reading symbols from /home/barry/proj/segfault/segfault/segfault...done.
warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2 0x00000000004004f4 in main (argc=1, argv=0x7fffd1c0a728) at segfault.c:4
(gdb) up
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2 0x00000000004004f4 in main (argc=1, argv=0x7fffd1c0a728) at segfault.c:4
4 printf("badpointer: %s\n", badpointer);
(gdb) p argc
$1 = 1
(gdb) q
2016-09-16 16:04:39 barry#somehost ~/proj/segfault/segfault
$ gdb -q -e ./segfault.stripped -s ./segfault.debug -c /tmp/core.segfault.stripp.11
Reading symbols from /home/barry/proj/segfault/segfault/segfault.debug...done.
warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0 0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2 0x00000000004004f4 in main (argc=can't compute CFA for this frame
) at segfault.c:4
(gdb) up
#1 0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2 0x00000000004004f4 in main (argc=can't compute CFA for this frame
) at segfault.c:4
4 printf("badpointer: %s\n", badpointer);
(gdb) p argc
can't compute CFA for this frame
(gdb) q
I have some questions about this:
Why does it display the warning "warning: core file may not match specified executable file.", even though I'm using the exact same executable path as was used when the core dump was originally generated?
Why does using the separate debug symbols (-s ./segfault.debug) result in the error "can't compute CFA for this frame" when attempting to inspect local variables?
What is a CFA anyway?
Am I using an incorrect method to product the debug symbol file?
I confirmed that using "objcopy --strip-debug" gives the same result as "strip -g".
Am I using the right options to feed the debug info into gdb?
My intention is that the stripped executables will be installed on a binary-compatible production system and any core dumps generated due to segfaults can be copied back to the devel system where we can feed them into gdb with the debug info and analyse the crash position and stack variables. But as a first step I'm trying to sort out the issues with using separate debug info files on the devel system.
It seems that using a separate debug symbols file causes the "can't compute CFA for this frame" error, even when a core file is not used.
My gcc version:
2016-09-16 16:07:39 barry#somehost ~/proj/segfault/segfault
$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)
I suspect that gdb might be looking for symbols related to the variables in the segfault.debug file when objcopy actually only put them in the segfault.stripped file. If this is the case, perhaps some small adjustment to the options to objcopy could put those symbols in the place gdb is looking?
I commend you for wanting to keep a set of symbol files for everything that is deployed to the production server; in my opinion this is an often overlooked practice, but you will not regret it -- one day it will save you a lot of debugging trouble.
As I have had similar issues in the past, I will try to answer some of your questions, although you have quite an ancient toolchain, if you don't mind me saying so, so I'm not sure how much that really applies here. I'll put up here anyway.
CFA = Canonical Frame Address. This is the base pointer to the stack frame that every local variable is addressed relative to. If you have done some traditional x86 assembly programming, the BP register was used for this. So "can't compute CFA for this frame" basically says "I know of these local variables, but I don't know where they are located on the stack".
There used to be code in GDB that worked only for the DWARF-2 debugging format, and non-conformance triggered this particular error at least. That restriction was lifted some time ago, but that change won't be in your version.
The other thing is there are debug information regarding how variables may be moved around is not always generated. This usually happens in newer compilers though, as they get better at optimizing.
I was able to get rid of my problems by compiling like this:
gcc -g3 -gdwarf-2 -fvar-tracking -fvar-tracking-assignments -o segfault segfault.c
you can try to see if this solves your problem, too.
Regarding the message about the location of the symbol file; it seems that the debugger wants to load it from the system directory. Maybe you have to link the executable to the symbol file with:
objcopy --add-gnu-debuglink=segfault.debug segfault
I found this question while searching for an answer to the following part of the original question:
Why does it display the warning "warning: core file may not match
specified executable file.", even though I'm using the exact same
executable path as was used when the core dump was originally
generated?
There was not an answer to this particular question but through experimentation and research I believe I have found the answer.
Below is a transcript of using gdb to debug a core file. Notice that the "warning: core file may not match specified executable file." error appears when the executable file that caused the core is greater than 15 characters in length.
[~/t]$cat do_abort.c
#include <stdlib.h>
int func4(int f) { if(f) {abort();} return 0;}
int func3(int f) { return func4(f); }
int func2(int f) { return func3(f); }
int func1(int f) { return func2(f); }
int main(void) { return func1(1); }
[~/t]$gcc -g -o 123456789012345 do_abort.c
[~/t]$./123456789012345
Aborted (core dumped)
[~/t]$ll core*
-rw-------. 1 dev wheel 240K Apr 22 03:19 core.42697
[~/t]$gdb -q -c core.42697 123456789012345
Reading symbols from /home/dev/t/123456789012345...done.
[New LWP 42697]
Core was generated by `./123456789012345'.
Program terminated with signal 6, Aborted.
#0 0x00007f0be67631d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 0x00007f0be67631d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f0be67648c8 in __GI_abort () at abort.c:90
#2 0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3 0x000000000040055f in func3 (f=1) at do_abort.c:4
#4 0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5 0x000000000040058d in func1 (f=1) at do_abort.c:6
#6 0x000000000040059d in main () at do_abort.c:7
(gdb) quit
[~/t]$rm core.42697
[~/t]$
[~/t]$mv 123456789012345 1234567890123456
[~/t]$./1234567890123456
Aborted (core dumped)
[~/t]$ll core*
-rw-------. 1 dev wheel 240K Apr 22 03:20 core.42721
[~/t]$gdb -q -c core.42721 1234567890123456
Reading symbols from /home/dev/t/1234567890123456...done.
warning: core file may not match specified executable file.
[New LWP 42721]
Core was generated by `./1234567890123456'.
Program terminated with signal 6, Aborted.
#0 0x00007f5b271fa1d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 0x00007f5b271fa1d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f5b271fb8c8 in __GI_abort () at abort.c:90
#2 0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3 0x000000000040055f in func3 (f=1) at do_abort.c:4
#4 0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5 0x000000000040058d in func1 (f=1) at do_abort.c:6
#6 0x000000000040059d in main () at do_abort.c:7
(gdb) quit
[~/t]$mv 1234567890123456 123456789012345
[~/t]$gdb -q -c core.42721 123456789012345
Reading symbols from /home/dev/t/123456789012345...done.
[New LWP 42721]
Core was generated by `./1234567890123456'.
Program terminated with signal 6, Aborted.
#0 0x00007f5b271fa1d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 0x00007f5b271fa1d7 in __GI_raise (sig=sig#entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f5b271fb8c8 in __GI_abort () at abort.c:90
#2 0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3 0x000000000040055f in func3 (f=1) at do_abort.c:4
#4 0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5 0x000000000040058d in func1 (f=1) at do_abort.c:6
#6 0x000000000040059d in main () at do_abort.c:7
(gdb) quit
Following through the gdb source code I discovered that the ELF core file structure only reserves sixteen bytes to hold the executable filename, pr_fname[16], including the nul terminator (reference):
35 struct elf_external_linux_prpsinfo32_ugid32
36 {
37 char pr_state; /* Numeric process state. */
38 char pr_sname; /* Char for pr_state. */
39 char pr_zomb; /* Zombie. */
40 char pr_nice; /* Nice val. */
41 char pr_flag[4]; /* Flags. */
42 char pr_uid[4];
43 char pr_gid[4];
44 char pr_pid[4];
45 char pr_ppid[4];
46 char pr_pgrp[4];
47 char pr_sid[4];
48 char pr_fname[16]; /* Filename of executable. */
49 char pr_psargs[80]; /* Initial part of arg list. */
50 };
The "warning: core file may not match specified executable file." warning will be issued by gdb when the name of the executable passed on the command-line to gdb doesn't match the value stored in pr_fname[] in the core file (references here, here, and here).
Using the demonstration I showed at the start of this answer, when the filename is 1234567890123456 the filename stored in the core file as pr_fname[] is 123456789012345 (truncated to 15 characters). If gdb is started using gdb -c core.XXXX 1234567890123456 then the warning will be issued. If gdb is started using gdb -c core.XXXX 123456789012345 then the warning will not be issued.
It should follow that in the example from the original question, if segfault.stripped was renamed to segfault.stripp and gdb was run using gdb ./segfault.stripp /tmp/core.segfault.stripp.11 then the warning should not be issued.
I'm using gdb-7.11.1 and I get this message on my embedded powerpc system. Some more background, the libpthread I use has been stripped off all the non-dynamic symbols, including nptl_version, which libthread_db uses to make sure it is compatible with libpthread.
Coming to my problem, gdb says it won't be able to debug threads, but it seemingly can as evidenced below. Am I simply misunderstanding what 'thread debugging' means? (The ?? you see are naturally due to the missing symbol table in libpthread)
(gdb) break fn2
Breakpoint 1 at 0x1000052c: file test.c, line 7.
(gdb) run
Starting program: /tmp/test
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
[New LWP 21312]
[New LWP 21313]
[New LWP 21314]
[New LWP 21315]
[New LWP 21316]
[New LWP 21317]
[Switching to LWP 21315]
Thread 5 hit Breakpoint 1, fn2 () at test.c:7
7 test.c: No such file or directory.
(gdb) thread apply all bt
Thread 7 (LWP 21317):
#0 0x0fdcf030 in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 6 (LWP 21316):
#0 0x0fdcf030 in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 5 (LWP 21315):
#0 fn2 () at test.c:7
#1 0x0fdc6d8c in ?? () from /lib/libpthread.so.0
#2 0x0fd26074 in clone () from /lib/libc.so.6
Thread 4 (LWP 21314):
#0 0x0fdcf030 in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 3 (LWP 21313):
#0 0x0fdcf030 in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 2 (LWP 21312):
#0 0x0fdcefdc in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 1 (LWP 21309):
#0 0x0fd26038 in clone () from /lib/libc.so.6
#1 0x0fdc5f2c in ?? () from /lib/libpthread.so.0
#2 0x0fde6150 in ?? () from /lib/libpthread.so.0
#3 0x0fdc6424 in pthread_create () from /lib/libpthread.so.0
#4 0x100006a4 in main () at test.c:23
(gdb)
On Linux (at least, and others), an important part of the threading library is implemented in the kernel: that the "kernel-thread", called LWPs (for light-weight process).
GDB doesn't need libthread_db help to track them, as the OS itself can give the information the key information about them: their CPU registers (mainly IP, SP, FP).
I'm not sure what libthread_db provides in that context. The only thing I can think of is the LWP <-> Thread id mapping:
* 3 Thread 0x7ffff6d19700 (LWP 21571) "erato" primes_computer_runner2 (param=0x7fffffffca50) at erato.c:46
1 Thread 0x7ffff7fad700 (LWP 21565) "erato" 0x00007ffff7bc568d in pthread_join () from /usr/lib/libpthread.so.0
(gdb) print/x thread_handle
$1 = 0x7ffff6d19700
See, Thread 0x7ffff7fad700 maps to LWP 21565.
In comparison, without libthread_db it just gives the LWP id (in another run):
* 3 LWP 22060 "erato" primes_computer_runner2 (param=0x7fffffffca50) at erato.c:46
1 LWP 22058 "erato" 0x00007ffff76037b1 in clone () from /usr/lib/libc.so.6
If you want further details about pthread_db purpose, and why it's mandatory (or something equivalent) for user and hybrid threading libraries, you can take a look at this article I wrote several years ago:
User Level DB: a Debugging API for User-Level Thread Libraries
The common cause for this error message:
Unable to find libthread_db matching inferior's thread library, ...
is having libpthread.so.0 that is fully stripped. Don't do that.
In particular, libthread_db.so needs nptl_version (local) symbol. You can verify whether your libpthread.so.0 has it with:
nm /path/to/libpthread.so.0 | grep version
which should produce something like:
0000000000012cc6 r nptl_version
I've been messing around with libspotify for a while now but I get a segmentation fault every once in a while when I call the sp_session_logout() API.
I did a test that does the following loop:
Create a session
Login
Logout
Release the session
It usually crashes after 3-4 iterations.
The code and the stdout can be found here:
https://gist.github.com/jonas-lundqvist/4bb759a1f1ed09de46a1
Compiled with:
gcc -std=gnu99 -g -Wall `pkg-config --cflags libspotify` -o test_login_logout login_logout.c appkey.c cred.c -lpthread `pkg-config --libs libspotify`
I noticed however that two threads was created in the start of each iteration, but only one exited.
This is the output from gdb
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7fbf700 (LWP 31308)]
[New Thread 0x7ffff7f3e700 (LWP 31309)]
[Thread 0x7ffff7f3e700 (LWP 31309) exited]
[New Thread 0x7ffff7f3e700 (LWP 31310)]
[New Thread 0x7ffff7ebd700 (LWP 31311)]
[Thread 0x7ffff7ebd700 (LWP 31311) exited]
[New Thread 0x7ffff7ebd700 (LWP 31313)]
[New Thread 0x7ffff6638700 (LWP 31314)]
[Thread 0x7ffff6638700 (LWP 31314) exited]
[New Thread 0x7ffff6638700 (LWP 31315)]
[New Thread 0x7ffff65b7700 (LWP 31316)]
Program received signal SIGSEGV, Segmentation fault.
Inspecting the threads I can see:
Id Target Id Frame
9 Thread 0x7ffff65b7700 (LWP 31316) "Network Thread" 0x00007ffff744b50d in poll () at ../sysdeps/unix/syscall-template.S:81
8 Thread 0x7ffff6638700 (LWP 31315) "Dns Thread" 0x00007ffff744b50d in poll () at ../sysdeps/unix/syscall-template.S:81
6 Thread 0x7ffff7ebd700 (LWP 31313) "Dns Thread" sem_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
4 Thread 0x7ffff7f3e700 (LWP 31310) "Dns Thread" sem_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
2 Thread 0x7ffff7fbf700 (LWP 31308) "Dns Thread" sem_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
* 1 Thread 0x7ffff7fc1700 (LWP 31304) "test_login_logo" 0x00007ffff77eb5ac in ?? () from /home/jonas/lib/libspotify/lib/libspotify.so.12
And if I pick on of the threds created in one of the iterations I can see this:
[Switching to thread 4 (Thread 0x7ffff7f3e700 (LWP 31310))]
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85 ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007ffff780c1f8 in ?? () from /home/jonas/lib/libspotify/lib/libspotify.so.12
#2 0x00007ffff77c608d in ?? () from /home/jonas/lib/libspotify/lib/libspotify.so.12
#3 0x00007ffff7bc70a4 in start_thread (arg=0x7ffff7f3e700) at pthread_create.c:309
#4 0x00007ffff745404d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Unfortunatley the libspotify development seems quite dead, but does anybody know if there is a work around?
I need to be able to do multiple sp_session_create()/sp_session_release() in my application...
Edit: bt of the crashing thread (from another, yet similar, run)
#0 0x00007ffff77eb5ac in ?? () from /home/jonas/lib/libspotify/lib/libspotify.so.12
#1 0x00007ffff7860b5a in ?? () from /home/jonas/lib/libspotify/lib/libspotify.so.12
#2 0x00007ffff781683f in ?? () from /home/jonas/lib/libspotify/lib/libspotify.so.12
#3 0x00007ffff7817141 in ?? () from /home/jonas/lib/libspotify/lib/libspotify.so.12
#4 0x00007ffff78add7d in sp_session_logout () from /home/jonas/lib/libspotify/lib/libspotify.so.12
#5 0x0000000000400d51 in main (argc=1, argv=0x7fffffffe3a8) at login_logout.c:99
Does anyone know a way to go from a pthread_t to what GDB displays with info threads?
So I have:
(gdb) info threads
37 Thread 22887 0xb7704422 in __kernel_vsyscall ()
36 Thread 22926 0xb7704422 in __kernel_vsyscall ()
35 Thread 22925 0xb7704422 in __kernel_vsyscall ()
34 Thread 22924 0xb7704422 in __kernel_vsyscall ()
33 Thread 22922 0xb7704422 in __kernel_vsyscall ()
32 Thread 22921 0xb7704422 in __kernel_vsyscall ()
(gdb) p m_messageQueue->m_creationThread
$3 = 2694822768
(gdb) p/x m_messageQueue->m_creationThread
$4 = 0xa09fbb70
Does anyone know how I figure out which thread this is? It would appear to be 22768, but none of my threads go that low.
New versions of GDB actually output the value of pthread_t in the info thread, making association of pthread_t with thread number trivial.
For example, using GDB 7.0:
cat t.c
#include <pthread.h>
void *fn(void *p)
{
sleep(180);
}
int main()
{
pthread_t pth1, pth2;
pthread_create(&pth1, 0, fn, 0);
pthread_create(&pth2, 0, fn, 0);
pthread_join(pth1, 0);
return 0;
}
gcc -g -m32 -pthread t.c && gdb -q ../a.out
(gdb) r
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0xf7e56b90 (LWP 25343)]
[New Thread 0xf7655b90 (LWP 25344)]
Program received signal SIGINT, Interrupt.
0xffffe405 in __kernel_vsyscall ()
(gdb) info thread
3 Thread 0xf7655b90 (LWP 25344) 0xffffe405 in __kernel_vsyscall ()
2 Thread 0xf7e56b90 (LWP 25343) 0xffffe405 in __kernel_vsyscall ()
* 1 Thread 0xf7e576b0 (LWP 25338) 0xffffe405 in __kernel_vsyscall ()
(gdb) up 2
#2 0x080484e2 in main () at t.c:13
13 pthread_join(pth1, 0);
(gdb) p/x pth1
$1 = 0xf7e56b90 ## this is thread #2 above
(gdb) p/x pth2
$2 = 0xf7655b90 ## this is thread #3 above
The value of pthread_t is not the same as that thread's system dependent thread id (in Linux gettid(2)) which you see in GDB.
AFAIK, there isn't any function to convert between the two. You need to keep track of that yourself.