gdb: thread debugging will not be available? - c

Question

I'm using gdb-7.11.1 and I get this message on my embedded powerpc system. Some more background, the libpthread I use has been stripped off all the non-dynamic symbols, including nptl_version, which libthread_db uses to make sure it is compatible with libpthread.
Coming to my problem, gdb says it won't be able to debug threads, but it seemingly can as evidenced below. Am I simply misunderstanding what 'thread debugging' means? (The ?? you see are naturally due to the missing symbol table in libpthread)
(gdb) break fn2
Breakpoint 1 at 0x1000052c: file test.c, line 7.
(gdb) run
Starting program: /tmp/test
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
[New LWP 21312]
[New LWP 21313]
[New LWP 21314]
[New LWP 21315]
[New LWP 21316]
[New LWP 21317]
[Switching to LWP 21315]
Thread 5 hit Breakpoint 1, fn2 () at test.c:7
7 test.c: No such file or directory.
(gdb) thread apply all bt
Thread 7 (LWP 21317):
#0 0x0fdcf030 in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 6 (LWP 21316):
#0 0x0fdcf030 in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 5 (LWP 21315):
#0 fn2 () at test.c:7
#1 0x0fdc6d8c in ?? () from /lib/libpthread.so.0
#2 0x0fd26074 in clone () from /lib/libc.so.6
Thread 4 (LWP 21314):
#0 0x0fdcf030 in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 3 (LWP 21313):
#0 0x0fdcf030 in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 2 (LWP 21312):
#0 0x0fdcefdc in ?? () from /lib/libpthread.so.0
#1 0x0fdc892c in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Thread 1 (LWP 21309):
#0 0x0fd26038 in clone () from /lib/libc.so.6
#1 0x0fdc5f2c in ?? () from /lib/libpthread.so.0
#2 0x0fde6150 in ?? () from /lib/libpthread.so.0
#3 0x0fdc6424 in pthread_create () from /lib/libpthread.so.0
#4 0x100006a4 in main () at test.c:23
(gdb)

Answer 1

On Linux (at least, and others), an important part of the threading library is implemented in the kernel: that the "kernel-thread", called LWPs (for light-weight process).
GDB doesn't need libthread_db help to track them, as the OS itself can give the information the key information about them: their CPU registers (mainly IP, SP, FP).
I'm not sure what libthread_db provides in that context. The only thing I can think of is the LWP <-> Thread id mapping:
* 3 Thread 0x7ffff6d19700 (LWP 21571) "erato" primes_computer_runner2 (param=0x7fffffffca50) at erato.c:46
1 Thread 0x7ffff7fad700 (LWP 21565) "erato" 0x00007ffff7bc568d in pthread_join () from /usr/lib/libpthread.so.0
(gdb) print/x thread_handle
$1 = 0x7ffff6d19700
See, Thread 0x7ffff7fad700 maps to LWP 21565.
In comparison, without libthread_db it just gives the LWP id (in another run):
* 3 LWP 22060 "erato" primes_computer_runner2 (param=0x7fffffffca50) at erato.c:46
1 LWP 22058 "erato" 0x00007ffff76037b1 in clone () from /usr/lib/libc.so.6
If you want further details about pthread_db purpose, and why it's mandatory (or something equivalent) for user and hybrid threading libraries, you can take a look at this article I wrote several years ago:
User Level DB: a Debugging API for User-Level Thread Libraries

Answer 2

The common cause for this error message:
Unable to find libthread_db matching inferior's thread library, ...
is having libpthread.so.0 that is fully stripped. Don't do that.
In particular, libthread_db.so needs nptl_version (local) symbol. You can verify whether your libpthread.so.0 has it with:
nm /path/to/libpthread.so.0 | grep version
which should produce something like:
0000000000012cc6 r nptl_version