I am writing a plugin for audacious, and I am experiencing random segfaults. I looked around and I found that I can process the program's core dumps with gdb.
So I did that, and I got this output:
http://pastebin.com/m7d0d663d
As you can see, it says no debugging symbols where found anywhere. I want to compile audacious with debugging symbols, but I am not sure how. I tried editing configure, which only includes a file named buildsys.mk, so I edited that and removed the -s flag from the linker, and made sure that the -g flag is passed to the compiler. The gdb output above is after I did that, so apparently what I did had no effect.
So how can I retain debugging symbols when compiling audacious? The problem is that I am only writing a small plugin, and haven't got a grasp of the while audacious code.
UPDATE: I added debugging symbols for gtk+ and glib (and also tried the CFLAGS=-g option), and I got a couple of coredumps analyzed. The bottom line is this:
(gdb) bt
#0 gtk_text_iter_make_real (_iter=<value optimized out>) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c:202
#1 0xb7c1cf5e in _gtk_text_iter_get_any_segment (iter=0x0) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c:474
#2 0xb7c24cd6 in IA__gtk_text_layout_get_line_display (layout=0x93a4318, line=0x9af6270, size_only=1) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextlayout.c:2196
#3 0xb7c29172 in gtk_text_layout_real_wrap (layout=0x93a4318, line=0x9af6270, line_data=0xb10036b8) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextlayout.c:1147
#4 0xb7c2358f in IA__gtk_text_layout_wrap (layout=0x93a4318, line=0x9af6270, line_data=0x0) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextlayout.c:693
#5 0xb7c060a1 in _gtk_text_btree_validate_line (tree=0x9407370, line=0x9af6270, view_id=0x93a4318) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextbtree.c:5422
#6 0xb7c27dc1 in IA__gtk_text_layout_validate_yrange (layout=0x93a4318, anchor=0xbfb0e624, y0=0, y1=635) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextlayout.c:1062
#7 0xb7c34999 in gtk_text_view_validate_onscreen (text_view=0x9406000) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextview.c:3502
#8 0xb7c35f85 in gtk_text_view_flush_first_validate (text_view=0x9406000) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextview.c:3558
#9 0xb7c35fde in first_validate_callback (data=0x9406000) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextview.c:3577
#10 0xb79c88fb in gdk_threads_dispatch (data=0x9bce910) at /build/buildd/gtk+2.0-2.16.1/gdk/gdk.c:498
#11 0xb7e38c81 in g_idle_dispatch (source=0x938a400, callback=0, user_data=0x9bce910) at /build/buildd/glib2.0-2.20.1/glib/gmain.c:3922
#12 0xb7e3ab88 in IA__g_main_context_dispatch (context=0x9250760) at /build/buildd/glib2.0-2.20.1/glib/gmain.c:1814
#13 0xb7e3e0eb in g_main_context_iterate (context=0x9250760, block=1, dispatch=1, self=0x92333e8) at /build/buildd/glib2.0-2.20.1/glib/gmain.c:2448
#14 0xb7e3e5ba in IA__g_main_loop_run (loop=0x9a92c88) at /build/buildd/glib2.0-2.20.1/glib/gmain.c:2656
#15 0xb7b707d9 in IA__gtk_main () at /build/buildd/gtk+2.0-2.16.1/gtk/gtkmain.c:1205
#16 0xb268d56a in skins_init () from /usr/local/lib/audacious/General/skins.so
#17 0x0805b42a in ?? ()
#18 0xb7540775 in __libc_start_main () from /lib/tls/i686/cmov/libc.so.6
#19 0x08055361 in ?? ()
(gdb)
And the exact error is:
#0 gtk_text_iter_make_real (_iter=<value optimized out>) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c:202
202 /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c: No such file or directory.
in /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c
The exact line is this:
if (iter->segments_changed_stamp !=
Can anyone make anything out of this? :-\
While it's true that doing either:
./configure CFLAGS=-g && make && make install
or
make CFLAGS=-g
will get you a build with debug symbols, this is quite unlikely to help you solve the problem.
You program crashed in /usr/lib/libgtk-x11-2.0.so.0, not in audacious2 (whatever that is). You are also analyzing the core incorrectly: list doesn't make sense at that stage. Your very first (gdb) command when analyzing a core should almost always be where, followed by thread apply all where.
You might also get better results from installing libgtk2-debuginfo or some such package, which should provide debug info for libgtk-x11-2.0.so.0, and may allow you to see the source and variables in libgtk at crash point.
As far as I know, audacious uses autotools. No need to modify anything, just configure with:
CFLAGS="-g $CFLAGS" ./configure
followed by the usual steps to install it. The flags are stored (in config.status I think), so any subsequent call to make will build a debug-enabled audacious.
Related
I'm having some issues with enabling the res_srtp module in Asterisk. Every attempt results in this not-so-helpful error message:
WARNING[47044] res_srtp.c: Failed to initialize libsrtp
ERROR[47044] loader.c: *** Failed to load module res_srtp.so
ERROR[47044] asterisk.c: Module initialization failed. ASTERISK EXITING!
I'd like to fix this and get it running, so now we get into the programming side of things. I'm not a C programmer by trade, but I was able to use my limited gdb skills to trace the error back through libsrtp's crypto_kernel_init() function. The error occurs when trying to enable the AES-GCM-128 cipher.
I was only able to get as far as PK11_Encrypt() in the NSS library, where I was unable to step into this line of code. I think this is because (again, very limited C knowledge) it's a macro and not a true function?
crv = PK11_GETTAB(slot)->C_EncryptInit(session, &mech, symKey->objectID);
// returns 113
So my question is either, how can I proceed with debugging this to a point where I can file a bug report with someone, or (preferably) has anyone got libsrtp working in this environment? There were very few other reports of similar problems, likely because EL9 is not in wide use yet.
My distro (AlmaLinux) is running NSS 3.71, and I've tried updating to NSS 3.79 with no change.
Here's the backtrace from where I was able to get to, if it's of any help.
#0 PK11_Encrypt (symKey=0x555555ea1a00, mechanism=mechanism#entry=4231, param=param#entry=0x7fffffffc9a0,
out=out#entry=0x7fffffffcb50 "\331\061\062%\370\204\006\345\245Y\tů\365&\232\206\247\251S\025\064\367\332.L0=\212\061\212r\034<\f\225\225h\tS/\317\016$I\246\265%\261j\355\365\252\r\346W\272c{9", outLen=outLen#entry=0x7fffffffca3c, maxLen=<optimized out>,
data=0x7fffffffcb50 "\331\061\062%\370\204\006\345\245Y\tů\365&\232\206\247\251S\025\064\367\332.L0=\212\061\212r\034<\f\225\225h\tS/\317\016$I\246\265%\261j\355\365\252\r\346W\272c{9", dataLen=60) at ../pk11wrap/pk11obj.c:972
#1 0x00007ffff456a455 in srtp_aes_gcm_nss_do_crypto (enc_len=0x7fffffffca3c,
buf=0x7fffffffcb50 "\331\061\062%\370\204\006\345\245Y\tů\365&\232\206\247\251S\025\064\367\332.L0=\212\061\212r\034<\f\225\225h\tS/\317\016$I\246\265%\261j\355\365\252\r\346W\272c{9", encrypt=1, cv=0x5555562a19c0) at crypto/cipher/aes_gcm_nss.c:297
#2 srtp_aes_gcm_nss_encrypt (cv=0x5555562a19c0, buf=<optimized out>, enc_len=0x7fffffffca3c) at crypto/cipher/aes_gcm_nss.c:345
#3 0x00007ffff456cb24 in srtp_cipher_type_test (ct=0x7ffff457a6c0 <srtp_aes_gcm_128>, test_data=0x7ffff457a420 <srtp_aes_gcm_test_case_0>)
at crypto/cipher/cipher.c:297
#4 0x00007ffff456d545 in srtp_cipher_type_test (ct=<optimized out>, test_data=<optimized out>) at crypto/cipher/cipher.c:605
#5 0x00007ffff456d58d in srtp_cipher_type_self_test (ct=<optimized out>) at crypto/cipher/cipher.c:613
#6 0x00007ffff457005d in srtp_crypto_kernel_do_load_cipher_type (replace=0, id=6, new_ct=0x7ffff457a6c0 <srtp_aes_gcm_128>)
at crypto/kernel/crypto_kernel.c:293
#7 srtp_crypto_kernel_load_cipher_type (new_ct=new_ct#entry=0x7ffff457a6c0 <srtp_aes_gcm_128>, id=id#entry=6) at crypto/kernel/crypto_kernel.c:343
#8 0x00007ffff457036e in srtp_crypto_kernel_init () at crypto/kernel/crypto_kernel.c:139
#9 srtp_crypto_kernel_init () at crypto/kernel/crypto_kernel.c:72
#10 0x00007ffff457040d in srtp_init () at srtp/srtp.c:2729
#11 0x00007ffff458718d in res_srtp_init () at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/res/res_srtp.c:1237
#12 load_module () at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/res/res_srtp.c:1272
#13 0x000055555566c4dc in start_resource.part.0.lto_priv.0 (mod=0x555555a469d0) at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/main/loader.c:1718
#14 0x0000555555665517 in start_resource (mod=0x555555a469d0) at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/main/loader.c:1692
#15 start_resource_attempt (mod=mod#entry=0x555555a469d0, count=count#entry=0x7fffffffce94)
at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/main/loader.c:1894
#16 0x000055555566862f in start_resource_list (mod_count=0x7fffffffce94, resources=0x7fffffffceb0)
at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/main/loader.c:1991
#17 load_resource_list (mod_count=<synthetic pointer>, load_order=0x7fffffffcea0) at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/main/loader.c:2173
#18 load_modules () at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/main/loader.c:2396
#19 0x000055555559e074 in asterisk_daemon (isroot=<optimized out>, rungroup=<optimized out>, runuser=<optimized out>)
at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/main/asterisk.c:4258
#20 main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/asterisk-16.28.0-0.el9.x86_64/main/asterisk.c:4025
After going through the debug steps a few more times, I tried searching the initial return value from srtp_kernel_init() which was srtp_err_status_cipher_fail; I ended up finding a resolved issue with the same symptoms for Fedora Linux:
The issue turns out to be an incompatibility with nss-3.63 shipped with F34. The attached patch adds NSS_PKCS11_2_0_COMPAT defines to 2 header files to enable the backward compatibility. The same fix is already in the upstream github repo and is targeted for the next libsrtp release.
I was able to apply their patch to the libsrtp 2.3.0 codebase, and the library is successfully loading now:
index 4d6031f..b1da343 100644
--- a/crypto/include/aes_gcm.h
+++ b/crypto/include/aes_gcm.h
## -66,6 +66,8 ## typedef struct {
#ifdef NSS
+#define NSS_PKCS11_2_0_COMPAT 1
+
#include <nss.h>
#include <pk11pub.h>
index ad306dd..a57564f 100644
--- a/crypto/include/aes_icm_ext.h
+++ b/crypto/include/aes_icm_ext.h
## -65,6 +65,8 ## typedef struct {
#ifdef NSS
+#define NSS_PKCS11_2_0_COMPAT 1
+
#include <nss.h>
#include <pk11pub.h>
I'm not sure the second hunk is needed, as ICM ciphers were loading fine, but I applied it anyway.
I'm writing a library that hooks some CUDA functions to add some functionality. The "constructor" hooks the CUDA functions and set up message queue and shared memory to communicate with other hooked CUDA binaries. When launching several hooked CUDA binaries (by python subprocess.Popen('<path-to-binary>', shell=True)) some processes hangs. So I used gdb -p <pid> to attach one suspended process, hoping to figure out what's going wrong. Here's the result:
Attaching to process 7445
Reading symbols from /bin/dash...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.27.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done.
done.
0x00007f9cefe8b76a in wait4 () at ../sysdeps/unix/syscall-template.S:78
78 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007f9cefe8b76a in wait4 () at ../sysdeps/unix/syscall-template.S:78
#1 0x000055fff93be8a0 in ?? ()
#2 0x000055fff93c009d in ?? ()
#3 0x000055fff93ba6d8 in ?? ()
#4 0x000055fff93b949e in ?? ()
#5 0x000055fff93b9eda in ?? ()
#6 0x000055fff93b7944 in ?? ()
#7 0x00007f9cefdc8b97 in __libc_start_main (main=0x55fff93b7850, argc=3, argv=0x7ffca7c7beb8, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffca7c7bea8) at ../csu/libc-start.c:310
#8 0x000055fff93b7a4a in ?? ()
I've added -g flag but it seems that the program hangs on wait4 before entering main.
Thanks for any insights on:
How can I load these debug symbols to get rid of ??
Where is ../csu/libc-start.c:310 located?
What else can I do to locate the bug?
System Info: gcc 6.5.0, Ubuntu 18.04 with 4.15.0-54-generic.
How can I load these debug symbols to get rid of ??
You appear to need the debug symbols for /bin/dash, which are probably going to be in a package called dash-dbg or dash-dbgsym or something like that.
Also, I suspect your stack trace would make more sense if you compiled your library with -fno-optimize-sibling-calls.
Where is ../csu/libc-start.c:310 located?
See this answer.
What else can I do to locate the bug?
You said that you are writing a library that uses __attribute__((constructor)), but you showed a stack trace for /bin/dash (which I presume is DASH and not a program you wrote) that does not appear to involve symbols from your library. I infer from this, that your library is loaded with LD_PRELOAD into programs that are not expecting it to be there.
Both of those things -- LD_PRELOAD and __attribute__((constructor)) -- break the normal expectations of both whatever unsuspecting program is involved, and the C library. You should only do those things if you have no other choice, and you should try to do as little as possible within the injected code. (In particular, I do not think any design that involves spawning processes from a constructor function will be workable, period.) If you tell us about your larger goals we may be able to suggest alternative means that are less troublesome.
EDIT:
subprocess.Popen('<path-to-binary>', shell=True)
With shell=True, Python doesn't invoke the program directly, it runs a command of the form /bin/sh -c 'string passed to Popen'. In many cases this will naturally produce a /bin/dash process sleeping (not hung) in a wait syscall for the entire lifetime of the actual binary. Unless you actually need to evaluate some shell code before running the program, try the default shell=False instead and see if that makes your problem go away. (If you do need to evaluate shell code, try Popen('<shell code>; exec <binary>', shell=True).)
My program is multi thread. i got a core file and when i try to debug it i got this.
Program terminated with signal 11, Segmentation fault.
#0 memcpy () at ../sysdeps/x86_64/memcpy.S:91
91 movl %ecx, (%rdi)
Missing separate debuginfos, use: debuginfo-install libssh2-1.8.0-2.0.cf.rhel6.x86_64
(gdb) bt
#0 memcpy () at ../sysdeps/x86_64/memcpy.S:91
#1 0x00007f981b342feb in ?? ()
#2 0x00000000025f1ef0 in ?? ()
#3 0x00000000025edef0 in ?? ()
#4 0x00007fff4b65a810 in ?? ()
#5 0x0000000000000001 in ?? ()
#6 0x00000000025cb800 in ?? ()
#7 0x00000000025ccea0 in ?? ()
#8 0x0000000000000000 in ?? ()
Why the bt infos are "???" Can i identify which thread and where case the seg fault?
Thank you.
In order to run gdb and make the best use of it, firstly, you need to compile your source with the -g or -ggdb3 option of gcc in the following way:
gcc -ggdb3 sample.c -o sample
After this you will get an executable or binary file which you can execute. Upon execution the program will generate a segfault and a coredump will be created. You can use this core file in the following way with gdb to obtain the backtrace:
gdb ./sample /path/to/core/file
You can even launch your program using gdb without actually executing it separately and generating the core file explicitly. If you want to do this, execute the following command:
gdb ./sample
The "??" entries are where symbol translation failed. Stack walking – which produces the stack trace – can also fail. In that case you'll likely see a single valid frame, then a small number of bogus addresses. If symbols or stacks are too badly broken to make sense of the stack trace, then there are usually ways to fix it: installing debug info packages (giving gdb more symbols, and letting it do DWARF-based stack walks), or recompiling the software from source with frame pointers and debugging information (-fno-omit-frame-pointer -g).
I downloaded the source for libc6 and completed the build process successfully. (Though I did not performed a make install deliberately).
With the new linker built in buil-dir/elf/ld.so I ran a program supplying it as the argument to the newly built linker.
The test code prints some string and then malloc(sizeof(char)*1024).
On running the test binary as an argument to the newly built linker I get a Seg Fault at elf/dl-addr.c:132 which is:
131 /* Protect against concurrent loads and unloads. */
132 __rtld_lock_lock_recursive (GL(dl_load_lock));
This is the last frame before the seg fault and is called through malloc() call from the test program.
Stack Trace at that point :
#0 0x0000000000000000 in ?? ()
#1 0x00007f11a6a94928 in __GI__dl_addr (address=0x7f11a69e67a0 <ptmalloc_init>, info=0x7fffe9393be0, mapp=0x7fffe9393c00, symbolp=0x0) at dl-addr.c:132
#2 0x00007f11a69e64d7 in ptmalloc_init () at arena.c:381
#3 0x00007f11a69e72b8 in ptmalloc_init () at arena.c:371
#4 malloc_hook_ini (sz=<optimized out>, caller=<optimized out>) at hooks.c:32
#5 0x00000000004005b3 in main () at test.c:20
On running the same program with the default installed linker on the machine the program runs fine.
I am not able to understand what can be the issue behind this? (Is it faulting because I am using the newly built linker without installing it first)
-Any suggestions or pointers are highly appreciated.
Thanks
(System details GCC 4.8.22, eglibc-2.15 Ubuntu 12.10 64bit
With the new linker built in buil-dir/elf/ld.so I ran a program supplying it as the argument to the newly built linker.
It that's all you did, then the crash is expected, because you are mixing newly-built loader with system libraries (which doesn't work: all parts of glibc must come from the same build of it).
What you need to do is:
buil-dir/elf/ld.so \
--library-path buil-dir:buil-dir/dlfcn:buil-dir/nptl:... \
/path/to/a.out
The list of directories to search must include all the libraries (parts of glibc) that your program uses.
I am analyzing this core dump
Program received signal SIGABRT, Aborted.
0xb7fff424 in __kernel_vsyscall ()
(gdb) where
#0 0xb7fff424 in __kernel_vsyscall ()
#1 0x0050cd71 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x0050e64a in abort () at abort.c:92
#3 0x08083b3b in ?? ()
#4 0x08095461 in ?? ()
#5 0x0808bdea in ?? ()
#6 0x0808c4e2 in ?? ()
#7 0x080b683b in ?? ()
#8 0x0805d845 in ?? ()
#9 0x08083eb6 in ?? ()
#10 0x08061402 in ?? ()
#11 0x004f8cc6 in __libc_start_main (main=0x805f390, argc=15, ubp_av=0xbfffef64, init=0x825e220, fini=0x825e210,
rtld_fini=0x4cb220 <_dl_fini>, stack_end=0xbfffef5c) at libc-start.c:226
#12 0x0804e5d1 in ?? ()
I'm not able to know which function ?? maps to OR for instance #10 0x08061402 in ?? ()
falls in which address range ...
Please help me debug this.
Your program has no debugging symbols. Recompile it with -g. Make sure you haven't stripped your executable, e.g. by passing -s to the linker.
Even though #user794080 didn't say so, it appears exceedingly likely that his program is a 32-bit linux executable.
There are two possible reasons (I can think of) for symbols from main executable (and all symbols in the stack trace in the range [0x08040000,0x08100000) are from the main executable) not to show up.
The main executable has in fact been stripped (this is the same as
ninjalj's answer), and often happens when '-s' is passed into the linker, perhaps inadvertently.
The executable has been compiled with a new(er) GCC, but is being debugged by an old(er) GDB, which chokes on some newer dwarf construct (there should be a warning from GDB about that).
To know what libraries are mapped into the application, record a pid of you program, stopped in gdb and run in other console
cat /proc/$pid/maps
wher $pid is the pid of stopped process. Format of the maps file is described at http://linux.die.net/man/5/proc - starting from "/proc/[number]/maps
A file containing the currently mapped memory regions and their access permissions."
Also, if your OS don't use a ASLR (address space layout randomization) or it is disabled for your program, you can use
ldd ./program
to list linked libraries and their memory ranges. But if ASLR is turned on, you will be not able to get real memory mapping ranges info, as it will change for each run of program. But even then you will know, what libraries are linked in dynamically and install a debuginfo for them.
The stack might be corrupted. The "??" can happen if the return address on the stack has been overwritten by, for example, a buffer overflow.