I am analyzing this core dump
Program received signal SIGABRT, Aborted.
0xb7fff424 in __kernel_vsyscall ()
(gdb) where
#0 0xb7fff424 in __kernel_vsyscall ()
#1 0x0050cd71 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x0050e64a in abort () at abort.c:92
#3 0x08083b3b in ?? ()
#4 0x08095461 in ?? ()
#5 0x0808bdea in ?? ()
#6 0x0808c4e2 in ?? ()
#7 0x080b683b in ?? ()
#8 0x0805d845 in ?? ()
#9 0x08083eb6 in ?? ()
#10 0x08061402 in ?? ()
#11 0x004f8cc6 in __libc_start_main (main=0x805f390, argc=15, ubp_av=0xbfffef64, init=0x825e220, fini=0x825e210,
rtld_fini=0x4cb220 <_dl_fini>, stack_end=0xbfffef5c) at libc-start.c:226
#12 0x0804e5d1 in ?? ()
I'm not able to know which function ?? maps to OR for instance #10 0x08061402 in ?? ()
falls in which address range ...
Please help me debug this.
Your program has no debugging symbols. Recompile it with -g. Make sure you haven't stripped your executable, e.g. by passing -s to the linker.
Even though #user794080 didn't say so, it appears exceedingly likely that his program is a 32-bit linux executable.
There are two possible reasons (I can think of) for symbols from main executable (and all symbols in the stack trace in the range [0x08040000,0x08100000) are from the main executable) not to show up.
The main executable has in fact been stripped (this is the same as
ninjalj's answer), and often happens when '-s' is passed into the linker, perhaps inadvertently.
The executable has been compiled with a new(er) GCC, but is being debugged by an old(er) GDB, which chokes on some newer dwarf construct (there should be a warning from GDB about that).
To know what libraries are mapped into the application, record a pid of you program, stopped in gdb and run in other console
cat /proc/$pid/maps
wher $pid is the pid of stopped process. Format of the maps file is described at http://linux.die.net/man/5/proc - starting from "/proc/[number]/maps
A file containing the currently mapped memory regions and their access permissions."
Also, if your OS don't use a ASLR (address space layout randomization) or it is disabled for your program, you can use
ldd ./program
to list linked libraries and their memory ranges. But if ASLR is turned on, you will be not able to get real memory mapping ranges info, as it will change for each run of program. But even then you will know, what libraries are linked in dynamically and install a debuginfo for them.
The stack might be corrupted. The "??" can happen if the return address on the stack has been overwritten by, for example, a buffer overflow.
Related
I'm writing a library that hooks some CUDA functions to add some functionality. The "constructor" hooks the CUDA functions and set up message queue and shared memory to communicate with other hooked CUDA binaries. When launching several hooked CUDA binaries (by python subprocess.Popen('<path-to-binary>', shell=True)) some processes hangs. So I used gdb -p <pid> to attach one suspended process, hoping to figure out what's going wrong. Here's the result:
Attaching to process 7445
Reading symbols from /bin/dash...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.27.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done.
done.
0x00007f9cefe8b76a in wait4 () at ../sysdeps/unix/syscall-template.S:78
78 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007f9cefe8b76a in wait4 () at ../sysdeps/unix/syscall-template.S:78
#1 0x000055fff93be8a0 in ?? ()
#2 0x000055fff93c009d in ?? ()
#3 0x000055fff93ba6d8 in ?? ()
#4 0x000055fff93b949e in ?? ()
#5 0x000055fff93b9eda in ?? ()
#6 0x000055fff93b7944 in ?? ()
#7 0x00007f9cefdc8b97 in __libc_start_main (main=0x55fff93b7850, argc=3, argv=0x7ffca7c7beb8, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffca7c7bea8) at ../csu/libc-start.c:310
#8 0x000055fff93b7a4a in ?? ()
I've added -g flag but it seems that the program hangs on wait4 before entering main.
Thanks for any insights on:
How can I load these debug symbols to get rid of ??
Where is ../csu/libc-start.c:310 located?
What else can I do to locate the bug?
System Info: gcc 6.5.0, Ubuntu 18.04 with 4.15.0-54-generic.
How can I load these debug symbols to get rid of ??
You appear to need the debug symbols for /bin/dash, which are probably going to be in a package called dash-dbg or dash-dbgsym or something like that.
Also, I suspect your stack trace would make more sense if you compiled your library with -fno-optimize-sibling-calls.
Where is ../csu/libc-start.c:310 located?
See this answer.
What else can I do to locate the bug?
You said that you are writing a library that uses __attribute__((constructor)), but you showed a stack trace for /bin/dash (which I presume is DASH and not a program you wrote) that does not appear to involve symbols from your library. I infer from this, that your library is loaded with LD_PRELOAD into programs that are not expecting it to be there.
Both of those things -- LD_PRELOAD and __attribute__((constructor)) -- break the normal expectations of both whatever unsuspecting program is involved, and the C library. You should only do those things if you have no other choice, and you should try to do as little as possible within the injected code. (In particular, I do not think any design that involves spawning processes from a constructor function will be workable, period.) If you tell us about your larger goals we may be able to suggest alternative means that are less troublesome.
EDIT:
subprocess.Popen('<path-to-binary>', shell=True)
With shell=True, Python doesn't invoke the program directly, it runs a command of the form /bin/sh -c 'string passed to Popen'. In many cases this will naturally produce a /bin/dash process sleeping (not hung) in a wait syscall for the entire lifetime of the actual binary. Unless you actually need to evaluate some shell code before running the program, try the default shell=False instead and see if that makes your problem go away. (If you do need to evaluate shell code, try Popen('<shell code>; exec <binary>', shell=True).)
My program is multi thread. i got a core file and when i try to debug it i got this.
Program terminated with signal 11, Segmentation fault.
#0 memcpy () at ../sysdeps/x86_64/memcpy.S:91
91 movl %ecx, (%rdi)
Missing separate debuginfos, use: debuginfo-install libssh2-1.8.0-2.0.cf.rhel6.x86_64
(gdb) bt
#0 memcpy () at ../sysdeps/x86_64/memcpy.S:91
#1 0x00007f981b342feb in ?? ()
#2 0x00000000025f1ef0 in ?? ()
#3 0x00000000025edef0 in ?? ()
#4 0x00007fff4b65a810 in ?? ()
#5 0x0000000000000001 in ?? ()
#6 0x00000000025cb800 in ?? ()
#7 0x00000000025ccea0 in ?? ()
#8 0x0000000000000000 in ?? ()
Why the bt infos are "???" Can i identify which thread and where case the seg fault?
Thank you.
In order to run gdb and make the best use of it, firstly, you need to compile your source with the -g or -ggdb3 option of gcc in the following way:
gcc -ggdb3 sample.c -o sample
After this you will get an executable or binary file which you can execute. Upon execution the program will generate a segfault and a coredump will be created. You can use this core file in the following way with gdb to obtain the backtrace:
gdb ./sample /path/to/core/file
You can even launch your program using gdb without actually executing it separately and generating the core file explicitly. If you want to do this, execute the following command:
gdb ./sample
The "??" entries are where symbol translation failed. Stack walking – which produces the stack trace – can also fail. In that case you'll likely see a single valid frame, then a small number of bogus addresses. If symbols or stacks are too badly broken to make sense of the stack trace, then there are usually ways to fix it: installing debug info packages (giving gdb more symbols, and letting it do DWARF-based stack walks), or recompiling the software from source with frame pointers and debugging information (-fno-omit-frame-pointer -g).
I'm using gdb to attach a running process, however, it always stops at __kernel_vsyscall. It looks like it stopped at my system call msgrcv(). I have to constantly "cont" it and don't know when it could jump out of kernel and go back to application. How can I make it continue? The following is my procedure.
How did I get this situation?
How to make it continue?
Thanks!
gdb
(gdb) attach PID
...
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
0xffffe410 in __kernel_vsyscall ()
(gdb)bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x009ed573 in msgrcv () from /lib/libc.so.6
#2 0xf7f3a487 in _UX_wgetmsg (mode=0, msgp=0xffbb4178, pmaxtime=0xffbb4164,
pdata=0xf7f7a860, ux_type=0) at ../../../ux/com_ux/libux/com/UXipc.c:2550
#3 0xf7f3ad05 in UX_wgetmsg_v2 (mode=0, msgp=0xffbb4178, maxtime=10000,
ux_type=0) at ../../../ux/com_ux/libux/com/UXipc.c:2237
#4 0x0804bb9b in main (argc=1, argv=0xffbb5394)
at /path/to/my_application:243
How did I get this situation?
That situation is completely normal for when you attach to a process which is blocked in a system call (waiting for message, or for read to complete).
How to make it continue?
You type continue (at which point the application would again block, waiting for a message). If you want to debug some part of the application, set breakpoints before continuing.
I'm trying to analyze core dump, but i get following result.
If i make main.elf statically linked everything is OK and i can see bt of all threads.
Any ideas?
GNU gdb 6.6.0.20070423-cvs
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "--host=mipsel-linux --target=mipsel-linux-uclibc".
(gdb) file main.elf
Reading symbols from /home/tobi/main.elf...Reading symbols from /home/tobi/main.dbg...done.
done.
(gdb) core-file /srv/tobi/core
warning: .dynamic section for "/lib/libpthread.so.0" is not at the expected address (wrong library or version mismatch?)
Error while mapping shared library sections:
/lib/libdl.so.0: No such file or directory.
Error while mapping shared library sections:
/lib/librt.so.0: No such file or directory.
Error while mapping shared library sections:
/lib/libm.so.0: No such file or directory.
Error while mapping shared library sections:
/lib/libstdc++.so.6: No such file or directory.
Error while mapping shared library sections:
/lib/libc.so.0: No such file or directory.
warning: .dynamic section for "/lib/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?)
Error while mapping shared library sections:
/lib/ld-uClibc.so.0: No such file or directory.
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Symbol file not found for /lib/libdl.so.0
Symbol file not found for /lib/librt.so.0
Symbol file not found for /lib/libm.so.0
Symbol file not found for /lib/libstdc++.so.6
Symbol file not found for /lib/libc.so.0
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Symbol file not found for /lib/ld-uClibc.so.0
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
Core was generated by 'root/main.elf'.
Program terminated with signal 11, Segmentation fault.
#0 0x0046006c in NullPtr (parse_p=0x2ac9dc80, result_sym_p=0x13e3d6c "") at folder/my1.c:1624
1624 *ptr += 13;
(gdb) bt
#0 0x0046006c in NullPtr (parse_p=0x2ac9dc80, result_sym_p=0x13e3d6c "") at folder/my1.c:1624
#1 0x0047a31c in fn1 (line_ptr=0x2ac9dd18 "ccore_null_pointer", target_ptr=0x13e3d6c "", result_ptr=0x2ac9dd14) at folder/my2.c:980
#2 0x0047b9d0 in fn2 (macro_ptr=0x0, rtn_exp_ptr=0x0) at folder/my3.c:1483
/... some functions .../
#8 0x2aab7f9c in __nptl_setxid () from /lib/libpthread.so.0
Backtrace stopped: frame did not save the PC
(gdb) thread apply all bt
Thread 159 (process 1093):
#0 0x2aac15dc in _Unwind_GetCFA () from /lib/libpthread.so.0
#1 0x2afdfde8 in ?? ()
warning: GDB cant find the start of the function at 0x2afdfde8.
GDB is unable to find the start of the function at 0x2afdfde8
and thus cant determine the size of that functions stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
This problem is most likely caused by an invalid program counter or
stack pointer.
However, if you think GDB should simply search farther back
from 0x2afdfde8 for code which looks like the beginning of a
function, you can increase the range of the search using the set
heuristic-fence-post command.
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
Thread 158 (process 1051):
#0 0x2aac17bc in pthread_mutexattr_getprioceiling () from /lib/libpthread.so.0
#1 0x2aac17a0 in pthread_mutexattr_getprioceiling () from /lib/libpthread.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 157 (process 1057):
#0 0x2aabf908 in ?? () from /lib/libpthread.so.0
#1 0x00000000 in ?? ()
Thread 156 (process 1090):
#0 0x2aac17bc in pthread_mutexattr_getprioceiling () from /lib/libpthread.so.0
#1 0x2aac17a0 in pthread_mutexattr_getprioceiling () from /lib/libpthread.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 155 (process 1219):
#0 0x2aabf908 in ?? () from /lib/libpthread.so.0
#1 0x00000000 in ?? ()
Thread 154 (process 1218):
#0 0x2aabfb44 in connect () from /lib/libpthread.so.0
#1 0x00000000 in ?? ()
Thread 153 (process 1096):
#0 0x2abc92b4 in ?? ()
warning: GDB cant find the start of the function at 0x2abc92b4.
#1 0x2abc92b4 in ?? ()
warning: GDB cant find the start of the function at 0x2abc92b4.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 152 (process 1170):
#0 0x2aabfb44 in connect () from /lib/libpthread.so.0
#1 0x00000000 in ?? ()
You need tell the gdb where it can find its shared-libraries. I suppose you are cross-debugging a core that was generated on another system?
Google for "set solib-absolute-prefix".
Guys, thanks you all a lot!
All i needed to do is
set solib-absolute-prefix [path-to-libs]
BEFORE loading an executable and core files :DDDD
I am writing a plugin for audacious, and I am experiencing random segfaults. I looked around and I found that I can process the program's core dumps with gdb.
So I did that, and I got this output:
http://pastebin.com/m7d0d663d
As you can see, it says no debugging symbols where found anywhere. I want to compile audacious with debugging symbols, but I am not sure how. I tried editing configure, which only includes a file named buildsys.mk, so I edited that and removed the -s flag from the linker, and made sure that the -g flag is passed to the compiler. The gdb output above is after I did that, so apparently what I did had no effect.
So how can I retain debugging symbols when compiling audacious? The problem is that I am only writing a small plugin, and haven't got a grasp of the while audacious code.
UPDATE: I added debugging symbols for gtk+ and glib (and also tried the CFLAGS=-g option), and I got a couple of coredumps analyzed. The bottom line is this:
(gdb) bt
#0 gtk_text_iter_make_real (_iter=<value optimized out>) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c:202
#1 0xb7c1cf5e in _gtk_text_iter_get_any_segment (iter=0x0) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c:474
#2 0xb7c24cd6 in IA__gtk_text_layout_get_line_display (layout=0x93a4318, line=0x9af6270, size_only=1) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextlayout.c:2196
#3 0xb7c29172 in gtk_text_layout_real_wrap (layout=0x93a4318, line=0x9af6270, line_data=0xb10036b8) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextlayout.c:1147
#4 0xb7c2358f in IA__gtk_text_layout_wrap (layout=0x93a4318, line=0x9af6270, line_data=0x0) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextlayout.c:693
#5 0xb7c060a1 in _gtk_text_btree_validate_line (tree=0x9407370, line=0x9af6270, view_id=0x93a4318) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextbtree.c:5422
#6 0xb7c27dc1 in IA__gtk_text_layout_validate_yrange (layout=0x93a4318, anchor=0xbfb0e624, y0=0, y1=635) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextlayout.c:1062
#7 0xb7c34999 in gtk_text_view_validate_onscreen (text_view=0x9406000) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextview.c:3502
#8 0xb7c35f85 in gtk_text_view_flush_first_validate (text_view=0x9406000) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextview.c:3558
#9 0xb7c35fde in first_validate_callback (data=0x9406000) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextview.c:3577
#10 0xb79c88fb in gdk_threads_dispatch (data=0x9bce910) at /build/buildd/gtk+2.0-2.16.1/gdk/gdk.c:498
#11 0xb7e38c81 in g_idle_dispatch (source=0x938a400, callback=0, user_data=0x9bce910) at /build/buildd/glib2.0-2.20.1/glib/gmain.c:3922
#12 0xb7e3ab88 in IA__g_main_context_dispatch (context=0x9250760) at /build/buildd/glib2.0-2.20.1/glib/gmain.c:1814
#13 0xb7e3e0eb in g_main_context_iterate (context=0x9250760, block=1, dispatch=1, self=0x92333e8) at /build/buildd/glib2.0-2.20.1/glib/gmain.c:2448
#14 0xb7e3e5ba in IA__g_main_loop_run (loop=0x9a92c88) at /build/buildd/glib2.0-2.20.1/glib/gmain.c:2656
#15 0xb7b707d9 in IA__gtk_main () at /build/buildd/gtk+2.0-2.16.1/gtk/gtkmain.c:1205
#16 0xb268d56a in skins_init () from /usr/local/lib/audacious/General/skins.so
#17 0x0805b42a in ?? ()
#18 0xb7540775 in __libc_start_main () from /lib/tls/i686/cmov/libc.so.6
#19 0x08055361 in ?? ()
(gdb)
And the exact error is:
#0 gtk_text_iter_make_real (_iter=<value optimized out>) at /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c:202
202 /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c: No such file or directory.
in /build/buildd/gtk+2.0-2.16.1/gtk/gtktextiter.c
The exact line is this:
if (iter->segments_changed_stamp !=
Can anyone make anything out of this? :-\
While it's true that doing either:
./configure CFLAGS=-g && make && make install
or
make CFLAGS=-g
will get you a build with debug symbols, this is quite unlikely to help you solve the problem.
You program crashed in /usr/lib/libgtk-x11-2.0.so.0, not in audacious2 (whatever that is). You are also analyzing the core incorrectly: list doesn't make sense at that stage. Your very first (gdb) command when analyzing a core should almost always be where, followed by thread apply all where.
You might also get better results from installing libgtk2-debuginfo or some such package, which should provide debug info for libgtk-x11-2.0.so.0, and may allow you to see the source and variables in libgtk at crash point.
As far as I know, audacious uses autotools. No need to modify anything, just configure with:
CFLAGS="-g $CFLAGS" ./configure
followed by the usual steps to install it. The flags are stored (in config.status I think), so any subsequent call to make will build a debug-enabled audacious.