Consider the following Linux kernel dump stack trace; e.g., you can trigger a panic from the kernel source code by calling panic("debugging a Linux kernel panic");:
[<001360ac>] (unwind_backtrace+0x0/0xf8) from [<00147b7c>] (warn_slowpath_common+0x50/0x60)
[<00147b7c>] (warn_slowpath_common+0x50/0x60) from [<00147c40>] (warn_slowpath_null+0x1c/0x24)
[<00147c40>] (warn_slowpath_null+0x1c/0x24) from [<0014de44>] (local_bh_enable_ip+0xa0/0xac)
[<0014de44>] (local_bh_enable_ip+0xa0/0xac) from [<0019594c>] (bdi_register+0xec/0x150)
In unwind_backtrace+0x0/0xf8 what does +0x0/0xf8 stand for?
How can I see the C code of unwind_backtrace+0x0/0xf8?
How to interpret the panic's content?
It's just an ordinary backtrace, those functions are called in reverse order (first one called was called by the previous one and so on):
unwind_backtrace+0x0/0xf8
warn_slowpath_common+0x50/0x60
warn_slowpath_null+0x1c/0x24
ocal_bh_enable_ip+0xa0/0xac
bdi_register+0xec/0x150
The bdi_register+0xec/0x150 is the symbol + the offset/length there's more information about that in Understanding a Kernel Oops and how you can debug a kernel oops. Also there's this excellent tutorial on Debugging the Kernel
Note: as suggested below by Eugene, you may want to try addr2line first, it still needs an image with debugging symbols though, for example
addr2line -e vmlinux_with_debug_info 0019594c(+offset)
Here are two alternatives for addr2line. Assuming you have the proper target's toolchain, you can do one of the following:
Use objdump:
locate your vmlinux or the .ko file under the kernel root directory, then disassemble the object file :
objdump -dS vmlinux > /tmp/kernel.s
Open the generated assembly file, /tmp/kernel.s. with a text editor such as vim. Go to
unwind_backtrace+0x0/0xf8, i.e. search for the address of unwind_backtrace + the offset. Finally, you have located the problematic part in your source code.
Use gdb:
IMO, an even more elegant option is to use the one and only gdb. Assuming you have the suitable toolchain on your host machine:
Run gdb <path-to-vmlinux>.
Execute in gdb's prompt: list *(unwind_backtrace+0x10).
For additional information, you may checkout the following resources:
Kernel Debugging Tricks.
Debugging The Linux Kernel Using Gdb
In unwind_backtrace+0x0/0xf8 what the +0x0/0xf8 stands for?
The first number (+0x0) is the offset from the beginning of the function (unwind_backtrace in this case). The second number (0xf8) is the total length of the function. Given these two pieces of information, if you already have a hunch about where the fault occurred this might be enough to confirm your suspicion (you can tell (roughly) how far along in the function you were).
To get the exact source line of the corresponding instruction (generally better than hunches), use addr2line or the other methods in other answers.
Related
I am looking for help with GDB to reverse engineer shared library written in C that is preloaded in /etc/ld.so.preload.
Current library hooks accept() call if source port is correct it returns reverse shell back to user.
Strings command doesn't give out source port, so my target is to try to find it within GDB.
Program consist of two files headers.h where I have my definitions and variable #define SECRET_PORT 11111
source.c contains accept hook with reverse shell.
My problem is I cannot figure out a way how to retrieve PORT within GDB - I can load mylib.so within gdb and run: info functions to see whats inside - I can see accept function but when I try to disass accept I only get instructions that I barely can understand.
Problem when I run mylib it gives out SIGSEGV (maybe thats the reason I cannot see variables) there is no main function where to set break and if I do set it on function accept is still gives SIGSEGV error.
I tested with starti instead of run then I got Program stopped 0xSOMEADRESGOESHERE in deregister_tm_clones() I don't even know if this is correct way to test .so file. maybe there are some oser switches.
Im thinking I need to find a way how to set BP in HTONS() checking function where if statement compares source port and extract values from there but so far no luck.
p.s. when mylib is loaded in gdb there is message No debugging symbols found. So I cannot run like list accept or anything like that to view a source.
Compilation code gcc -Wall -shared -fPIC mylib.c -o mylib.so -ldl
Im thinking I need to find a way how to set BP in HTONS() checking function where if statement compares source port and extract values from there
You don't need to do that -- the instructions will be the same whether you run the application, or disassemble the function without running.
Compilation code ...
So you are trying to reverse-engineer the library for which you have a source?
That makes it very easy to find the constant you are looking for.
Start by setting the constant to easily recognizable value, e.g. 0x12131415. Compile the library and disassemble it. Look for your constant.
If you don't see it, save the disassembled output, and rebuild the library with a different value, e.g. 0xA1B1C1D1. Disassemble it again and compare to previous disassembled output. It should be easy to spot the difference.
P.S. If you really want to debug this library with a live process, do this:
gdb ./myprog
(gdb) set env LD_PRELOAD /path/to/mylib.so
(gdb) run
At this point, you should be able to set breakpoints and observe your library "in action".
Ok managed solve this with a help
when running GDB on shared library You will have to check hex value for 11111 and it should be 2B67 so in registers this will become something like 0x2b67 & it will be passed to htons() as check for source port.
So let's assume I didn't have the source code I could still run: gdb -q *.so
then: info functions and see with disass functionNameGoesHere where some accept / htons calls are made. Correct value should be found right above htons line.
Then decoded hex to dec and thats how You can find it.
This took some while to figure out as I coudn't set BP's.
Again thanks for input from community! Cheers
I am trying to print all the Undefined function calls from a shared object file along with file name.
I tried with "nm" command, It print all the undefined function calls .But could not get the file name.
Example:
bash$ nm -u my_test.so
:
U _ZNSs4_Rep20_S_empty_rep_storageE##GLIBCXX_3.4
:
Environment : Ubuntu 18.04 , X86 Arch (Intel processor)
Study in details the specification of the DWARF format (which is the format used by debugging information on Linux). So you could extract the information (but it is not exactly simple) by parsing the DWARF inside your ELF binary.
Consider looking inside the source code of Ian Taylor's libbacktrace. It is doing this extraction of file name from DWARF inside ELF.
Perhaps your real problem is getting precise backtrace information, and then that libbacktrace is exactly what you need!
You might also use gdb : it is extensible and scriptable in Python (or Guile) and you could write your own specialized script.
Perhaps you'll better solve your real problem with some GCC plugin working when you compile your code.
Read How to write shared libraries by Drepper and read more about ELF.
You could for example collect all the undefined symbols in your shared library using nm (or readelf). Then a second script will find the occurrences of these in your source code. It could be even a simple awk script (or some for shell loop using grep), or something as sophisticated as a GCC plugin.
Your example shows (probably) a mangled C++ name. You could use nm -C to get it unmangled. And later write a GCC plugin to find all the GIMPLE CALL instructions using it.
Writing a GCC plugin may take some time, in particular if you are not familiar with GCC internals.
I wanted to debug printf function, so when I step inside the printf function (gdb debugger) it showed me this:
__printf (format=0x80484d0 " my name is Adam") at printf.c:28
28 printf.c: No such file or directory.
What is the meaning of this?
And when I again started step then there are a lot more statements like this.
Please help me to understand this.
I think it's pretty clear. There is a place where the gdb expects the source code to be, so download glibc's source code and put it there. I think the error message contains the full path.
If it's a linux distro it's fairly simple in fact because usually source packages are shipped too. Otherwise you need to find the source code yourself, note that it MUST be exactly the same that was used to compile the c library components, not just the same version because distributors often make changes to the sources.
Well, for the debugger to show you the code that was compiled into the binaries you're using, you need the original code somewhere.
You don't seem to have that, so your debugger can't find it.
Notice that you usually do not want to debug the source code of your std library functions, but only the way they are being called. For that, the usual "debug symbol" packages of your operating systems are optimal.
As others have answered, GDB was unable to find the source file.
For the C runtime libraries, Linux distributions may provide a debuginfo RPM that you can install, which may allow GDB to view the files. For example:
$ yum search glibc-debuginfo
...
glibc-debuginfo.x86_64 : Debug information for package glibc
glibc-debuginfo-common.x86_64 : Debug information for package glibc
...
The glibc package and the glibc-debuginfo are a matched pair. There is no explicit dependency, but glibc-debuginfo package won't work unless it is matched with the same version of glibc.
If you have the sources unpacked somewhere, but not where GDB is expecting them to be, you can attempt to use either the directory or the set substitute-path command to let GDB know where the sources are.
The directory command tells GDB to prepend a prefix ahead of any source file path it is attempting to find. For example, if the source tree is actually located under the /tmp, you could use:
(gdb) directory /tmp
The set substitute-path command is used to tell GDB to replace a matching prefix in a source file path with a different path prefix. For example, if the compiled source file was in /build/path/source.c, but in debugging the source file is actually in /usr/home/alice/release-1.1/source.c, then you could use:
(gdb) set substitute-path /build/path /usr/home/alice/release-1.1
The command assumes that you are only specifying a complete path names, so it won't perform the substitution on /build/pathological/source.c.
I wanted to debug printf function, so when I step inside the printf function (gdb debugger) it showed me this:
__printf (format=0x80484d0 " my name is Adam") at printf.c:28
28 printf.c: No such file or directory.
What is the meaning of this?
And when I again started step then there are a lot more statements like this.
Please help me to understand this.
I think it's pretty clear. There is a place where the gdb expects the source code to be, so download glibc's source code and put it there. I think the error message contains the full path.
If it's a linux distro it's fairly simple in fact because usually source packages are shipped too. Otherwise you need to find the source code yourself, note that it MUST be exactly the same that was used to compile the c library components, not just the same version because distributors often make changes to the sources.
Well, for the debugger to show you the code that was compiled into the binaries you're using, you need the original code somewhere.
You don't seem to have that, so your debugger can't find it.
Notice that you usually do not want to debug the source code of your std library functions, but only the way they are being called. For that, the usual "debug symbol" packages of your operating systems are optimal.
As others have answered, GDB was unable to find the source file.
For the C runtime libraries, Linux distributions may provide a debuginfo RPM that you can install, which may allow GDB to view the files. For example:
$ yum search glibc-debuginfo
...
glibc-debuginfo.x86_64 : Debug information for package glibc
glibc-debuginfo-common.x86_64 : Debug information for package glibc
...
The glibc package and the glibc-debuginfo are a matched pair. There is no explicit dependency, but glibc-debuginfo package won't work unless it is matched with the same version of glibc.
If you have the sources unpacked somewhere, but not where GDB is expecting them to be, you can attempt to use either the directory or the set substitute-path command to let GDB know where the sources are.
The directory command tells GDB to prepend a prefix ahead of any source file path it is attempting to find. For example, if the source tree is actually located under the /tmp, you could use:
(gdb) directory /tmp
The set substitute-path command is used to tell GDB to replace a matching prefix in a source file path with a different path prefix. For example, if the compiled source file was in /build/path/source.c, but in debugging the source file is actually in /usr/home/alice/release-1.1/source.c, then you could use:
(gdb) set substitute-path /build/path /usr/home/alice/release-1.1
The command assumes that you are only specifying a complete path names, so it won't perform the substitution on /build/pathological/source.c.
I am trying to navigate and understand whoami (and other coreutils) all the way down to the lowest level source code, just as an exercise.
My dive so far:
Where is the actual binary?
which whoami
/usr/bin/whoami
Where is it maintained?
http://www.gnu.org/software/coreutils/coreutils.html
How do I get source?
git clone git://git.sv.gnu.org/coreutils
Where is whoami source code within the repository?
# find . | grep whoami
./man/whoami.x
./man/whoami.1
./src/whoami.c
./src/whoami
./src/whoami.o
./src/.deps/src_libsinglebin_whoami_a-whoami.Po
./src/.deps/whoami.Po
relevant line (84):
uid = geteuid ();
This is approximately where my rabbit hole stops. geteuid() is mentioned in gnulib/lib/euidaccess.c, but not explicitly defined AFAICT. It's also referenced in /usr/local/unistd.h as extern but there's no heavy lifting related to grabbing a uid that I can see.
I got here by mostly grepping for geteuid within known system headers and includes as I'm having trouble backtracing its definition.
Question: How can I dive down further and explore the source code of geteuid()? What is the most efficient way to explore this codebase quickly without grepping around?
I'm on Ubuntu server 15.04 using Vim and some ctags (which hasn't been very helpful for navigating existing system headers). I'm a terrible developer and this is my method of learning, though I can't get through this roadblock.
Normally you should read the documentation for geteuid. You can either read GNU documentation, the specification from the Open Group or consult the man page.
If that doesn't help you could install the debug symbols for the c-library (it's called libc6-dbg or something similar) and download the source code for libc6) then you point out the path to the source file when you step into the library.
In this case I don't think this would take you much further, what probably happens in geteuid is that it simply issues an actual syscall and then it's into kernel space. You cannot debug that (kernel) code in the same way as you would debug a normal program.
So in your case you should better consult the documentation and read it carefully and try to figure out why geteuid doesn't return what you expect. Probably this will lead to you changing your expectation of what geteuid should return to match what's actually returned.